As you’d expect, rDigg is a Digg API wrapper in the form of a Ruby on Rails plugin. It still needs some work, but works pretty well.

For example:

#create new Rdigg object
digg = Rdigg.new

# find the 3 newest submissions from Kevin Rose
stories = digg.user.find_submissions("kevinrose", :count => 3)

# stories is now an array with a hash for each story
stories.first[:story] #=> the story's text
stories.first[:href] #=> the story's url
stories.first[:diggs] #=> number of diggs the story has

Grab it at: http://github.com/johnyerhot/rdigg/tree/master. I’ll have the Rdoc up at rdigg.yerhot.org later tonight You can peruse the documentation at rdigg.yerhot.org. I highly recommend you check out all methods that are available to you.

If you want to really dig in (sorry couldn’t resist) I’d go over the Digg API wiki to see what arguments you can pass.

I hope you enjoy the plugin!

First - Jen and I are taking a trip to Seattle for 7 days starting this Saturday.  We’re super pumped.  

Second – Over Christmas last year I started on this lil Rails app I was calling duluthrentalreview.com.  Pretty obvious what it was.  I get so pissed at landlords in this town, and even more pissed at my fellow renters for putting up with the crap quality of said rentals.  The problem is this:  Landlords in this town do not keep up their rentals because renters do not demand that they are nice and decent and that the landlords keep them up.  You get the idea. 

Ok, so yeah I started writing this rental review site over christmas, got like halfway though it and just didn’t have time to finish it.. till now.  I re-wrote almost all of it, convertging it to REST.   I’m gonna try to continue updating it over the next couple days till our trip.  Oh, ad the url changed from duluthrentalreview.com to rentalrundown.com… liked the sound of rentalrundown better.

DISCLAIMER: Web Scraping is kind of a gray area.. don’t steal things that are copywritten and don’t be a jerk. Give credit where credit is due..

First thing is first. You’ll need to install the Mechanize Ruby Gem.

sudo gem install mechanize

Mechanize is pretty slick. It will iterate through a given url and let you access various html elements. Further, you can use Hpricot methods to further grab data.Lets get going..

require ‘rubygems’require ‘mechanize’require ‘uri’

url = “http://www.johnyerhot.com”

The way this is set up, you MUST have a complete url.

@mech = WWW::Mechanize.new

@page = @mech.get(url)

Now, lets say we want to get all the urls for embedded images from the webpage (http://www.johnyerhot.com)..

@imgs = @page.search(”img[@src]“).map {|src| src['src']}

You’ve now got an array (@imgs) with all the urls for embeded images! What we actually did was use Hpricot’s search method to look for and image tags and sucked out the src attribute of that tag. Mechanize does have its own methods for grabbing tags also, for example, you can grab all the link targets from every link in the web page.

#remember @page is just our mechanize instance
# w/ http://www.johnyerhot.com

@links = Array.new
@page.links.each do |link|
@links << link.href
end

Now lets weed out any links to non-images:

@links.each do |link| #yeah we’re only collecting jpgs

if (link.to_s.include? “.jpg”) || (link.to_s.include? “.JPG”) || (link.to_s.include? “.jpeg”) || (link.to_s.include? “.JPEG”)

@imgs << link
end
end

Finally, lets actually grab all those pictures and save them locally using Open URI…

@counter =0
@imgs.each do |image|
url = URI.parse(image)#parse the url and separate need info
Net::HTTP.start(url.host, url.port) { |http|
#appeand the image path with the web root.
image = http.get(image)#actually make the file to save
open(”#{url.host}_#{counter}.jpg”, “wb”) { |file|
file.write(image.body)
counter = counter + 1
}
end

And there you have it! Put it all together and you should have a functioning Ruby web scraper…Sort of. You still have to account for relative vs. absolute urls, are you gonna let in more than jpgs?, what if you need basic authentication for the url? There are still some missing pieces that need to be implemented to have this be ready for general use, but the core is there.
Further Reading
Mechanize Docs
Hpricot
Open URI
Ruby net/http Docs

sudo gem install pdf-writer

I was greeted with nothing, just

Updating Gem source index for: http://gems.rubyforge.org

and that was it. After searching and searching I realized the the version of Gem I had installed was 0.94 and that was the problem.

The solution is as follows:

wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
tar zxvf rubygems-1.0.1.tgz
cd rubygems-1.0.1
ruby setup.rb
sudo gem update

You just need to manually install the updated Gem. Cinch.

In a previous post I talked about how I just set up my new Slicehost account using Nginx to power this blog (Wordpress – PHP) and some Rails Apps. Here’s a mini how to on how I got the whole thing working, as its not quite as easy as it sounds.

First though – Why not use Apache + mod_proxy… There is a ton of documentation out there for Apache, and the majority of the web runs on it?

Well, Nginx is super tiny and super light weight and super fast. If you are in a situation like many people who are using Slicehost and a 256 MB slice, Apache has too big a resource foot print. If you are a Rails guy (or gal) Apache is just used as a proxy to send requests to Mongrel and Apache will just use resources that could be used else where. Nginx is stable, well supported, and growing in popularity. Plus the name is cool.. NGINX.. yeah…

This will (should) work with a fresh install of Ubuntu Server 7.10, but as always YMMV.

A lot of this is adapted from a great post on How To Forge, thanks 3uropa.

Install only the OpenSSH packages NOT the LAMP packages when installing Ubuntu Server 7.10. You’ll have to setup networking for whatever your situation is so can’t help you there. If you have a Slicehost slice, don’t do anything. :)

Once that is finished installing Ubuntu Server, the first thing you want to do is update and get some prerequisites out of the way!

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential

Ok. Now, you need to go in and get all your files/directories in order. That mean, go to /var/www/ (or wherever) and make directories for your PHP site. In my case, I used /home/joyerhot/public_html/johnyerhot.com/wordpress as the root of my Wordpress blog. You should be able to either use wget to grab stuff or SFTP in with your username and password.

Ok. Next we will install PHP5 and MySQL

apt-get install php5-cli php5-cgi build-essential mysql-server mysql-client libmysqlclitene15-dev

Now, lets get Nginx while we’re at it! The version that is in Ubuntu’s repos is old and not cool, so we’ll manually grab a newer version:

wget http://technokracy.net/nginx/nginx_0.5.32~grrr-1_i386.deb
sudo dpkg nginx_*

And now Nginx is running and listening to port 8000.. we’ll change that later though.

Now, here is where it gets sticky. You are going to create and edit /etc/nginx/sites-enabled/mysite

sudo nano /etc/nginx/sites-enabled/mysite

and put in it the following, changing directories and whatnot to what is appropriate for you.

server {

listen 80;
server_name yourdomain.com;
rewrite ^/(.*) http://www.yourdomain.com permanent;

}

server {

listen 80;
server_name www.yourdomain.com;

access_log /home/joyerhot/public_html/yourdomain.com/logs/access.log;
error_log /home/joyerhot/public_html/yourdomain.com/logs/error.log;

if (!-e $request_filename) {
rewrite ^([_0-9a-zA-Z-]+)?(/wp-.*) $2 last;
rewrite ^([_0-9a-zA-Z-]+)?(/.*\.php)$ $2 last;
rewrite ^ /index.php last;
}

location / {

root /home/joyerhot/public_html/johnyerhot.com/wordpress;
index index.html index.php index.htm;

}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_FILENAME /home/joyerhot/public_html/yourdomain.com/wordpress/$fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
}
}

WHEW! There is A LOT of stuff there! This will setup a Virtual Host for your PHP site, and enable rewrites (for pretty urls). Here is what you need to change in yours though:

1. the server_name is whatever your domain is.
2. rewrite ^/(.*) http://www.whatever_goes_here.com permanent;
3. access_log directory
4. error_log directory
5. root directory
6. fastcgi_param SCRIPT_FILENAME /path/to/your/php/app/root/$fastcgi_script_name;
7. MAKE SURE YOU’RE LISTENING TO PORT 80.

Ok, now we’ll get PHP5 working as a fcgi process. To do this, we need to grab some of Lighttpd.

wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2
tar -xvjf lighttpd-*
cd lighttpd*
./configure
make

DO NOT “make install”. :)

sudo cp src/spawn-fcgi /usr/bin/spawn-fcgi
sudo nano /usr/bin/php-fastcgi

Now, add the following into php-fastcgi:

/usr/bin/spawn-fcgi -a 127.0.0.1 -p 9000 -C 2 -u www-data -f /usr/bin/php5-cgi

-C controls how many fcgi instances of PHP5 are spawned, so you can put however many are appropriate for you in there. I only needed 2.

sudo nano /etc/init.d/init-fastcgi

And add:

#!/bin/bash
PHP_SCRIPT=/usr/bin/php-fastcgi
RETVAL=0
case “$1″ in
start)
$PHP_SCRIPT
RETVAL=$?
;;
stop)
killall -9 php
RETVAL=$?
;;
restart)
killall -9 php
$PHP_SCRIPT
RETVAL=$?
;;
*)
echo “Usage: php-fastcgi {start|stop|restart}”
exit 1
;;
esac
exit $RETVAL

Do some permission magic:

sudo chmod 755 /etc/init.d/init-fastcgi
sudo chmod 755 /usr/bin/php-fastcgi

Now, you can make sure everything is working by:

/etc/init.d/init-fastcgi start
top #then shift + M

You should see a couple of PHP5 fcgi processes! YAY!
Finally, we want them to start on reboots and whatnot so…

update-rc.d init-fastcgi defaults

Ok, now lets restart Nginx and see if things are working!

/etc/init.d/nginx restart

You should be able to hit your Php app now!

Now for Ruby and friends!

sudo apt-get install libmysql-ruby1.8 ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8 libreadline-ruby1.8 libruby1.8 libopenssl-ruby irb1.8 libdbd-mysql-perl libdbi-perl libmysql-ruby1.8 libmysqlclient15-dev libmysqlclient15off libnet-daemon-perl libopenssl-ruby libopenssl-ruby1.8 libplrpc-perl libreadline-ruby1.8 libruby1.8 mysql-client mysql-client-5.0 mysql-common mysql-server mysql-server-5.0 rdoc1.8 ri1.8 ruby1.8 ruby1.8-dev zlib1g-dev

And you might need to create some symlinks (I had to), thanks Vince Wadhwanl.

sudo ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby
sudo ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc
sudo ln -s /usr/bin/ri1.8 /usr/local/bin/ri
sudo ln -s /usr/bin/irb1.8 /usr/local/bin/irb

Make sure things are working ok…

ruby -v

You should get that you have 1.8.6 installed.Ok, now lets install RubyGems.

wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
tar xvzf rubygems-1.0.1.tgz
cd rubygems-1.0.1
sudo ruby setup.rb
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem

And finally for Rails and Mongrel:

sudo gem install rails
sudo gem install mongrel mongrel_cluster

You can now set up a mongrel_cluster or whatever you need. There’s plenty of tutorials out there for that, so Google is your friend. The rest will assume you have your Rails app up and running on port 3000.
We now have to set up a site profile for Nginx for your mongrel_cluster.

sudo nano /etc/nginx/sites-enabled/rails_app

And add to it, once again changing the relevant parts….

upstream domain1 {
server 127.0.0.1:3000;
}

server {

listen 80;
server_name www.your2nddomain.com;
rewrite ^/(.*) http://subdomain.your2nddomain.com permanent;

}

server {

listen 80;
server_name subdomain.your2nddomain.com;

access_log /home/joyerhot/public_html/your2nddomain.com/logs/access.log;
error_log /home/joyerhot/public_html/your2nddomain.com/logs/error.log;

location / {

root /home/joyerhot/public_html/your2nddomain.com/public/RAILS_ROOT/public/;
index index.html;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect false;

if (-f $request_filename/index.html) {
rewrite (.*) $1/index.html break;
}

if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}

if (!-f $request_filename) {
proxy_pass http://domain1;
break;
}
}

}

}

Again, heres what you need to change:
1. server_name should be the domain
2. In this case the “rewrite” line will point to subdomain.your2nddomain.com, you should prolly just put your domain here(www.your2nddomain.com).
3. access_log directory
4. error_log directory
5. root directory (pointing to RAILS_ROOT/public)

That’s it! You should now restart Nginx (/etc/init.d/nginx restart) and when you should have everything working! You’ve got Virtual Hosts setup with PHP running as Fast CGI on one domain and Ruby On Rails on the other!

I’m sure I’ve got a typo or two in here and a mistake here and there, please leave a message if this worked or didn’t work for you. Remember, this is the process I went through getting everything working on my Slicehost Slice and another server at work, both running Ubuntu 7.10.

PS. Here is one little nugget of info that may be helpful – Your MySQL socket file location (for your RAILS_ROOT/config/database.yml file) is located at /var/run/mysqld/mysqld.sock. I didn’t realize this at first. :)

Obviously check your RAILS_ROOT/log and check backgroundrb.log, backgroundrb_server.log, and backgroundrb_debug.log. I’ve found backgroundrb.log most useful, but many times I’ll restart backgroundrb by

script/backgroundrb stop

script/backgroundrb start #restart doesn’t work for me

and nothing will happen. Check your backgroundrb_server.log file and you may see that the ‘address is already in use”.

Type “top” into your command line (on a Linux system) and then press shift+M. You should get a nice list of services running and you’ll more than likely have a couple of rubys on there. For some reason backgroundrb doesn’t stop all the time when you tell it to and I’ve had to kill -9 the rubys and then restart backgroundrb and mongrel.

You may also notice that the backgroundrb ruby instance uses ALOT of memory! Make sure you have

:environment: production

in your backgroundrb.yml file in RAILS_ROOT/config. Otherwise backgroundrb is working in development mode.

Finally, to debug an syntax or other error try this.

1. open 2 command lines.

2. start your rails app with ruby script/server. Make sure to start it in development mode and not using mongrel_rails.

3. start backgroundrb in the other terminal.

4. watch the first terminal and you can see what background rb is doing and where exactly it gets messed up.

You can do this by monitoring the log file or tailing it, but I like this better and it provides you with more info.

Hope this helps any of you out.

As mentioned, I decided to give Nginx a try. I have to admit, initial set up is very easy.  I first setup the Rails enviornment and had Nginx proxy to two mongrels for each app.  (I’m running two apps for four total Mongrels).  No problem.

Next, I had to get php working.  Not so easy.

I ended up looking for help and followed this, setting up php5 to run as fast-cgi.  Ugh.  To make things worse, I had to move over the Wordpress install from Godaddy.  I assummed it would be easy as pie, and it was to an extent, but I ran into trouble with the way the virtual hosts configs nginx uses handle subdomains and .. long story short I had redirect loop happening and it just sucked.

I’m blabering on and on here, but in the end I’ve got everything up and running.  We’ll see how Nginx works in the long term, but right now it seems pretty snappy and the memory footprint is much smaller than Apache.

So, Royner and another project of mine have a place to live and I’m just gonna do some final tweaks this weekend and then you can check them out.

First, create a new class in your application controller:

class Helper
include Singleton
include ActionView::Helpers::PdfHelper #or whatever helpers you want
end

I wanted to use my pdf_helpers
..and then:

def pdf_helpers
Helper.instance
end
Will create an instance of your new class.

Now, go to your pdf controller (or whatever) and simply:

pdf.text pdf_helpers.my_crazy_helper(@foo.bar)


The pdf.text part is not the important part, thats just for pdf-writer, but what is is using “pdf_helpers.your_helper” to create a new instance of the helper class which will allow you to use that helper.