Archive for the ‘Ruby On Rails’ Category

Mongrel + Nginx: Deploying to a subdirectory

Tuesday, May 27th, 2008

Though using subdomains is all the rage right now, there are certianly instances where you may want to deploy your rails application to a subdirectory such as:

http://www.johnyerhot.com/myrailsapp

Of course Nginx makes it super easy to do so. If you need to get your webserver ready with Nginx, PHP running as a FCGI instance, and Rails check out my other how to.  

Now, onward!

First create a new virtual host.  In my case, for yerhot.org it would look like this:
server {
listen 80;
server_name yerhot.org;
access_log /var/www/yerhot.org/logs/access.log;
error_log /var/www/yerhot.org/logs/error.log;
location / {
root /var/www/yerhot.org/;
index index.html;
}
}

Pretty simple setup, telling Nginx to listen on port 80 for requests for yerhot.org, where to store logs, and finally setting up the site root at /var/www/yerhot.org.

All we would have to do to have Nginx redirect to our Rails app when looking for yerhot.org/myrailsapp is make another location block (in other words, place this right before the last curly brace).

location /myrailsapp {
proxy_pass http://localhost:8000;
}

Now, all requests for /myrailsapp will get proxied to port 8000. Now fire up your Rails app on port 8000.
mongrel_rails start -e production -p 8000 -d
Restart Nginx:
/etc/init.d/nginx stop
/etc/init.d/nginx start

And….

Crap. Rails is looking for a ‘myrailsapp’ route, which there is none.  No, no, don’t create one – we’ll need to use a little known feature of Mongrel to fix the problem.. the prefix.
Stop Mongrel…
mongrel_rails stop
And try this:
mongrel_rails start -e production -p 8000 -d --prefix=/myrailsapp

And… your app should fire right up. Pretty neat, if you wanted to you could use it forward to a Mongrel Cluster, a FCGI instance of PHP (from my other post), or lots of stuff.

Super Simple Ruby Web Scraper

Monday, May 19th, 2008

Alrighty folks. Quick walk though for scraping remote web data with Ruby. This is how I did it for my little web scraper I wrote on Saturday..

DISCLAIMER: Web Scraping is kind of a gray area.. don’t steal things that are copywritten and don’t be a jerk. Give credit where credit is due..

First thing is first. You’ll need to install the Mechanize Ruby Gem.

sudo gem install mechanize

Mechanize is pretty slick. It will iterate through a given url and let you access various html elements. Further, you can use Hpricot methods to further grab data.Lets get going..

require ‘rubygems’require ‘mechanize’require ‘uri’

url = “http://www.johnyerhot.com”

The way this is set up, you MUST have a complete url.

@mech = WWW::Mechanize.new

@page = @mech.get(url)

Now, lets say we want to get all the urls for embedded images from the webpage (http://www.johnyerhot.com)..

@imgs = @page.search(”img[@src]“).map {|src| src['src']}

You’ve now got an array (@imgs) with all the urls for embeded images! What we actually did was use Hpricot’s search method to look for and image tags and sucked out the src attribute of that tag. Mechanize does have its own methods for grabbing tags also, for example, you can grab all the link targets from every link in the web page.

#remember @page is just our mechanize instance
# w/ http://www.johnyerhot.com

@links = Array.new
@page.links.each do |link|
@links << link.href
end

Now lets weed out any links to non-images:

@links.each do |link| #yeah we’re only collecting jpgs

if (link.to_s.include? “.jpg”) || (link.to_s.include? “.JPG”) || (link.to_s.include? “.jpeg”) || (link.to_s.include? “.JPEG”)

@imgs << link
end
end

Finally, lets actually grab all those pictures and save them locally using Open URI…

@counter =0
@imgs.each do |image|
url = URI.parse(image)#parse the url and separate need info
Net::HTTP.start(url.host, url.port) { |http|
#appeand the image path with the web root.
image = http.get(image)#actually make the file to save
open(”#{url.host}_#{counter}.jpg”, “wb”) { |file|
file.write(image.body)
counter = counter + 1
}
end

And there you have it! Put it all together and you should have a functioning Ruby web scraper…Sort of. You still have to account for relative vs. absolute urls, are you gonna let in more than jpgs?, what if you need basic authentication for the url? There are still some missing pieces that need to be implemented to have this be ready for general use, but the core is there.
Further Reading
Mechanize Docs
Hpricot
Open URI
Ruby net/http Docs

Slicehost problems with Gem *FIXED*

Monday, March 17th, 2008

Today I set up 470.johnyerhot.com and set up a Rails app I worked on to run there, but I needed to install the Pdf-writer Gem for proper functionality. Low and behold when I tried

sudo gem install pdf-writer

I was greeted with nothing, just

Updating Gem source index for: http://gems.rubyforge.org

and that was it. After searching and searching I realized the the version of Gem I had installed was 0.94 and that was the problem.

The solution is as follows:

wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
tar zxvf rubygems-1.0.1.tgz
cd rubygems-1.0.1
ruby setup.rb
sudo gem update

You just need to manually install the updated Gem. Cinch.

How to: Nginx, FCGI, PHP, MySQL, Ruby On Rails, Rewrite

Tuesday, February 12th, 2008

In a previous post I talked about how I just set up my new Slicehost account using Nginx to power this blog (Wordpress – PHP) and some Rails Apps. Here’s a mini how to on how I got the whole thing working, as its not quite as easy as it sounds.

First though – Why not use Apache + mod_proxy… There is a ton of documentation out there for Apache, and the majority of the web runs on it?

Well, Nginx is super tiny and super light weight and super fast. If you are in a situation like many people who are using Slicehost and a 256 MB slice, Apache has too big a resource foot print. If you are a Rails guy (or gal) Apache is just used as a proxy to send requests to Mongrel and Apache will just use resources that could be used else where. Nginx is stable, well supported, and growing in popularity. Plus the name is cool.. NGINX.. yeah…

This will (should) work with a fresh install of Ubuntu Server 7.10, but as always YMMV.

A lot of this is adapted from a great post on How To Forge, thanks 3uropa.

Install only the OpenSSH packages NOT the LAMP packages when installing Ubuntu Server 7.10. You’ll have to setup networking for whatever your situation is so can’t help you there. If you have a Slicehost slice, don’t do anything. :)
install ubuntu
install ubuntu 2

Once that is finished installing Ubuntu Server, the first thing you want to do is update and get some prerequisites out of the way!

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential

Ok. Now, you need to go in and get all your files/directories in order. That mean, go to /var/www/ (or wherever) and make directories for your PHP site. In my case, I used /home/joyerhot/public_html/johnyerhot.com/wordpress as the root of my Wordpress blog. You should be able to either use wget to grab stuff or SFTP in with your username and password.

Ok. Next we will install PHP5 and MySQL

apt-get install php5-cli php5-cgi build-essential mysql-server mysql-client libmysqlclitene15-dev

Now, lets get Nginx while we’re at it! The version that is in Ubuntu’s repos is old and not cool, so we’ll manually grab a newer version:

wget http://technokracy.net/nginx/nginx_0.5.32~grrr-1_i386.deb
sudo dpkg nginx_*

And now Nginx is running and listening to port 8000.. we’ll change that later though.
nginx setup

Now, here is where it gets sticky. You are going to create and edit /etc/nginx/sites-enabled/mysite

sudo nano /etc/nginx/sites-enabled/mysite

and put in it the following, changing directories and whatnot to what is appropriate for you.

server {

listen 80;
server_name yourdomain.com;
rewrite ^/(.*) http://www.yourdomain.com permanent;

}

server {

listen 80;
server_name www.yourdomain.com;

access_log /home/joyerhot/public_html/yourdomain.com/logs/access.log;
error_log /home/joyerhot/public_html/yourdomain.com/logs/error.log;

if (!-e $request_filename) {
rewrite ^([_0-9a-zA-Z-]+)?(/wp-.*) $2 last;
rewrite ^([_0-9a-zA-Z-]+)?(/.*\.php)$ $2 last;
rewrite ^ /index.php last;
}

location / {

root /home/joyerhot/public_html/johnyerhot.com/wordpress;
index index.html index.php index.htm;

}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_FILENAME /home/joyerhot/public_html/yourdomain.com/wordpress/$fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
}
}

editing_nginx

WHEW! There is A LOT of stuff there! This will setup a Virtual Host for your PHP site, and enable rewrites (for pretty urls). Here is what you need to change in yours though:

1. the server_name is whatever your domain is.
2. rewrite ^/(.*) http://www.whatever_goes_here.com permanent;
3. access_log directory
4. error_log directory
5. root directory
6. fastcgi_param SCRIPT_FILENAME /path/to/your/php/app/root/$fastcgi_script_name;
7. MAKE SURE YOU’RE LISTENING TO PORT 80.

Ok, now we’ll get PHP5 working as a fcgi process. To do this, we need to grab some of Lighttpd.

wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2
tar -xvjf lighttpd-*
cd lighttpd*
./configure
make

DO NOT “make install”. :)

sudo cp src/spawn-fcgi /usr/bin/spawn-fcgi
sudo nano /usr/bin/php-fastcgi

Now, add the following into php-fastcgi:

/usr/bin/spawn-fcgi -a 127.0.0.1 -p 9000 -C 2 -u www-data -f /usr/bin/php5-cgi

-C controls how many fcgi instances of PHP5 are spawned, so you can put however many are appropriate for you in there. I only needed 2.

sudo nano /etc/init.d/init-fastcgi

And add:

#!/bin/bash
PHP_SCRIPT=/usr/bin/php-fastcgi
RETVAL=0
case “$1″ in
start)
$PHP_SCRIPT
RETVAL=$?
;;
stop)
killall -9 php
RETVAL=$?
;;
restart)
killall -9 php
$PHP_SCRIPT
RETVAL=$?
;;
*)
echo “Usage: php-fastcgi {start|stop|restart}”
exit 1
;;
esac
exit $RETVAL

Do some permission magic:

sudo chmod 755 /etc/init.d/init-fastcgi
sudo chmod 755 /usr/bin/php-fastcgi

Now, you can make sure everything is working by:

/etc/init.d/init-fastcgi start
top #then shift + M

You should see a couple of PHP5 fcgi processes! YAY!
Finally, we want them to start on reboots and whatnot so…

update-rc.d init-fastcgi defaults

Ok, now lets restart Nginx and see if things are working!

/etc/init.d/nginx restart

You should be able to hit your Php app now!

Now for Ruby and friends!

sudo apt-get install libmysql-ruby1.8 ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8 libreadline-ruby1.8 libruby1.8 libopenssl-ruby irb1.8 libdbd-mysql-perl libdbi-perl libmysql-ruby1.8 libmysqlclient15-dev libmysqlclient15off libnet-daemon-perl libopenssl-ruby libopenssl-ruby1.8 libplrpc-perl libreadline-ruby1.8 libruby1.8 mysql-client mysql-client-5.0 mysql-common mysql-server mysql-server-5.0 rdoc1.8 ri1.8 ruby1.8 ruby1.8-dev zlib1g-dev

And you might need to create some symlinks (I had to), thanks Vince Wadhwanl.

sudo ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby
sudo ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc
sudo ln -s /usr/bin/ri1.8 /usr/local/bin/ri
sudo ln -s /usr/bin/irb1.8 /usr/local/bin/irb

Make sure things are working ok…

ruby -v

You should get that you have 1.8.6 installed.Ok, now lets install RubyGems.

wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
tar xvzf rubygems-1.0.1.tgz
cd rubygems-1.0.1
sudo ruby setup.rb
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem

And finally for Rails and Mongrel:

sudo gem install rails
sudo gem install mongrel mongrel_cluster

You can now set up a mongrel_cluster or whatever you need. There’s plenty of tutorials out there for that, so Google is your friend. The rest will assume you have your Rails app up and running on port 3000.
We now have to set up a site profile for Nginx for your mongrel_cluster.

sudo nano /etc/nginx/sites-enabled/rails_app

And add to it, once again changing the relevant parts….

upstream domain1 {
server 127.0.0.1:3000;
}

server {

listen 80;
server_name www.your2nddomain.com;
rewrite ^/(.*) http://subdomain.your2nddomain.com permanent;

}

server {

listen 80;
server_name subdomain.your2nddomain.com;

access_log /home/joyerhot/public_html/your2nddomain.com/logs/access.log;
error_log /home/joyerhot/public_html/your2nddomain.com/logs/error.log;

location / {

root /home/joyerhot/public_html/your2nddomain.com/public/RAILS_ROOT/public/;
index index.html;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect false;

if (-f $request_filename/index.html) {
rewrite (.*) $1/index.html break;
}

if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}

if (!-f $request_filename) {
proxy_pass http://domain1;
break;
}
}

}

}

Again, heres what you need to change:
1. server_name should be the domain
2. In this case the “rewrite” line will point to subdomain.your2nddomain.com, you should prolly just put your domain here(www.your2nddomain.com).
3. access_log directory
4. error_log directory
5. root directory (pointing to RAILS_ROOT/public)

That’s it! You should now restart Nginx (/etc/init.d/nginx restart) and when you should have everything working! You’ve got Virtual Hosts setup with PHP running as Fast CGI on one domain and Ruby On Rails on the other!

I’m sure I’ve got a typo or two in here and a mistake here and there, please leave a message if this worked or didn’t work for you. Remember, this is the process I went through getting everything working on my Slicehost Slice and another server at work, both running Ubuntu 7.10.

PS. Here is one little nugget of info that may be helpful – Your MySQL socket file location (for your RAILS_ROOT/config/database.yml file) is located at /var/run/mysqld/mysqld.sock. I didn’t realize this at first. :)

Debugging Backgroundrb

Monday, February 11th, 2008

Backgroundrb is a great tool for getting cron like functionality in your Rails app, but it can be a pain if its not working correctly. I’m not going to go into how to get backgroundrb working, its rubyforge page has decent instructions for that, but what I will share are some tips on debugging a non-working worker.

Obviously check your RAILS_ROOT/log and check backgroundrb.log, backgroundrb_server.log, and backgroundrb_debug.log. I’ve found backgroundrb.log most useful, but many times I’ll restart backgroundrb by

script/backgroundrb stop

script/backgroundrb start #restart doesn’t work for me

and nothing will happen. Check your backgroundrb_server.log file and you may see that the ‘address is already in use”.

Type “top” into your command line (on a Linux system) and then press shift+M. You should get a nice list of services running and you’ll more than likely have a couple of rubys on there. For some reason backgroundrb doesn’t stop all the time when you tell it to and I’ve had to kill -9 the rubys and then restart backgroundrb and mongrel.

You may also notice that the backgroundrb ruby instance uses ALOT of memory! Make sure you have

:environment: production

in your backgroundrb.yml file in RAILS_ROOT/config. Otherwise backgroundrb is working in development mode.

Finally, to debug an syntax or other error try this.

1. open 2 command lines.

2. start your rails app with ruby script/server. Make sure to start it in development mode and not using mongrel_rails.

3. start backgroundrb in the other terminal.

4. watch the first terminal and you can see what background rb is doing and where exactly it gets messed up.

You can do this by monitoring the log file or tailing it, but I like this better and it provides you with more info.

Hope this helps any of you out.

Royner & Co.

Sunday, February 10th, 2008

Well, after a little bit of down time and some re-adjusting my slice at slicehost we’re back and I’ve gotten Royner up and running at royner.johnyerhot.com.

What is Royner?

Good question. Well, you sign up with either a Google Talk (Gtalk) or Jabber instant message screen name, give Royner a url for a RSS/ATOM feed (i.e. Feedburner)

Please give me some feed back on Royner. I didn’t put tons and tons of hours into Royner (the majority on Backgroundrb, which is working beautifully now), but I’d like to keep it alive if the demand is there.

There are some to-do’s left:

Set Royner up so that you can respond to IM’s with “OK” or something and Royner will quit looking for that key word.

Hasta pasta

Moved Hosts

Friday, February 1st, 2008

Well I’ve finished consolodating my two hosting accounts, one at Godaddy (cheap Linux hosting) and my shared Rails hosting at OcsSolutions (which was very good for shared Rails host) to a 512 mb Ubuntu 7.10 ’slice’ at Slicehost.  Was gonna go straight Debian, but figured what the hell.  For any of you who don’t know, a ’slice’ is pretty much get VPS hosting account.  This is my first VPS.  I set up Apache/Rails proxy servers and your run of the mill LAMP servers for customers/clients and at work all the time, but this is MY first server that is all mine to mess with.

As mentioned, I decided to give Nginx a try. I have to admit, initial set up is very easy.  I first setup the Rails enviornment and had Nginx proxy to two mongrels for each app.  (I’m running two apps for four total Mongrels).  No problem.

Next, I had to get php working.  Not so easy.

I ended up looking for help and followed this, setting up php5 to run as fast-cgi.  Ugh.  To make things worse, I had to move over the Wordpress install from Godaddy.  I assummed it would be easy as pie, and it was to an extent, but I ran into trouble with the way the virtual hosts configs nginx uses handle subdomains and .. long story short I had redirect loop happening and it just sucked.

I’m blabering on and on here, but in the end I’ve got everything up and running.  We’ll see how Nginx works in the long term, but right now it seems pretty snappy and the memory footprint is much smaller than Apache.

So, Royner and another project of mine have a place to live and I’m just gonna do some final tweaks this weekend and then you can check them out.

Weekly Roundup

Monday, January 28th, 2008

Heres the week in review:

-Linux Mint home networking is now done and working beautifully. Had some trouble getting the onboard Via Chrome9 Video working correctly, but I followed the instructions here(compiling it from source) and everything was great. I wanted to use the OpenChrome driver instead of the VESA driver because the OpenChrome one has support for XvMC which accellerates all kinds of video (i.e. Xvid, mpeg4…). Only thing left to do is get a static IP so I can get to my machine from the outside world.

-Royner – I haven’t had much time to work on ironing out any of the kinks before a first release. Hopefully this week will be different. Thinking about Slicehost to host it, as I’ve heard great things about them. I’m also going to try out deprec for deployment, using Nginx instead of Apache. Should be interesting.


Other than thats, its another work week. See ya’ll.

Royner – New Project

Thursday, January 24th, 2008

I’m happy to say that I’m almost done with my newest project – Royner. It doesn’t live anywhere yet, just on my local machine, but heres what it is:

Really simple – you give Royner a feed (Atom, RSS, XML) and Royner monitors that feed for some keywords that you define. So, lets say you want to monitor the Slickdeals.net RSS feed of deals for “FAR” (Free after rebate). When something is posted that is free after rebate, Royner will send you an IM letting you know of the deal and the url. You could use it for whatever you want. Monitor blogs, Digg, hell, even your email.

Currently, only Jabber/GTalk is supported since it is the easiest to integrate.

Pretty neat eh?
Heres some screenies. Hopefully it will be ready for the public in a week or so.
royner1royner2royner3

The one thing I’ll say is this though – I used backgroundrb to monitor the feeds. Now, once I got it running ok, it worked wonderfully, but I swear it was the most difficult Rails plugin to get working. I spent an ENTIRE day trying to get it to work. In the end, I uninstalled and reinstalled the plugin and that did it. The first time I installed, for some reason an old version was installed and none of the documentation worked. Blah.

Rake Leaves

Thursday, January 10th, 2008

Yeah, I came up with this little Rails rake task called ‘rake leaves’. It will just create a text file that has all your tables, column names, and their data types. I’ve always thought it would be handy to have this so you don’t need to switch to MySQL and spend the time to look it up.

Anyways, if you want to download it go for it. All you need to do is put leaves.rake in /lib/tasks and then from the console

$rake leaves

You’ll get a file called table_structure_whatever_the_date_is.txt.
You can download if right here.