Problem

We want to be able to allow users to upload files via FTP, and we want them to use the same accounts that they use on our Rails based site. How can we authenticate an FTP server against our Rails app?

Solution

If we use pure-ftp as a server, we can create a custom authentication module that the FTP service will use to validate logins.

Details on creating a custom authentication module for pure-ftp are found here: http://download.pureftpd.org/pub/pure-ftpd/doc/README.Authentication-Modules
The Gist: All we need to do is create a script that will use incoming environment variables and echo a handful of directives.

At minimum, we just need to handle the AUTHD_ACCOUNT and AUTHD_PASSWORD variables. We could do this with a simple rake task, being that rake already has access to environment variables. The downside to that solution is that rake is going to load up our Rails app every time someone tries to log in via FTP. The result is a slight delay in the login process and a spike in memory usage. Think of one of those FTP bots coming along and trying out as many logins as possible and imagine rake loading our Rails app into memory that many times at once.

Of course, we could always use a background process to load up one Rails instance once and then respond to it. Setting up a background processor can be a complicated process if you don’t already have one going for your app.

Since all we really need to do is return a blob of text, why not just hit the Rails app itself, via HTTP? We can create a controller action that returns plain text in the format required by pure-ftp and then use wget to grab it and echo it. With this solution, we’re hitting the web server, which is already loaded into memory, so there is no delay and no memory spike during login (well, only enough to run wget).

Let’s look at the minimum response we need to tell pure-ftp the user is valid:

auth_ok:1
uid:1000
gid:1000
dir:/path/to/rails/app/public/files/userid
end

The first line is either 1 or 0. 1 means the user is valid, 0 means they are invalid (bad login or password). According to the docs, pure-ftp will also accept -1, which means that the login was correct but the password was not. Since authentication plugins for Rails typically don’t distinguish between bad login/password and just bad password (why let the hackers know they guessed one half of the combo?) I’m going to return only 1 or 0.

uid and gid are the system level user and group that you want to use to run the service under. Pure-ftp is giving you the chance to increase security by assigning a unix-level user account to the login here, but since we’re not using unix-level accounts, we can just specify the user and group that has rights to the directory where the files will reside. If you’ve only created one user on your system and you’re deploying your Rails app under that user, chances are pretty good it’s going to be uid 1000 and gid 1000. You can find out by checking /etc/passwd:

jason@jason-1501:~$ grep jason /etc/passwd
jason:x:1000:1000:Jason LaPier,,,:/home/jason:/bin/bash

The dir directive is the directory we want the FTP user to start out in. In this example, we’re saying that if a user has id ‘99’, they get put into a directory at “RAILS_ROOT/public/files/99”.

Let’s take a look at the controller action that will spit out this block of text (I put this in a Users controller, but you can put it wherever you want):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

  # somewhere, we have to define SYSTEM_USER, SYSTEM_GROUP, and FTP_ROOT
  # I always stick these kinds of constants in another module so I can change them 
  # depending on the RAILS_ENV
  FTP_ROOT = "#{RAILS_ROOT}/public/files"
  SYSTEM_USER = '1000'
  SYSTEM_GROUP = '1000'

  # here's the action - we're expecting params: login, password
  def ftp_auth
    user = User.authenticate(params[:login], params[:password])
    if user
      user_root = File.join FTP_ROOT, user.id.to_s
      # mkdir_p is like "mkdir -p" - it creates the directory and parents as necessary,
      # doing nothing if they already exist
      FileUtils.mkdir_p user_root
      # chown_R is like "chown -R" - make sure the system user owns the files directory
      FileUtils.chown_R SYSTEM_USER, SYSTEM_GROUP, FTP_ROOT
      # build the output for a valid user. Be careful not to get whitespace at the
      # beginning of each line, or pure-ftp will ignore the output
      output = <<-END
auth_ok:1
uid:#{SYSTEM_USER}
gid:#{SYSTEM_GROUP}
dir:#{user_files_root(user)}
end
      END
    else
      # invalid user, so all we need is for auth_ok to be 0
      output = "auth_ok:0\n" + "end\n"
    end
    render :text => output
  end

If you are using sessions or other before_filters, don’t forget to take those out for this action. Since we’re not really making this connection with a web browser, most of that stuff is unnecessary. If you want to send the request via post, you may also need to take out forgery protection.

1
2
3
4
5

  # at the top of your controller, of course
  skip_before_filter :my_ridiculous_sidebar_loader, :only => :ftp_auth
  skip_before_filter :login_required, :only => :ftp_auth
  skip_before_filter :verify_authenticity_token, :only => :ftp_auth
  session :off, :only => :ftp_auth

Now let’s point our browser at the action and see what happens. If we go to http://localhost:3000/users/ftp_auth, we should get:

auth_ok:0 end

Since HTML ignores whitespace, you’ll have to view the page source to verify that the whitespace is correct. Better yet, whip up a controller test!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

  # stick this in your controller's Test::Unit test

  # try with no params and you should get a failure
  def test_ftp_auth_fail
    out = post :ftp_auth
    assert_not_nil out.body
    assert_equal "auth_ok:0\nend\n", out.body
  end

  # this is a test user that came with the restful_authentication plugin
  def test_ftp_auth_valid
    out = post :ftp_auth, :login => 'aaron', :password => 'test'
    assert_not_nil out.body
    assert_equal "auth_ok:1\nuid:1000\ngid:1000\n" + 
      "dir:#{RAILS_ROOT}/public/files/#{users(:aaron).id}\nend\n", out.body
  end

Alright, we’re halfway there! Now we just need to set up pure-ftp to use a script for custom authentication.

Getting pure-ftp:
Most package installers will have it, or you can download it and compile it yourself fairly easily.

Setting up pure-ftp:
Pure-ftp doesn’t use a configuration file in it’s natural state – only command line arguments. Different Linux distributions come with different “wrappers” which parse configuration files and run pure-ftp with the appropriate arguments. Let’s just run pure-ftp from the command line for now.

First, we need a script that pure-ftp can call. This script just needs to hit the Rails app via HTTP and return the response, so we could whip up something in our favorite scripting language. For now, in the name of simplicity, let’s just shell-script a wget call:

#!/bin/sh
wget --post-data="login=$AUTHD_ACCOUNT&password=$AUTHD_PASSWORD" -qO- http://localhost:3000/users/ftp_auth | cat

Save this file (for example, as “ftp-auth.sh”) and make it executable (“chmod +x ftp-auth.sh”). If you run it now, you should get the “auth_ok:0” text (since the environment variables are not set). Start up pure-ftp’s auth daemon by running:

pure-authd -s /var/run/ftpd.sock -r /path/to/ftp-auth.sh -B

Next, start up the ftp server (remember to kill it first if it’s already running – especially if you installed pure-ftp with a package manager). Here’s an example with command line options:

pure-ftpd -lextauth:/var/run/ftpd.sock -X -A -F /pathto/banner -E -B -c 20 -C 2 -n 50:100

I’m using the following options in this example:
-A —chrooteveryone (very important! this will make the user’s home directory the virtual “root”, so the user can’t go up the directory tree and into your system)
-B —daemonize
-C —maxclientsperip (most ftp clients will try to use as many connections as possible, so you might want to limit this)
-c —maxclientsnumber (total number of connections you want to allow)
-d —verboselog (leave this out for production usage)
-E —noanonymous (no anonymous ftp users allowed)
-F —fortunesfile (optional – provide a custom banner – just point at a text file)
-n —quota (virtual quota – [number of files max:max file size]. 50:100 means 50 files, max of a single file is 100 Mb)
-X —prohibitdotfilesread (don’t show ‘hidden’ files or folders)

Now point your favorite FTP client at your server and try it out. You should be able to log in with a user in your Rails app, no problemo!

Craig Ambrose writes about how he came up with a way to receive mail RESTfully in Rails. Basically, he makes the point that the method for processing email described in the Rails wiki is a little scary, as it can lead to the loading of the full Rails stack on every email you want to process. Craig speculates that this method could be problematic in terms of memory usage, and I tend to agree with him. He comes up with a pretty clever hack to basically redirect each incoming email at his actual running web application by grabbing it from postfix and posting it to his site via the use of Ruby’s net/http library. After that, it’s up to the Rails controller to handle the request.

While I like the cleverness of the solution, I can’t help but feel like it’s somehow asking for trouble. As some comments on the post point out, there doesn’t seem to be an elegant way to handle the mail if the web server happens to be down. Another concern is hammering the web server with too many requests in the case of multiple emails or text messages coming in, as each incoming email takes a up an http connection and occupies a mongrel server (or whatever) for however long it is needed (it’s no worse than submitting form data, but I think the concern is that there is potential for more volume).

I’ve been thinking about a way to handle incoming mail myself, and here’s the solution I’ve come up with. I have not had time to implement it, and maybe it’s been done before (please let me know if it has and has been documented).

1. Postfix accepts incoming mail and stores the mail as a file somewhere on the server. A simple procmail recipe would probably be useful here to identify where to store incoming mail depending on it’s address, contents, day of the week, or whatever.
2. A backgroundrb worker hangs around and processes the mail files on a periodic basis, by reading files in a specified directory one at a time, parsing each message with TMail, and doing any other Rails stuff you need it to do with each message. Remove each file after the message has been processed.

Backgroundrb has a great built-in scheduling system, so there’s no need to think cron in this setup. Backgroundrb will load the whole rails stack once. The workers sit in a separate process. Backgroundrb won’t run itself over: if you set the schedule to process mail every 2 minutes and the mail takes more than 2 minutes to process, backgroundrb will queue it’s jobs up. In this scenario, a worker is just looking in one directory, so once all the mail is processed, it gets deleted. If no new mail comes in by the next time the worker looks at the directory, it can see there’s nothing to do and go back to sleep. Knowing the expected volume of email coming in is important, but I’m willing to bet in most cases you could ask backgroundrb to run your mail-processing worker once a minute.

This method doesn’t get to claim the label RESTful, but to be honest I don’t think regurgitating incoming mail out to your web server is really in the spirit of “REST” either. If you stay true to the skinny controller, fat model mantra, handling email via model methods (all accessible via backgroundrb) or even module/other non-model class methods in a backgroundrb worker should be as easy as (if not easier than) doing so from within a controller.

The other day I wanted to resize a bunch of images using ImageMagick’s convert command-line utility. I wanted to both resize and change their names from image_name.png to image_name-sm.png, but I couldn’t figure out how to guide ‘convert’ to use part of the original name in the new name. The next obvious tactic would be to use some common shell utilities like ‘for’ and ‘do’. My shell-fu is rusty however, and I still couldn’t get things quite the way I wanted them. Then I think to myself, this would be much easier to do with Ruby.

First, I use ‘puts’ to see if the syntax will come out right:

1
2
3
4
5
6

irb(main):017:0> Dir['*.png'].each { |f| puts "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }
convert character_creation.png -resize 400x400 character_creation-sm.png
convert battle-mountain_pass.png -resize 400x400 battle-mountain_pass-sm.png
convert loot-drag_and_drop.png -resize 400x400 loot-drag_and_drop-sm.png
convert market.png -resize 400x400 market-sm.png
...(etc)...

Then I just run the same line with system instead of puts:

Dir['*.png'].each { |f| system "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }

With a little practice, this can be done on the command line - who needs shell scripting?

1
2

# run from the shell prompt:
ruby -e 'Dir["*.png"].each { |f| system "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }'

One of my servers has a registration page that sends an activation code via email, and if someone signs up with a yahoo account, my logs were showing:

The help page is decidedly unhelpful (I filled out a form and got an email back from yahoo that pointed me to the same page that the error message points to). After some googling, it seems to be the best way to get around this yahoo deferring problem is to set up DomainKeys (notice there is no mention of DomainKeys on that yahoo ‘help’ page). Now it seems there are several different ways of implementing DomainKeys or one of the alternative methods of signing outgoing mail. In the end, I went with DKIM Proxy.

DKIM Proxy is written in perl. It’s designed to sit on your mail server, open a couple ports, and let your mail server pass messages in and out of it. On the outgoing side, it applies a DomainKey signature (actually two – it applies both the Domain Keys Identified Mail standard and the older Yahoo! DomainKeys standard). On the incoming side, it reads signatures and verifies their integrity.

As the DKIM Proxy page notes, if you’re already using something like SpamAssassin, you’ve already got DKIM checking on your incoming mail. If you want to be able to manage spam, I’d highly recommend using SpamAssassin.

I only needed DKIM Proxy for outgoing mail. The DKIM Proxy page has fairly decent instructions, but I ran into a few snags, so hopefully this will help someone.

Before you even get started: some things you need to know.

1. You need to have perl on your system (I don’t know of any distros that don’t come with perl) and you need to be able to install perl modules.
2. You’ll need the SSL dev packages for your distro installed for these perl modules to install correctly (for example: apt-get install libssl-dev )
3. You need to be able to edit your mail server’s configuration file – if you’re using postfix, there’s an example provided.
4. You need to be able to add a sub-domain to your DNS listing for your domain.

First up is perl. There are a bunch of modules you need to get the Mail::DKIM module installed, and then you can install DKIM Proxy. The required modules are listed on the DKIM Proxy site – here is the easy way to get them installed (make sure you are superuser/root):

Note 1: If you’re using Debian or Ubuntu, you should be able to skip this step by installing the libmail-dkim-perl and libnet-server-perl packages. I believe that will get your perl install all the necessary modules and you can go on to installing DKIM-Proxy.

Note 2: If you’ve never run CPAN before, it’s going to ask you to do a manual configuration right off the bat – I usually say ‘no’ to that question and let it attempt autoconfiguration. If you don’t need to set up any proxies or anything, autoconfigure should work.

Note 3: Some of these modules have their own dependencies. If it asks you: “Shall I follow them and prepend them to the queue of modules we are processing right now?” Just hit Enter to take the default answer of [Yes].

perl -MCPAN -e'CPAN::Shell->install("Crypt::OpenSSL::RSA")'
perl -MCPAN -e'CPAN::Shell->install("Digest::SHA")'
perl -MCPAN -e'CPAN::Shell->install("Digest::SHA1")'
perl -MCPAN -e'CPAN::Shell->install("Error")'
perl -MCPAN -e'CPAN::Shell->install("Mail::Address")'
perl -MCPAN -e'CPAN::Shell->install("MIME::Base64")'
perl -MCPAN -e'CPAN::Shell->install("Net::DNS")'
perl -MCPAN -e'CPAN::Shell->install("Net::Server")'

Once your perl modules are installed, you can install dkimproxy. I won’t go into detail – this part is pretty straightforward (follow the instructions on the site).

If you’ve got dkimproxy installed now, the instructions tell you how to generate a private/public key pair and then set up your DNS record. You have to put the public key in the actual DNS record, and this means removing the line breaks and white space.

Before you go any further, reload bind/named (on most systems: /etc/init.d/named restart or /etc/init.d/bind9 restart) and make sure you see your big long public key when you do the following command:

dig selector1._domainkey.my-domain-name.com TXT

If you don’t see your public key there in a big long string, then you did something wrong in DNS. Go back and fix it before you try anything else – trust me. You’ll just end up in circles if your DNS is wrong. I had misspelled something and wasted about 20 minutes trying to figure out what was wrong with DKIM Proxy when it was my DNS entry that was the problem all along.

At this point, if you haven’t already, you should probably create a user and group called ‘dkfilter’ (or something similar).

The instructions give you an example of how the ‘dkimproxy.out’ script could be run. There is also a file in the root of the dkimproxy archive called sample-dkim-init-script.sh. This script is the easiest way to get dkimproxy running as a daemon. Edit the vars at the top of the script to meet your needs – if you used the default selector name of selector1 and the user name dkfilter you probably don’t need to change anything. I changed the domain line to read: DOMAIN=my-domain1.com,my-domain2.com,my-domain3.com

(note: if you are doing this for multiple domains, I hope you remembered to add the TXT DNS record to each domain’s zone file – and verify with dig!)

If the init script works good, copy it to /etc/init.d or wherever your init scripts live, and set it up to start at boot time (distro-dependent; it’s something I do rarely enough that I always have to look it up).

For the last part, you need to set up your mail server to direct it’s outgoing messages at dkimproxy for signing before they head out. Hopefully, you’re using postifx, because there is a cut-and-paste example for the postfix master.cf on the DKIM Proxy site. Don’t forget to do a postfix reload.

Once postfix is set up, they have a couple of test mail addresses you can use. Watch your maillog to see if things are working right. I still got a few delays from Yahoo at first, but now they seem to be flowing normally.

I just got my slice at Slicehost and I went with the Ubuntu Dapper install, following this quick guide to getting apache, mongrel, and rails installed. It worked out ok for the most part – my only problem with these “cut and paste” type guides is it doesn’t explain what you’re actually installing. Thankfully, the author Chris links to the three different articles he collected the steps from, and for someone who’s already aware of the concepts, this how-to makes going through the motions that much easier. I got a newer version of apache than is shown – he has 2.2.2 and 2.2.4 is the newest stable release.

Once I got dapper set up with rails, mongrel and apache, I started to use the deprec gem following the guide posted on Slicehost’s wiki. Once I started going through the steps, I realized that these deprec recipes were way more automated than I wanted right now, and I had to cancel halfway through. I was expecting a per-app kind of automation, but I was in copy-and-paste mode and ran the “cap install_rails_stack” line and realized it was trying to do all the stuff I had just set up. The author of the deprec gem warns that it’s really a set up scripts set up to work for him, and use at your own risk, blah blah blah. It may work for some, but like I said, I found it a little too automated.

I went back and grabbed the deploy.rb I’d used before from Slingshot’s Capistrano support page. This script has worked pretty good for me in the past, but I had to make a few alterations. The script, called apache_2_2_mongrel_deploy.rb, had been updated since the last time I tried it, but it’s still a tad buggy.

One problem I had right off the bat was the reference to a variable called “crontab_email”. This is for a special crontab-related task that I don’t need right at the moment, but since this variable was referenced and not set, it caused an error.

The next major problem is that there’s a place to set the database user and password – so I assumed that the database.yml file that gets generated in setup would be using that user and password. Oddly enough, it actually only uses the username, not the password. Took me a few minutes to realize my initial migrations were bombing because there was no password referenced in database.yml.

Thirdly, the “mongrel_conf” variable that gets set for the mongrel recipes to use is pointing to the wrong directory. I changed the line to: set :mongrel_conf, ”/etc/mongrel_cluster/#{application}.yml”

Lastly, and this is the big one – when trying to start the mongrel cluster for the first time, I got an error from mongrel about the “mongrel_prefix” needing to start with a slash (/) but not end with one. Since the line was: set :mongrel_prefix, ”/usr/local/bin/” I simply took the last slash off the line and my error message went away. Happy as can be, I went ahead and got apache going and fired up my mongrel servers, pointed my web browser at my new server and got:

NOT FOUND

This happened to me the last time I used an earlier version of this script, and for the life of me, I couldn’t remember what the problem was. The apache logs were sparse. The mongrel log offered nothing but telling me when the servers were started or stopped. The production log was blank. After an hour of poking and prodding, I finally remembered what it was – the mongrel config file that had mongrel_prefix set to /usr/local/bin. My mongrel bin directory should have been /usr/bin (actually, I just blanked it out and that worked just fine).

A very frustrating experience. Mongrel is great and you can’t go wrong with this stack for Rails, but man – a better error message would be nice. Something in the log. Anything….

Lately I’ve been playing with wmii, the window manager. The ruby-addons for wmii by mfp open up the base wmii quite a bit. Once I got used to this dynamic window manager, I can’t imagine going back to WIMP.

I started to poke around in the ruby-wmii config files, because I was curious about how whenever I opened Firefox, the client was automatically tagged with “web”. Since I was constantly tagging my other apps with names using MODKEY-ctrl-T, I decided I wanted some of them to auto-tag like the browsers do.

After initially just copying the existing code that tags the browsers, I finally worked out a way to get it to tag things the way I want them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

 # *** excerpt of wmiirc-config.rb ***

 # first, I commented out this orginal tagger

 # {{{ Tag all browser instances as 'web' in addition to the current tag
 # browsers = %w[Firefox Konqueror]
 # browser_re = /^#{browsers.join("|")}/
 # on_createclient(condition{|c| browser_re =~ read("/client/#{c}/class")}) do |cid|
 #   write("/client/#{cid}/tags", normalize(read("/client/#{cid}/tags") + "+web"))
 # end

  #############################################
  # here's the stuff I added
        
  # these are tags for particular apps that I want to have 
  # new tags - note, they won't get appended to their current tag

  tag_for_apps = { 
    "irc" => "Xchat", 
    "gaim" => "Gaim", 
    "web" => "Firefox",
    "4-jedit" => "jedit"
  }
  
  # these are terminals that I've given specific titles to
  # so they can be tagged
  tag_for_named_terms = {
    "2-consoles" => "console:",
    "3-logs" => "log:",
    "1-terms" => "term:"
  }
    
  # now when a new client comes up, check for an autotag
  on_createclient do |cid|
    LOGGER.info "checking for autotag on class: #{read("/client/#{cid}/class")} " + 
        "and name: #{read("/client/#{cid}/name")} "
    # if this client is a terminal, check the 'name'
    if /terminal/ =~ read("/client/#{cid}/class")
      tag_for_named_terms.each do |tfn|
        names_re = /#{tfn[1]}/
        if names_re =~ read("/client/#{cid}/name")
          write("/client/#{cid}/tags", tfn[0])
          LOGGER.info "  ... autotag'd #{tfn[1]} with #{tfn[0]}!"
        end
      end
    # not a terminal, so go by the 'class'
    else
      tag_for_apps.each do |tfa|
        apps_re = /#{tfa[1]}/
        if apps_re =~ read("/client/#{cid}/class")
          write("/client/#{cid}/tags", tfa[0])
          LOGGER.info "  ... autotag'd #{tfa[1]} with #{tfa[0]}!"
        end
      end
    end
  end

Notice, for terminals, I check for certain terminal window titles. I changed my gnome-terminal script a bit, so now it looks like this:

#!/bin/sh

# set working dir
wd="/home/jason/dev/rails_app" 

gnome-terminal --working-directory=$wd --hide-menubar -e "mongrel_rails start" -t "log: web server" \
--window --working-directory=$wd --hide-menubar -e "tail -f log/development.log" -t "log: dev log" \
--window --working-directory=$wd --hide-menubar -e "script/console" -t "console: rails irb" \
--window --working-directory=$wd --hide-menubar -e "mysql -u root --password=mysecret myapp_development" -t "console: mysql" \
--window --working-directory=$wd --hide-menubar -t "term: approot"

For some of my tag names, I stick a number in the front – that’s just to force them into order so I can easily MODKEY-n over to them.

I’d like to work the auto-tagger out into a plugin, but after poking around some of the existing plugins I gave up. I have no idea how to check for on_createclient from within a plugin, and I’m pretty confused about how those long-winded option get passed to plugins. If anyone has any hints, I’d love to hear them.

I do most of my development in Ubuntu. The gnome-terminal app is one of my favorite terminals, mainly I think because of it’s tabbed structure. When I start working on Rails stuff, I usually end up opening a terminal and about 4 or 5 tabs. Everytime I do it, I think to myself “I should just script this to open – I wonder if I can tell gnome-terminal to open tabs with certain scripts loaded in each tab…”

Turns out, it’s not really that difficult. Here’s my shell script:

#!/bin/sh

# set working dir
wd="/home/jason/dev/rails_app" 

gnome-terminal --geometry=179x59+0+25 --working-directory=$wd  -e "mongrel_rails start" -t "web server" \
--tab --working-directory=$wd -e "tail -f log/development.log" -t "dev log" \
--tab --working-directory=$wd -e "script/console" -t "irb console" \
--tab --working-directory=$wd -e "mysql -u root --password=mysecret rails_app_development" -t "mysql" \
--tab

Ok, now I should get some stuff done…

There are a couple of previous walkthroughs on getting Rails working on Debian, so this is really just one of many.