Craig Ambrose writes about how he came up with a way to receive mail RESTfully in Rails. Basically, he makes the point that the method for processing email described in the Rails wiki is a little scary, as it can lead to the loading of the full Rails stack on every email you want to process. Craig speculates that this method could be problematic in terms of memory usage, and I tend to agree with him. He comes up with a pretty clever hack to basically redirect each incoming email at his actual running web application by grabbing it from postfix and posting it to his site via the use of Ruby’s net/http library. After that, it’s up to the Rails controller to handle the request.

While I like the cleverness of the solution, I can’t help but feel like it’s somehow asking for trouble. As some comments on the post point out, there doesn’t seem to be an elegant way to handle the mail if the web server happens to be down. Another concern is hammering the web server with too many requests in the case of multiple emails or text messages coming in, as each incoming email takes a up an http connection and occupies a mongrel server (or whatever) for however long it is needed (it’s no worse than submitting form data, but I think the concern is that there is potential for more volume).

I’ve been thinking about a way to handle incoming mail myself, and here’s the solution I’ve come up with. I have not had time to implement it, and maybe it’s been done before (please let me know if it has and has been documented).

1. Postfix accepts incoming mail and stores the mail as a file somewhere on the server. A simple procmail recipe would probably be useful here to identify where to store incoming mail depending on it’s address, contents, day of the week, or whatever.
2. A backgroundrb worker hangs around and processes the mail files on a periodic basis, by reading files in a specified directory one at a time, parsing each message with TMail, and doing any other Rails stuff you need it to do with each message. Remove each file after the message has been processed.

Backgroundrb has a great built-in scheduling system, so there’s no need to think cron in this setup. Backgroundrb will load the whole rails stack once. The workers sit in a separate process. Backgroundrb won’t run itself over: if you set the schedule to process mail every 2 minutes and the mail takes more than 2 minutes to process, backgroundrb will queue it’s jobs up. In this scenario, a worker is just looking in one directory, so once all the mail is processed, it gets deleted. If no new mail comes in by the next time the worker looks at the directory, it can see there’s nothing to do and go back to sleep. Knowing the expected volume of email coming in is important, but I’m willing to bet in most cases you could ask backgroundrb to run your mail-processing worker once a minute.

This method doesn’t get to claim the label RESTful, but to be honest I don’t think regurgitating incoming mail out to your web server is really in the spirit of “REST” either. If you stay true to the skinny controller, fat model mantra, handling email via model methods (all accessible via backgroundrb) or even module/other non-model class methods in a backgroundrb worker should be as easy as (if not easier than) doing so from within a controller.

The other day I wanted to resize a bunch of images using ImageMagick’s convert command-line utility. I wanted to both resize and change their names from image_name.png to image_name-sm.png, but I couldn’t figure out how to guide ‘convert’ to use part of the original name in the new name. The next obvious tactic would be to use some common shell utilities like ‘for’ and ‘do’. My shell-fu is rusty however, and I still couldn’t get things quite the way I wanted them. Then I think to myself, this would be much easier to do with Ruby.

First, I use ‘puts’ to see if the syntax will come out right:

1
2
3
4
5
6

irb(main):017:0> Dir['*.png'].each { |f| puts "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }
convert character_creation.png -resize 400x400 character_creation-sm.png
convert battle-mountain_pass.png -resize 400x400 battle-mountain_pass-sm.png
convert loot-drag_and_drop.png -resize 400x400 loot-drag_and_drop-sm.png
convert market.png -resize 400x400 market-sm.png
...(etc)...

Then I just run the same line with system instead of puts:

Dir['*.png'].each { |f| system "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }

With a little practice, this can be done on the command line - who needs shell scripting?

1
2

# run from the shell prompt:
ruby -e 'Dir["*.png"].each { |f| system "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }'

One of my servers has a registration page that sends an activation code via email, and if someone signs up with a yahoo account, my logs were showing:

The help page is decidedly unhelpful (I filled out a form and got an email back from yahoo that pointed me to the same page that the error message points to). After some googling, it seems to be the best way to get around this yahoo deferring problem is to set up DomainKeys (notice there is no mention of DomainKeys on that yahoo ‘help’ page). Now it seems there are several different ways of implementing DomainKeys or one of the alternative methods of signing outgoing mail. In the end, I went with DKIM Proxy.

DKIM Proxy is written in perl. It’s designed to sit on your mail server, open a couple ports, and let your mail server pass messages in and out of it. On the outgoing side, it applies a DomainKey signature (actually two – it applies both the Domain Keys Identified Mail standard and the older Yahoo! DomainKeys standard). On the incoming side, it reads signatures and verifies their integrity.

As the DKIM Proxy page notes, if you’re already using something like SpamAssassin, you’ve already got DKIM checking on your incoming mail. If you want to be able to manage spam, I’d highly recommend using SpamAssassin.

I only needed DKIM Proxy for outgoing mail. The DKIM Proxy page has fairly decent instructions, but I ran into a few snags, so hopefully this will help someone.

Before you even get started: some things you need to know.

1. You need to have perl on your system (I don’t know of any distros that don’t come with perl) and you need to be able to install perl modules.
2. You’ll need the SSL dev packages for your distro installed for these perl modules to install correctly (for example: apt-get install libssl-dev )
3. You need to be able to edit your mail server’s configuration file – if you’re using postfix, there’s an example provided.
4. You need to be able to add a sub-domain to your DNS listing for your domain.

First up is perl. There are a bunch of modules you need to get the Mail::DKIM module installed, and then you can install DKIM Proxy. The required modules are listed on the DKIM Proxy site – here is the easy way to get them installed (make sure you are superuser/root):

Note 1: If you’re using Debian or Ubuntu, you should be able to skip this step by installing the libmail-dkim-perl and libnet-server-perl packages. I believe that will get your perl install all the necessary modules and you can go on to installing DKIM-Proxy.

Note 2: If you’ve never run CPAN before, it’s going to ask you to do a manual configuration right off the bat – I usually say ‘no’ to that question and let it attempt autoconfiguration. If you don’t need to set up any proxies or anything, autoconfigure should work.

Note 3: Some of these modules have their own dependencies. If it asks you: “Shall I follow them and prepend them to the queue of modules we are processing right now?” Just hit Enter to take the default answer of [Yes].

perl -MCPAN -e'CPAN::Shell->install("Crypt::OpenSSL::RSA")'
perl -MCPAN -e'CPAN::Shell->install("Digest::SHA")'
perl -MCPAN -e'CPAN::Shell->install("Digest::SHA1")'
perl -MCPAN -e'CPAN::Shell->install("Error")'
perl -MCPAN -e'CPAN::Shell->install("Mail::Address")'
perl -MCPAN -e'CPAN::Shell->install("MIME::Base64")'
perl -MCPAN -e'CPAN::Shell->install("Net::DNS")'
perl -MCPAN -e'CPAN::Shell->install("Net::Server")'

Once your perl modules are installed, you can install dkimproxy. I won’t go into detail – this part is pretty straightforward (follow the instructions on the site).

If you’ve got dkimproxy installed now, the instructions tell you how to generate a private/public key pair and then set up your DNS record. You have to put the public key in the actual DNS record, and this means removing the line breaks and white space.

Before you go any further, reload bind/named (on most systems: /etc/init.d/named restart or /etc/init.d/bind9 restart) and make sure you see your big long public key when you do the following command:

dig selector1._domainkey.my-domain-name.com TXT

If you don’t see your public key there in a big long string, then you did something wrong in DNS. Go back and fix it before you try anything else – trust me. You’ll just end up in circles if your DNS is wrong. I had misspelled something and wasted about 20 minutes trying to figure out what was wrong with DKIM Proxy when it was my DNS entry that was the problem all along.

At this point, if you haven’t already, you should probably create a user and group called ‘dkfilter’ (or something similar).

The instructions give you an example of how the ‘dkimproxy.out’ script could be run. There is also a file in the root of the dkimproxy archive called sample-dkim-init-script.sh. This script is the easiest way to get dkimproxy running as a daemon. Edit the vars at the top of the script to meet your needs – if you used the default selector name of selector1 and the user name dkfilter you probably don’t need to change anything. I changed the domain line to read: DOMAIN=my-domain1.com,my-domain2.com,my-domain3.com

(note: if you are doing this for multiple domains, I hope you remembered to add the TXT DNS record to each domain’s zone file – and verify with dig!)

If the init script works good, copy it to /etc/init.d or wherever your init scripts live, and set it up to start at boot time (distro-dependent; it’s something I do rarely enough that I always have to look it up).

For the last part, you need to set up your mail server to direct it’s outgoing messages at dkimproxy for signing before they head out. Hopefully, you’re using postifx, because there is a cut-and-paste example for the postfix master.cf on the DKIM Proxy site. Don’t forget to do a postfix reload.

Once postfix is set up, they have a couple of test mail addresses you can use. Watch your maillog to see if things are working right. I still got a few delays from Yahoo at first, but now they seem to be flowing normally.

There are a couple of previous walkthroughs on getting Rails working on Debian, so this is really just one of many.