Craig Ambrose writes about how he came up with a way to receive mail RESTfully in Rails. Basically, he makes the point that the method for processing email described in the Rails wiki is a little scary, as it can lead to the loading of the full Rails stack on every email you want to process. Craig speculates that this method could be problematic in terms of memory usage, and I tend to agree with him. He comes up with a pretty clever hack to basically redirect each incoming email at his actual running web application by grabbing it from postfix and posting it to his site via the use of Ruby’s net/http library. After that, it’s up to the Rails controller to handle the request.

While I like the cleverness of the solution, I can’t help but feel like it’s somehow asking for trouble. As some comments on the post point out, there doesn’t seem to be an elegant way to handle the mail if the web server happens to be down. Another concern is hammering the web server with too many requests in the case of multiple emails or text messages coming in, as each incoming email takes a up an http connection and occupies a mongrel server (or whatever) for however long it is needed (it’s no worse than submitting form data, but I think the concern is that there is potential for more volume).

I’ve been thinking about a way to handle incoming mail myself, and here’s the solution I’ve come up with. I have not had time to implement it, and maybe it’s been done before (please let me know if it has and has been documented).

1. Postfix accepts incoming mail and stores the mail as a file somewhere on the server. A simple procmail recipe would probably be useful here to identify where to store incoming mail depending on it’s address, contents, day of the week, or whatever.
2. A backgroundrb worker hangs around and processes the mail files on a periodic basis, by reading files in a specified directory one at a time, parsing each message with TMail, and doing any other Rails stuff you need it to do with each message. Remove each file after the message has been processed.

Backgroundrb has a great built-in scheduling system, so there’s no need to think cron in this setup. Backgroundrb will load the whole rails stack once. The workers sit in a separate process. Backgroundrb won’t run itself over: if you set the schedule to process mail every 2 minutes and the mail takes more than 2 minutes to process, backgroundrb will queue it’s jobs up. In this scenario, a worker is just looking in one directory, so once all the mail is processed, it gets deleted. If no new mail comes in by the next time the worker looks at the directory, it can see there’s nothing to do and go back to sleep. Knowing the expected volume of email coming in is important, but I’m willing to bet in most cases you could ask backgroundrb to run your mail-processing worker once a minute.

This method doesn’t get to claim the label RESTful, but to be honest I don’t think regurgitating incoming mail out to your web server is really in the spirit of “REST” either. If you stay true to the skinny controller, fat model mantra, handling email via model methods (all accessible via backgroundrb) or even module/other non-model class methods in a backgroundrb worker should be as easy as (if not easier than) doing so from within a controller.

The other day I wanted to resize a bunch of images using ImageMagick’s convert command-line utility. I wanted to both resize and change their names from image_name.png to image_name-sm.png, but I couldn’t figure out how to guide ‘convert’ to use part of the original name in the new name. The next obvious tactic would be to use some common shell utilities like ‘for’ and ‘do’. My shell-fu is rusty however, and I still couldn’t get things quite the way I wanted them. Then I think to myself, this would be much easier to do with Ruby.

First, I use ‘puts’ to see if the syntax will come out right:

1
2
3
4
5
6

irb(main):017:0> Dir['*.png'].each { |f| puts "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }
convert character_creation.png -resize 400x400 character_creation-sm.png
convert battle-mountain_pass.png -resize 400x400 battle-mountain_pass-sm.png
convert loot-drag_and_drop.png -resize 400x400 loot-drag_and_drop-sm.png
convert market.png -resize 400x400 market-sm.png
...(etc)...

Then I just run the same line with system instead of puts:

Dir['*.png'].each { |f| system "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }

With a little practice, this can be done on the command line - who needs shell scripting?

1
2

# run from the shell prompt:
ruby -e 'Dir["*.png"].each { |f| system "convert #{f} -resize 400x400 #{f[0..-5]}-sm.png" }'

One of my servers has a registration page that sends an activation code via email, and if someone signs up with a yahoo account, my logs were showing:

The help page is decidedly unhelpful (I filled out a form and got an email back from yahoo that pointed me to the same page that the error message points to). After some googling, it seems to be the best way to get around this yahoo deferring problem is to set up DomainKeys (notice there is no mention of DomainKeys on that yahoo ‘help’ page). Now it seems there are several different ways of implementing DomainKeys or one of the alternative methods of signing outgoing mail. In the end, I went with DKIM Proxy.

DKIM Proxy is written in perl. It’s designed to sit on your mail server, open a couple ports, and let your mail server pass messages in and out of it. On the outgoing side, it applies a DomainKey signature (actually two – it applies both the Domain Keys Identified Mail standard and the older Yahoo! DomainKeys standard). On the incoming side, it reads signatures and verifies their integrity.

As the DKIM Proxy page notes, if you’re already using something like SpamAssassin, you’ve already got DKIM checking on your incoming mail. If you want to be able to manage spam, I’d highly recommend using SpamAssassin.

I only needed DKIM Proxy for outgoing mail. The DKIM Proxy page has fairly decent instructions, but I ran into a few snags, so hopefully this will help someone.

Before you even get started: some things you need to know.

1. You need to have perl on your system (I don’t know of any distros that don’t come with perl) and you need to be able to install perl modules.
2. You’ll need the SSL dev packages for your distro installed for these perl modules to install correctly (for example: apt-get install libssl-dev )
3. You need to be able to edit your mail server’s configuration file – if you’re using postfix, there’s an example provided.
4. You need to be able to add a sub-domain to your DNS listing for your domain.

First up is perl. There are a bunch of modules you need to get the Mail::DKIM module installed, and then you can install DKIM Proxy. The required modules are listed on the DKIM Proxy site – here is the easy way to get them installed (make sure you are superuser/root):

Note 1: If you’re using Debian or Ubuntu, you should be able to skip this step by installing the libmail-dkim-perl and libnet-server-perl packages. I believe that will get your perl install all the necessary modules and you can go on to installing DKIM-Proxy.

Note 2: If you’ve never run CPAN before, it’s going to ask you to do a manual configuration right off the bat – I usually say ‘no’ to that question and let it attempt autoconfiguration. If you don’t need to set up any proxies or anything, autoconfigure should work.

Note 3: Some of these modules have their own dependencies. If it asks you: “Shall I follow them and prepend them to the queue of modules we are processing right now?” Just hit Enter to take the default answer of [Yes].

perl -MCPAN -e'CPAN::Shell->install("Crypt::OpenSSL::RSA")'
perl -MCPAN -e'CPAN::Shell->install("Digest::SHA")'
perl -MCPAN -e'CPAN::Shell->install("Digest::SHA1")'
perl -MCPAN -e'CPAN::Shell->install("Error")'
perl -MCPAN -e'CPAN::Shell->install("Mail::Address")'
perl -MCPAN -e'CPAN::Shell->install("MIME::Base64")'
perl -MCPAN -e'CPAN::Shell->install("Net::DNS")'
perl -MCPAN -e'CPAN::Shell->install("Net::Server")'

Once your perl modules are installed, you can install dkimproxy. I won’t go into detail – this part is pretty straightforward (follow the instructions on the site).

If you’ve got dkimproxy installed now, the instructions tell you how to generate a private/public key pair and then set up your DNS record. You have to put the public key in the actual DNS record, and this means removing the line breaks and white space.

Before you go any further, reload bind/named (on most systems: /etc/init.d/named restart or /etc/init.d/bind9 restart) and make sure you see your big long public key when you do the following command:

dig selector1._domainkey.my-domain-name.com TXT

If you don’t see your public key there in a big long string, then you did something wrong in DNS. Go back and fix it before you try anything else – trust me. You’ll just end up in circles if your DNS is wrong. I had misspelled something and wasted about 20 minutes trying to figure out what was wrong with DKIM Proxy when it was my DNS entry that was the problem all along.

At this point, if you haven’t already, you should probably create a user and group called ‘dkfilter’ (or something similar).

The instructions give you an example of how the ‘dkimproxy.out’ script could be run. There is also a file in the root of the dkimproxy archive called sample-dkim-init-script.sh. This script is the easiest way to get dkimproxy running as a daemon. Edit the vars at the top of the script to meet your needs – if you used the default selector name of selector1 and the user name dkfilter you probably don’t need to change anything. I changed the domain line to read: DOMAIN=my-domain1.com,my-domain2.com,my-domain3.com

(note: if you are doing this for multiple domains, I hope you remembered to add the TXT DNS record to each domain’s zone file – and verify with dig!)

If the init script works good, copy it to /etc/init.d or wherever your init scripts live, and set it up to start at boot time (distro-dependent; it’s something I do rarely enough that I always have to look it up).

For the last part, you need to set up your mail server to direct it’s outgoing messages at dkimproxy for signing before they head out. Hopefully, you’re using postifx, because there is a cut-and-paste example for the postfix master.cf on the DKIM Proxy site. Don’t forget to do a postfix reload.

Once postfix is set up, they have a couple of test mail addresses you can use. Watch your maillog to see if things are working right. I still got a few delays from Yahoo at first, but now they seem to be flowing normally.

I just got my slice at Slicehost and I went with the Ubuntu Dapper install, following this quick guide to getting apache, mongrel, and rails installed. It worked out ok for the most part – my only problem with these “cut and paste” type guides is it doesn’t explain what you’re actually installing. Thankfully, the author Chris links to the three different articles he collected the steps from, and for someone who’s already aware of the concepts, this how-to makes going through the motions that much easier. I got a newer version of apache than is shown – he has 2.2.2 and 2.2.4 is the newest stable release.

Once I got dapper set up with rails, mongrel and apache, I started to use the deprec gem following the guide posted on Slicehost’s wiki. Once I started going through the steps, I realized that these deprec recipes were way more automated than I wanted right now, and I had to cancel halfway through. I was expecting a per-app kind of automation, but I was in copy-and-paste mode and ran the “cap install_rails_stack” line and realized it was trying to do all the stuff I had just set up. The author of the deprec gem warns that it’s really a set up scripts set up to work for him, and use at your own risk, blah blah blah. It may work for some, but like I said, I found it a little too automated.

I went back and grabbed the deploy.rb I’d used before from Slingshot’s Capistrano support page. This script has worked pretty good for me in the past, but I had to make a few alterations. The script, called apache_2_2_mongrel_deploy.rb, had been updated since the last time I tried it, but it’s still a tad buggy.

One problem I had right off the bat was the reference to a variable called “crontab_email”. This is for a special crontab-related task that I don’t need right at the moment, but since this variable was referenced and not set, it caused an error.

The next major problem is that there’s a place to set the database user and password – so I assumed that the database.yml file that gets generated in setup would be using that user and password. Oddly enough, it actually only uses the username, not the password. Took me a few minutes to realize my initial migrations were bombing because there was no password referenced in database.yml.

Thirdly, the “mongrel_conf” variable that gets set for the mongrel recipes to use is pointing to the wrong directory. I changed the line to: set :mongrel_conf, ”/etc/mongrel_cluster/#{application}.yml”

Lastly, and this is the big one – when trying to start the mongrel cluster for the first time, I got an error from mongrel about the “mongrel_prefix” needing to start with a slash (/) but not end with one. Since the line was: set :mongrel_prefix, ”/usr/local/bin/” I simply took the last slash off the line and my error message went away. Happy as can be, I went ahead and got apache going and fired up my mongrel servers, pointed my web browser at my new server and got:

NOT FOUND

This happened to me the last time I used an earlier version of this script, and for the life of me, I couldn’t remember what the problem was. The apache logs were sparse. The mongrel log offered nothing but telling me when the servers were started or stopped. The production log was blank. After an hour of poking and prodding, I finally remembered what it was – the mongrel config file that had mongrel_prefix set to /usr/local/bin. My mongrel bin directory should have been /usr/bin (actually, I just blanked it out and that worked just fine).

A very frustrating experience. Mongrel is great and you can’t go wrong with this stack for Rails, but man – a better error message would be nice. Something in the log. Anything….

Lately I’ve been playing with wmii, the window manager. The ruby-addons for wmii by mfp open up the base wmii quite a bit. Once I got used to this dynamic window manager, I can’t imagine going back to WIMP.

I started to poke around in the ruby-wmii config files, because I was curious about how whenever I opened Firefox, the client was automatically tagged with “web”. Since I was constantly tagging my other apps with names using MODKEY-ctrl-T, I decided I wanted some of them to auto-tag like the browsers do.

After initially just copying the existing code that tags the browsers, I finally worked out a way to get it to tag things the way I want them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

 # *** excerpt of wmiirc-config.rb ***

 # first, I commented out this orginal tagger

 # {{{ Tag all browser instances as 'web' in addition to the current tag
 # browsers = %w[Firefox Konqueror]
 # browser_re = /^#{browsers.join("|")}/
 # on_createclient(condition{|c| browser_re =~ read("/client/#{c}/class")}) do |cid|
 #   write("/client/#{cid}/tags", normalize(read("/client/#{cid}/tags") + "+web"))
 # end

  #############################################
  # here's the stuff I added
        
  # these are tags for particular apps that I want to have 
  # new tags - note, they won't get appended to their current tag

  tag_for_apps = { 
    "irc" => "Xchat", 
    "gaim" => "Gaim", 
    "web" => "Firefox",
    "4-jedit" => "jedit"
  }
  
  # these are terminals that I've given specific titles to
  # so they can be tagged
  tag_for_named_terms = {
    "2-consoles" => "console:",
    "3-logs" => "log:",
    "1-terms" => "term:"
  }
    
  # now when a new client comes up, check for an autotag
  on_createclient do |cid|
    LOGGER.info "checking for autotag on class: #{read("/client/#{cid}/class")} " + 
        "and name: #{read("/client/#{cid}/name")} "
    # if this client is a terminal, check the 'name'
    if /terminal/ =~ read("/client/#{cid}/class")
      tag_for_named_terms.each do |tfn|
        names_re = /#{tfn[1]}/
        if names_re =~ read("/client/#{cid}/name")
          write("/client/#{cid}/tags", tfn[0])
          LOGGER.info "  ... autotag'd #{tfn[1]} with #{tfn[0]}!"
        end
      end
    # not a terminal, so go by the 'class'
    else
      tag_for_apps.each do |tfa|
        apps_re = /#{tfa[1]}/
        if apps_re =~ read("/client/#{cid}/class")
          write("/client/#{cid}/tags", tfa[0])
          LOGGER.info "  ... autotag'd #{tfa[1]} with #{tfa[0]}!"
        end
      end
    end
  end

Notice, for terminals, I check for certain terminal window titles. I changed my gnome-terminal script a bit, so now it looks like this:

#!/bin/sh

# set working dir
wd="/home/jason/dev/rails_app" 

gnome-terminal --working-directory=$wd --hide-menubar -e "mongrel_rails start" -t "log: web server" \
--window --working-directory=$wd --hide-menubar -e "tail -f log/development.log" -t "log: dev log" \
--window --working-directory=$wd --hide-menubar -e "script/console" -t "console: rails irb" \
--window --working-directory=$wd --hide-menubar -e "mysql -u root --password=mysecret myapp_development" -t "console: mysql" \
--window --working-directory=$wd --hide-menubar -t "term: approot"

For some of my tag names, I stick a number in the front – that’s just to force them into order so I can easily MODKEY-n over to them.

I’d like to work the auto-tagger out into a plugin, but after poking around some of the existing plugins I gave up. I have no idea how to check for on_createclient from within a plugin, and I’m pretty confused about how those long-winded option get passed to plugins. If anyone has any hints, I’d love to hear them.

I do most of my development in Ubuntu. The gnome-terminal app is one of my favorite terminals, mainly I think because of it’s tabbed structure. When I start working on Rails stuff, I usually end up opening a terminal and about 4 or 5 tabs. Everytime I do it, I think to myself “I should just script this to open – I wonder if I can tell gnome-terminal to open tabs with certain scripts loaded in each tab…”

Turns out, it’s not really that difficult. Here’s my shell script:

#!/bin/sh

# set working dir
wd="/home/jason/dev/rails_app" 

gnome-terminal --geometry=179x59+0+25 --working-directory=$wd  -e "mongrel_rails start" -t "web server" \
--tab --working-directory=$wd -e "tail -f log/development.log" -t "dev log" \
--tab --working-directory=$wd -e "script/console" -t "irb console" \
--tab --working-directory=$wd -e "mysql -u root --password=mysecret rails_app_development" -t "mysql" \
--tab

Ok, now I should get some stuff done…

There are a couple of previous walkthroughs on getting Rails working on Debian, so this is really just one of many.