Fun with Fsync & Bind Mounts

The answer to “Why don’t you blog about your work Vlad” and the final dagger in the back of my female audiences libido: moving sendmail spools to tmpfs.

First off, why bother? Well, with the ram being as cheap as it is and finally some solid hardware and software tmpfs is really getting a lot of play. It also helps when yours truly wakes up from an one hour nap and finds a node with 68,000 messages waiting in the spool because of poor disk performance. Talk about a motivator to work on optimizing the mail stream!

Doin’ it the wrong way

The first step is to actually create a tmpfs disk and mount it.

/bin/mount -t tmpfs tmpfs -o size=256M,nr_inodes=1M /var/spool/mqueue.in

That works. This creates a 256MB ram disk and mounts it in /var/spool/mqueue.in. Start up sendmail and everything works fine until mail has to be moved around and processed. Then you get bit in the ass by fsync errors (fsync is enabled by default on the 2.6 kernels). Time to turn that beast off:

  # override compile time flag REQUIRES_DIR_FSYNC 
 O RequiresDirfsync=false

Now we got the ramdisk, we disabled fsync, it should work now without a problem, right? Heh. Nope. Sendmail queue’s need to be on the same filesystem and same partition. If they aren’t sendmail starts complaining and poof.

Doin’ it the right way

This one actually belongs to my bud Pablo who has hacked in bind mounts on pretty much every box I’ve ever had. Bind mounts function similarly (or exactly) like folder mounts with NTFS, a folder (or directory) from one file system can be seen at another point on another file system. Go Pablo: 

mkdir /var/sendmail/tmpfs
mount -t none /var/sendmail/tmpfs

mkdir /var/sendmail/tmpfs/mqueue
mkdir /var/sendmail/tmpfs/mqueue.in

mount /var/sendmail/tmpfs/mqueue /var/spool/mqueue -o bind
mount /var/sendmail/tmpfs/mqueue.in /var/spool/mqueue.in -o bind

So to sum it up

First, thank god I found a woman to marry me because girls don’t respond well to pickup lines referencing linux filesystem optimization. Second, moving sendmail to tmpfs really helped, remarkably. In the few hours since the tweak the load average really went down – Linux calculates the load average by the amount of cycles all running processes take up – so if a ton of children are fired up and waiting on IO for something the load goes through the roof. Memory and IO really improved as well but you’ll have to take my word for it because the output from vmstat looks hella ugly. See why I don’t write about what I do?