Load Balancer Fun

I really don’t like talking smack about my competitors both because I know how difficult of a task we all have always being a step behind spammers and I choose to put my energy into building our own products. After all, with all due professional respect, nobody has unplugged more of their appliances than me.

But yesterday I had a particularly frustrating day of learning more about Linux load balancing than I particularly wanted to. And after several hours of piecing together the concepts through Google and outdated documents and technologies I figured – screw it, I’ll just go buy one from them and move on to one of the other billion projects I have on my desk. I look at the model breakdown and the first thing that strikes me is obvious hard locks in the appliance to limit the number of real servers so you’d have to upgrade to the higher model just because. Ok, fine, what’s $500 wasted on top of $1499? So I go to the order form and they got more fees – multiyear IDS protection subscriptions.

Ok, so now this is just getting ridiculous. So I figured let me hit up chat and see if there is “I’m not a sucker” form. So I ask politely, “Can I buy your load balancer without the extended support contract?” and instead of saying yes or no, he/she responds with “I can send you more information about that, what is your email address?” and I say “No thanks, I don’t want you spamming me, I am buying right now I just want to know if its possible to get it without the energizer updates”; They let me sit around for a minute or two and come back with the barrage of questions: “The updates come with IDS, blah, blah, blah” and I respond with “I just need a load balancer, can I please just order one without updates” and they say “No.” and close the chat faster than I can even blink.

So suffice to say they lost that order. I mean, I can understand that they are crooks and are using the same deceptive advertising that has been available from the beginning of time – low advertised price but by the time you get to the counter you end up paying almost double. And would I have paid $2K? Yeah. But after that treatment I won’t. And this is perhaps yet another reason why you don’t want to be a sales prick, you just might end up pissing off the guys that run data centers and will now spend another day trying to figure it out – and when they do, you will lose a hell of a lot more than the $399 or whatever bs markup it was.

So that’s the lesson for the day: Don’t be a prick when people are trying to give you money. You can still sell by saying “no”  but you can’t sell if you’re throwing customer out of the store.

Anyhow, if you’ve got Linux Kung Fu, this is what I’m trying to do:

ipvsadm -A -t 65.99.255.235:25 -s wrr
ipvsadm -a -t 65.99.255.235:25 -r 65.99.255.242:25 -m -w 1
ipvsadm -a -t 65.99.255.235:25 -r 65.99.255.246:25 -m -w 1

Stock CentOS 5 (RHEL 5) 2.6.18 kernel with net.ipv4.ip_forward turned on and I have a public IP that I want to distribute traffic over the two real servers with direct return (direct path return) with both real servers on the public range. The load balancer is at 65.99.255.230 and here is the tcpdump:

08:29:33.214112 IP (tos 0x0, ttl  64, id 5746, offset 0, flags [DF], proto: TCP (6), length: 60) 65.99.255.230.56232 > 65.99.255.246.smtp: S, cksum 0x4bde (correct), 2861789973:2861789973(0) win 5840 <mss 1460,sackOK,timestamp 64775527 0,nop,wscale 7>
08:29:33.214171 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 65.99.255.246.smtp > 65.99.255.230.56232: S, cksum 0x1f4c (correct), 3193828587:3193828587(0) ack 2861789974 win 5792 <mss 1460,sackOK,timestamp 377709289 64775527,nop,wscale 2>
08:29:33.214179 IP (tos 0x0, ttl  64, id 5747, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.230.56232 > 65.99.255.246.smtp: ., cksum 0x6485 (correct), ack 1 win 46 <nop,nop,timestamp 64775527 377709289>
08:29:33.214232 IP (tos 0x0, ttl  64, id 5748, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.230.56232 > 65.99.255.246.smtp: F, cksum 0x6484 (correct), 1:1(0) ack 1 win 46 <nop,nop,timestamp 64775527 377709289>
08:29:33.214753 IP (tos 0x0, ttl  64, id 38405, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.246.smtp > 65.99.255.230.56232: ., cksum 0x5f0a (correct), ack 2 win 1448 <nop,nop,timestamp 377709289 64775527>
08:29:33.218077 IP (tos 0x0, ttl  64, id 38407, offset 0, flags [DF], proto: TCP (6), length: 117) 65.99.255.246.smtp > 65.99.255.230.56232: P 1:66(65) ack 2 win 1448 <nop,nop,timestamp 377709293 64775527>
08:29:33.218091 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 65.99.255.230.56232 > 65.99.255.246.smtp: R, cksum 0x33d0 (correct), 2861789975:2861789975(0) win 0
08:29:33.218095 IP (tos 0x0, ttl  64, id 38409, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.246.smtp > 65.99.255.230.56232: F, cksum 0x5ec4 (correct), 66:66(0) ack 2 win 1448 <nop,nop,timestamp 377709293 64775527>
08:29:33.218102 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 65.99.255.230.56232 > 65.99.255.246.smtp: R, cksum 0x33d0 (correct), 2861789975:2861789975(0) win 0
08:29:33.222272 IP (tos 0x0, ttl  64, id 23742, offset 0, flags [DF], proto: TCP (6), length: 60) 65.99.255.230.45127 > 65.99.255.242.smtp: S, cksum 0xb04d (correct), 2865052113:2865052113(0) win 5840 <mss 1460,sackOK,timestamp 64775535 0,nop,wscale 7>
08:29:33.222380 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 65.99.255.242.smtp > 65.99.255.230.45127: S, cksum 0xb60a (correct), 2798131280:2798131280(0) ack 2865052114 win 5792 <mss 1460,sackOK,timestamp 408392 64775535,nop,wscale 2>
08:29:33.222396 IP (tos 0x0, ttl  64, id 23743, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.230.45127 > 65.99.255.242.smtp: ., cksum 0xfb43 (correct), ack 1 win 46 <nop,nop,timestamp 64775535 408392>
08:29:33.222476 IP (tos 0x0, ttl  64, id 23744, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.230.45127 > 65.99.255.242.smtp: F, cksum 0xfb42 (correct), 1:1(0) ack 1 win 46 <nop,nop,timestamp 64775535 408392>
08:29:33.223193 IP (tos 0x0, ttl  64, id 8985, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.242.smtp > 65.99.255.230.45127: ., cksum 0xf5c7 (correct), ack 2 win 1448 <nop,nop,timestamp 408393 64775535>
08:29:33.242440 IP (tos 0x0, ttl  64, id 8987, offset 0, flags [DF], proto: TCP (6), length: 117) 65.99.255.242.smtp > 65.99.255.230.45127: P 1:66(65) ack 2 win 1448 <nop,nop,timestamp 408412 64775535>
08:29:33.242454 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 65.99.255.230.45127 > 65.99.255.242.smtp: R, cksum 0x9847 (correct), 2865052115:2865052115(0) win 0
08:29:33.242458 IP (tos 0x0, ttl  64, id 8989, offset 0, flags [DF], proto: TCP (6), length: 52) 65.99.255.242.smtp > 65.99.255.230.45127: F, cksum 0xf572 (correct), 66:66(0) ack 2 win 1448 <nop,nop,timestamp 408412 64775535>
08:29:33.242465 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) 65.99.255.230.45127 > 65.99.255.242.smtp: R, cksum 0x9847 (correct), 2865052115:2865052115(0) win 0

I see the connection come up, and the weights look right:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  65.99.255.235:smtp wrr
  -> i246 Route   1      0          0        
  -> i242 Route   1      0          1 

So I’m missing something here… If you can see it from there, let me know.