Accessing services behind NAT on public address

We have a gateway with haproxy+tproxy. In the haproxy.cfg we have transparent proxying:

source 0.0.0.0 usesrc clientip


Behind the gateway there are several NAT-ed hosts, acting as backends for haproxy. Everythong works fine, but when we want to access a website served by haproxy from the NAT-ed hosts, we get an 503 error, after a timeout.

The problem is the following:
Let there be host A, with ip address 172.21.0.2. This hosts wants to connect to the site hosted on B (172.21.0.3). This website has an address of B.example.com, which resolves to the public IP address on the gateway. The gateway receives a packet from A, and forwards it to the backend server B, but because of the tproxy setup, haproxy keeps the source IP address of A. Now B receives a packet from A and sends a response to A's IP address. But because they are on the same subnet, the packet will reach A directly and haproxy won't see it. A now received a packet from B (172.21.0.3), but he is expecting a packet from the public interface ip address, so he discards it and keep trying to connect. The gateway cannot see a reply from B either, so after a timeout haproxy sends back an 503 error reply to A.

After some trial and error I remembered a presentation from Kadlecsik Jozsef (in hungarian), where he also talks about how to address this problem. He talks about several solutions, and the NETMAP one sounds good for our setup.

So on the gateway we issue the following command:

iptables -t nat -I POSTROUTING -o eth1 -s 172.21.0.0/16 -j NETMAP --to 172.31.0.0/16

And voila! Everything works.

Some explanation about the command: The eth1 interface is the LAN interface, the 172.21.0.0/16 is the LAN subnet, and the 172.31.0.0/16 is an imaginary unused subnet. Packets originating from the LAN and going back to the same LAN will now have a source IP address from the 172.31.x.x range. And because LAN host don't have a route for this subnet, they will route their answer back through the default gateway, which will map the imaginary addresses back to their original.

But the same iptables command with the NETMAP target can be used on any other NAT-ed LAN. This way port forwarding from the public address can be used inside from the LAN too.

Update: There is a slight issue with the above command. Packets originating from the gateway machine will have their IP addresses NETMAP-ed too, which is not pretty. But we can fix that too:

iptables -t nat -I POSTROUTING -m addrtype ! --src-type LOCAL -o eth1 -s 172.21.0.0/16 -j NETMAP --to 172.31.0.0/16


Slowing down ssh brute-force attempts

Once I was attending a presentation about exim. And I heard a good configuration idea: when we decide that an incoming mail is a spam, and we will not receive it, we do not send an error back to the client right away. We delay the error message for a long time (for example 60 seconds). Why is this useful? We already know the client wants to send junk mail, by keeping the connection open we are slowing his rate of sending down, he won't move on to try other servers so soon.

SSH brute force attacks are common, and everybody can see them in their logs. I wanted to apply the same principle, to slow down ssh brute force attacks. After googling around and not finding a solution I tried to find it myself. The following will be gentoo specific, I haven't tried this on other distributions yet.

Somewhere I read that this should be done in pam, which sounds reasonable. So I tried searching for "pam delay", and found pam_delay.so module, but it's not available in gentoo. But i got a hunch:

grep delay /etc/pam.d/*
/etc/pam.d/samba:auth required pam_smbpass.so nodelay
/etc/pam.d/samba:password required pam_smbpass.so nodelay smbconf=/etc/samba/smb.conf

Dead end, but:

locate delay | grep pam
/lib/security/pam_faildelay.so
/usr/share/doc/pam-1.0.4/modules/README.pam_faildelay.bz2
/usr/share/man/man3/pam_fail_delay.3.bz2
/usr/share/man/man8/pam_faildelay.8.bz2

Sounds good. Read the man page. So into which file to put it? My first idea was to put it into /etc/pam.d/system-remote-login, but this is a hard link to system-local-login, and I wanted to leave that alone. It leaves us with /etc/pam.d/sshd which now looks like this:

# set fail delay to 60 sec:
auth optional pam_faildelay.so delay=60000000
auth include system-remote-login
account include system-remote-login
password include system-remote-login
session include system-remote-login

Note that I only added the auth optional line, the include system-remote-login lines were already there.

Introduction Post

I'm a (mostly) linux system administrator. There are some things I cannot find an answer for by googling, but I figure them out myself. So I decided to write them down so others can find it. It will be mainly a technical blog. Please excuse my english, I'm not a native english speaker, and feel free to correct any mistakes I make.