OpenSolaris guest on Linux KVM

I want a ZFS fileserver for my home network. Something similar to "A Home Fileserver using ZFS". The main difference is I cannot afford a dedicated ZFS server so I have to use virtualization to run Solaris. On linux the first choice is to use KVM, and I also found there is an OpenSolaris JeOS, which is a minimal OpenSolaris installation to use as a guest OS. So download, unpack and start it. Well, it's not as easy as it sounds.

The first problem is that early in the boot process the Solaris guest does a kernel panic. Luckily there is a workaround: you have to upgrade the OS. But since the kernel panics, you have to use VirtualBox to start your guest, upgrade from there, and then you can move back the image to KVM, and start your Solaris guest.

Second problem comes here, Solaris doesn't have a driver for virtio-net, and the only other gigabit nic in KVM is e1000. Solaris does have a driver for that, but unfortunately something is broken and the guest can only send packets, but for some reason cannot receive traffic. So, to have a working network connection you have to use the rtl8139 nic emulation which is 100Mbit, and that's really slow for a file server. But I decided to live with it, until it gets fixed (either in solaris or in qemu-kvm).

Here comes the third problem: Solaris doesn't have a driver for virtio storage, nor for the scsi card KVM emulates. So the only other option is to use IDE storage, but that is limited to 4 devices. I want to use ZFS on 8 HDDs plus there has to be a root zpool, so IDE is not enough. The same situation is described in this post: KVM, SCSI & OpenSolaris.

Now we have two options. Use ide only for the root device, and use iSCSI for the ZFS HDDs. When you take into account that you can have only 100Mbit network interfaces this is really bad. Sure, I could dedicate a virtual NIC for every HDD, but this would only unnecessarily complicate things.

The second option is to ditch KVM and use VirtualBox for OpenSolaris. This would not have the limitations described above, because under VirtualBox the gigabit network works fine, and there is also an option to emulate a SATA controller for the guest, this way I can give the guest 8 HDDs. And this is what I'm going to do next weekend.

So, the result from this weekend is: OpenSolaris - Bud 1:0
:(

The difference between rc_need, rc_use, rc_before and rc_after in gentoo

While trying to figure out why libvirtd starts before drbd (which is not good for me, because i have virtual machines on drbd, and autostarting virtual machines does not work this way) i wanted to know what is the difference between gentoo init script dependency keywords "need" "use" "before" and "after". Unfortunately i found no documentation about it anywhere, so i had to find it out myself:

• rc_need: A service which uses another service will only run if the another service is running. For example nfs needs portmap, so if you start nfs, then portmap will start automatically before it. If you stop portmap, nfs will be stopped before it. If you restart portmap, nfs will be stopped before, and started after portmap.
• rc_use: A service which uses another service will be started after the another service during bootup, and stopped before the another service during shutdown. Example: sshd uses logger and net, so sshd will be started before logger starts, and stopped before logger stops. But if you restart the logger it will not restart the ssh daemon.
• rc_after: The service will be started after another service during bootup, but during shutdown it does not need to be stopped before the other service stops. Example: nfs will start after quota.
• rc_before: The service will be started before another service during bootup. Example: iptables will start before network comes up.

I do not know what is the effect of rc_after and rc_before during shutdown.

Update: I found the documentation for baselayout-2: man 8 runscript

Enabling KSM on gentoo

Update: linux-headers-2.6.32 is now available in gentoo, i have updated the post to use that package, instead of patched 2.6.30. The patched linux-headers-2.6.30 is still available in the ebuild bundle if somebody needs it.

Last week I finally created some virtual machines on my home server (virt-manager is a pain to work with compared to VirtualBox), and I wanted to make sure Kernel Samepage Merging is working. I have kernel 2.6.32, CONFIG_KSM enabled in kernel, and qemu-kvm has support for it. But KSM didn't work, because /sys/kernel/mm/ksm/pages_sharing always returned 0. It turns out, that to get it working, linux-headers and also glibc needs to support KSM. KSM support appeared in 2.6.32, so you need linux-headers-2.6.32, which is now available in portage, see gentoo bug #297755. So let's make KSM work on gentoo:

First download into your local overlay two ebuilds I made for you:

• sys-libs/glibc-2.11-r1 with KSM patch from piavlo
• app-emulation/ksm package has an initscript to enable ksm, and ksmtuned daemon, all from fedora qemu rpm


cd /var/tmp/
wget -O gentoo-ksm-ebuilds.tbz2 http://sites.google.com/site/buddsite/gentoo-ksm-ebuilds.tbz2
tar xjf gentoo-ksm-ebuilds.tbz2 -C /usr/local/portage/
(Make sure you have something like PORTDIR_OVERLAY="/usr/local/portage" in your /etc/make.conf)

Unmask the packages we want to emerge (binutils is a dependency for glibc):
echo '~sys-kernel/gentoo-sources-2.6.32' >> /etc/portage/package.keywords
echo '~sys-kernel/linux-headers-2.6.32' >> /etc/portage/package.keywords
echo '=sys-libs/glibc-2.11-r1' >> /etc/portage/package.keywords
echo '=sys-devel/binutils-2.20' >> /etc/portage/package.keywords
echo 'app-emulation/ksm' >> /etc/portage/package.keywords

Now emerge linux-headers and glibc with the KSM patch:
emerge -v1 linux-headers glibc

After this, you need to recompile qemu-kvm too. Plus emerge the ksm ebuild, because it has a nice initscript to enable ksm:
emerge -v1 qemu-kvm ksm

Now enable KSM:
/etc/init.d/ksm start

Start some virtual machines, and examine how much memory are you saving with KSM:
echo "KSM pages shared: $(( $(cat /sys/kernel/mm/ksm/pages_sharing) * $(getconf PAGESIZE) / 1024 / 1024 ))Mb"

You can also try out the ksmtuned daemon from fedora:
/etc/init.d/ksmtuned start

Using proxy auto config to get around the "squid does not know IPv6 yet" limitation

I like making stuff which just works from the users perspective, and they dont have to care about how it works. But i do not like transparent proxies, diverting HTTP traffic without the applications knowing about it just does not seem to be right. But I like the idea of proxy autoconfiguration in web browsers. I did set it up a long time ago, with squid proxy, but there were some problems with it:

• When you go to an address with no DNS records associated with, squid displays an error page. When not using a proxy, browsers redirect you to a search page, or display a much nicer error. (I'm really not into redesigning squid error pages.)


• IPv6. Squid does not know about IPv6 yet.

Well, actually squid does support IPv6, in version 3.1, but as of this writing it is still RC. The most important part of the Internet is the Web, so i'm reluctant to use RC. But we could use the PAC file, which is just a javascript code to not use proxy for IPv6 sites. I wanted to make this work when I first created the proxy autoconfig file, but I could not find a good way to test how it works, so my script always returned the same string: "PROXY proxy:3128; DIRECT". This means: use the proxy server at proxy:3128, and if that fails, use direct connection. Yesterday I stumbled upon pactester, so I tought I'll give it a try.

I wanted the following funcionality:

• If the hostname is not resolvable, use direct.


• If the hostname resolves only to IPv4 addresses, use proxy


• If the hostname resolves only to IPv6 addresses, use direct


• If the hostname resolves to both IPv4 and IPv6 address, use direct.

First, since I use gentoo, I wanted to create an ebuild for pactester. The packaging of pactester is a bit ugly, you'll get 1Mb of bundled dependencies and 15kb of useful code. I did put it into /usr/local/portage/net-proxy/pactester/.

The wpad.dat file should have a function FindProxyForURL(url, host), and newer Internet Exploer also supports a function FindProxyForURLEx(url, host). Here are some things to know:

• Internet Explorer 8 calls the function FindProxyForURLEx.


• pactester and Google Chrome are using the FindProxyForURL function


• There are additional functions which you can use in your script, see pac_utils.js


• There are even more additional functions for you to use in FindProxyForURLEx, see proxy_resolver_v8.cc


• in pactester the function dnsResolveEx() is not available


• Google Chrome and Internet Explorer gives different results when calling dnsResolve()

So with a little experimentation I wrote the following proxy autoconfiguration script: wpad.dat

I found the following links useful:
http://kate-ward.blogspot.com/2008/11/im-up-and-running-with-ipv6.html
http://users.telenet.be/mydotcom/library/network/pac.htm

Accessing services behind NAT on public address

We have a gateway with haproxy+tproxy. In the haproxy.cfg we have transparent proxying:

source 0.0.0.0 usesrc clientip


Behind the gateway there are several NAT-ed hosts, acting as backends for haproxy. Everythong works fine, but when we want to access a website served by haproxy from the NAT-ed hosts, we get an 503 error, after a timeout.

The problem is the following:
Let there be host A, with ip address 172.21.0.2. This hosts wants to connect to the site hosted on B (172.21.0.3). This website has an address of B.example.com, which resolves to the public IP address on the gateway. The gateway receives a packet from A, and forwards it to the backend server B, but because of the tproxy setup, haproxy keeps the source IP address of A. Now B receives a packet from A and sends a response to A's IP address. But because they are on the same subnet, the packet will reach A directly and haproxy won't see it. A now received a packet from B (172.21.0.3), but he is expecting a packet from the public interface ip address, so he discards it and keep trying to connect. The gateway cannot see a reply from B either, so after a timeout haproxy sends back an 503 error reply to A.

After some trial and error I remembered a presentation from Kadlecsik Jozsef (in hungarian), where he also talks about how to address this problem. He talks about several solutions, and the NETMAP one sounds good for our setup.

So on the gateway we issue the following command:

iptables -t nat -I POSTROUTING -o eth1 -s 172.21.0.0/16 -j NETMAP --to 172.31.0.0/16

And voila! Everything works.

Some explanation about the command: The eth1 interface is the LAN interface, the 172.21.0.0/16 is the LAN subnet, and the 172.31.0.0/16 is an imaginary unused subnet. Packets originating from the LAN and going back to the same LAN will now have a source IP address from the 172.31.x.x range. And because LAN host don't have a route for this subnet, they will route their answer back through the default gateway, which will map the imaginary addresses back to their original.

But the same iptables command with the NETMAP target can be used on any other NAT-ed LAN. This way port forwarding from the public address can be used inside from the LAN too.

Update: There is a slight issue with the above command. Packets originating from the gateway machine will have their IP addresses NETMAP-ed too, which is not pretty. But we can fix that too:

iptables -t nat -I POSTROUTING -m addrtype ! --src-type LOCAL -o eth1 -s 172.21.0.0/16 -j NETMAP --to 172.31.0.0/16


Slowing down ssh brute-force attempts

Once I was attending a presentation about exim. And I heard a good configuration idea: when we decide that an incoming mail is a spam, and we will not receive it, we do not send an error back to the client right away. We delay the error message for a long time (for example 60 seconds). Why is this useful? We already know the client wants to send junk mail, by keeping the connection open we are slowing his rate of sending down, he won't move on to try other servers so soon.

SSH brute force attacks are common, and everybody can see them in their logs. I wanted to apply the same principle, to slow down ssh brute force attacks. After googling around and not finding a solution I tried to find it myself. The following will be gentoo specific, I haven't tried this on other distributions yet.

Somewhere I read that this should be done in pam, which sounds reasonable. So I tried searching for "pam delay", and found pam_delay.so module, but it's not available in gentoo. But i got a hunch:

grep delay /etc/pam.d/*
/etc/pam.d/samba:auth required pam_smbpass.so nodelay
/etc/pam.d/samba:password required pam_smbpass.so nodelay smbconf=/etc/samba/smb.conf

Dead end, but:

locate delay | grep pam
/lib/security/pam_faildelay.so
/usr/share/doc/pam-1.0.4/modules/README.pam_faildelay.bz2
/usr/share/man/man3/pam_fail_delay.3.bz2
/usr/share/man/man8/pam_faildelay.8.bz2

Sounds good. Read the man page. So into which file to put it? My first idea was to put it into /etc/pam.d/system-remote-login, but this is a hard link to system-local-login, and I wanted to leave that alone. It leaves us with /etc/pam.d/sshd which now looks like this:

# set fail delay to 60 sec:
auth optional pam_faildelay.so delay=60000000
auth include system-remote-login
account include system-remote-login
password include system-remote-login
session include system-remote-login

Note that I only added the auth optional line, the include system-remote-login lines were already there.

Introduction Post

I'm a (mostly) linux system administrator. There are some things I cannot find an answer for by googling, but I figure them out myself. So I decided to write them down so others can find it. It will be mainly a technical blog. Please excuse my english, I'm not a native english speaker, and feel free to correct any mistakes I make.