Enabling KSM on gentoo

Update: linux-headers-2.6.32 is now available in gentoo, i have updated the post to use that package, instead of patched 2.6.30. The patched linux-headers-2.6.30 is still available in the ebuild bundle if somebody needs it.

Last week I finally created some virtual machines on my home server (virt-manager is a pain to work with compared to VirtualBox), and I wanted to make sure Kernel Samepage Merging is working. I have kernel 2.6.32, CONFIG_KSM enabled in kernel, and qemu-kvm has support for it. But KSM didn't work, because /sys/kernel/mm/ksm/pages_sharing always returned 0. It turns out, that to get it working, linux-headers and also glibc needs to support KSM. KSM support appeared in 2.6.32, so you need linux-headers-2.6.32, which is now available in portage, see gentoo bug #297755. So let's make KSM work on gentoo:

First download into your local overlay two ebuilds I made for you:

• sys-libs/glibc-2.11-r1 with KSM patch from piavlo
• app-emulation/ksm package has an initscript to enable ksm, and ksmtuned daemon, all from fedora qemu rpm


cd /var/tmp/
wget -O gentoo-ksm-ebuilds.tbz2 http://sites.google.com/site/buddsite/gentoo-ksm-ebuilds.tbz2
tar xjf gentoo-ksm-ebuilds.tbz2 -C /usr/local/portage/
(Make sure you have something like PORTDIR_OVERLAY="/usr/local/portage" in your /etc/make.conf)

Unmask the packages we want to emerge (binutils is a dependency for glibc):
echo '~sys-kernel/gentoo-sources-2.6.32' >> /etc/portage/package.keywords
echo '~sys-kernel/linux-headers-2.6.32' >> /etc/portage/package.keywords
echo '=sys-libs/glibc-2.11-r1' >> /etc/portage/package.keywords
echo '=sys-devel/binutils-2.20' >> /etc/portage/package.keywords
echo 'app-emulation/ksm' >> /etc/portage/package.keywords

Now emerge linux-headers and glibc with the KSM patch:
emerge -v1 linux-headers glibc

After this, you need to recompile qemu-kvm too. Plus emerge the ksm ebuild, because it has a nice initscript to enable ksm:
emerge -v1 qemu-kvm ksm

Now enable KSM:
/etc/init.d/ksm start

Start some virtual machines, and examine how much memory are you saving with KSM:
echo "KSM pages shared: $(( $(cat /sys/kernel/mm/ksm/pages_sharing) * $(getconf PAGESIZE) / 1024 / 1024 ))Mb"

You can also try out the ksmtuned daemon from fedora:
/etc/init.d/ksmtuned start

Using proxy auto config to get around the "squid does not know IPv6 yet" limitation

I like making stuff which just works from the users perspective, and they dont have to care about how it works. But i do not like transparent proxies, diverting HTTP traffic without the applications knowing about it just does not seem to be right. But I like the idea of proxy autoconfiguration in web browsers. I did set it up a long time ago, with squid proxy, but there were some problems with it:

• When you go to an address with no DNS records associated with, squid displays an error page. When not using a proxy, browsers redirect you to a search page, or display a much nicer error. (I'm really not into redesigning squid error pages.)


• IPv6. Squid does not know about IPv6 yet.

Well, actually squid does support IPv6, in version 3.1, but as of this writing it is still RC. The most important part of the Internet is the Web, so i'm reluctant to use RC. But we could use the PAC file, which is just a javascript code to not use proxy for IPv6 sites. I wanted to make this work when I first created the proxy autoconfig file, but I could not find a good way to test how it works, so my script always returned the same string: "PROXY proxy:3128; DIRECT". This means: use the proxy server at proxy:3128, and if that fails, use direct connection. Yesterday I stumbled upon pactester, so I tought I'll give it a try.

I wanted the following funcionality:

• If the hostname is not resolvable, use direct.


• If the hostname resolves only to IPv4 addresses, use proxy


• If the hostname resolves only to IPv6 addresses, use direct


• If the hostname resolves to both IPv4 and IPv6 address, use direct.

First, since I use gentoo, I wanted to create an ebuild for pactester. The packaging of pactester is a bit ugly, you'll get 1Mb of bundled dependencies and 15kb of useful code. I did put it into /usr/local/portage/net-proxy/pactester/.

The wpad.dat file should have a function FindProxyForURL(url, host), and newer Internet Exploer also supports a function FindProxyForURLEx(url, host). Here are some things to know:

• Internet Explorer 8 calls the function FindProxyForURLEx.


• pactester and Google Chrome are using the FindProxyForURL function


• There are additional functions which you can use in your script, see pac_utils.js


• There are even more additional functions for you to use in FindProxyForURLEx, see proxy_resolver_v8.cc


• in pactester the function dnsResolveEx() is not available


• Google Chrome and Internet Explorer gives different results when calling dnsResolve()

So with a little experimentation I wrote the following proxy autoconfiguration script: wpad.dat

I found the following links useful:
http://kate-ward.blogspot.com/2008/11/im-up-and-running-with-ipv6.html
http://users.telenet.be/mydotcom/library/network/pac.htm