Backups with duply: the root file system

I like duply, but unfortunately there is not much in the documentation about best practices or howtos. So here is the first post in my duply series: how to back up the root file system.

The obvious approach is to set SOURCE=/ in the duply conf file and we are done. The problem is there are lots of other filesystems mounted under root, like /sys, /proc, /dev, /run, etc. So we could come up with a long list of excludes and try it that way and this setup would still be wrong and fragile.

The correct way to do this is to bind mount the root filesystem somewhere else, and back up that mountpoint. This way we shall get the root, the whole root and nothing but the root. So create a pre script like this:

# This script shouldn't have chmod +x because in older duply version we don't have access to variables like $SOURCE

set -o pipefail
set -o nounset
set -o errexit

# Bind mount:
if ! mountpoint -q "$SOURCE"; then
 mount "$REAL_SOURCE" "$SOURCE" -o bind
 mountpoint "$SOURCE" || exit $?
fi

Now add a new variable into the duply conf file and change the SOURCE variable to the bind mountpoint. I like to use /mnt/root-bind:

# The duply pre script will bind mount $REAL_SOURCE under $SOURCE
REAL_SOURCE="/" 
SOURCE="/mnt/root-bind"

One more thing to take care of is now we will have to make sure duply doesn't start the backup in case the pre script fails. So instead of starting the backup with

duply $profile backup

we'll have to do it like this:

duply $profile pre_and_bkp

That's it, enjoy your root filesystem backups.

Coming up next: how to run duply from cron, how to encrypt and sign your backups, how to backup btrfs, lvm, mysql and svn repos.

scp protocol error: mtime.sec not present

After setting up a chroot sftp as described here https://wiki.archlinux.org/index.php/SFTP-chroot, I ran into a problem. Trying to scp fails with the following error:

protocol error: mtime.sec not present

There was no google result for this error, so I'm documenting it, maybe it will come handy for others. The best thing to do in this case is to run scp -v. That gave me a bit more information:

Sink: This service allows sftp connections only.
protocol error: mtime.sec not present

And really, silly me, trying scp where I should use sftp. And indeed, with sftp it works.

"Boot failed: Could not read from CDROM (code 0003)" on ganeti

I couldn't find the right pages with a google search for the following error while booting a virtual machine on ganeti:

Boot failed: Could not read from CDROM (code 0003)

The virtual machine was started to boot from a cdrom like this:

gnt-instance start -H kvm:boot_order=cdrom -H cdrom_image_path=/boot/systemrescuecd-x86-4.0.0.iso $INSTANCE

The problem is that ganeti by default uses the same value for cdrom_disk_type than what's set for disk_type. This means if you set disk_type=paravirtual, then you'll get the following as a parameter to kvm:

-drive file=/boot/systemrescuecd-x86-4.0.0.iso,format=raw,media=cdrom,if=virtio

and this fails to boot because of the if=virtio part. It seems the kvm bios can't boot from virtio cdrom. The best way to fix this is to change cdrom_disk_type from default to scsi or ide:

gnt-cluster modify -H kvm:cdrom_disk_type=scsi

File "something.php" is writeable by group on cPanel

On a cPanel system, if your php files are group writable, then you'll get HTTP 500 Internal Server Errors, and the following in /usr/local/apache/logs/error_log (not in the error_log in the documentroot):

SoftException in Application.cpp:256: File "/home/netfecom/subdomains/store/index.php" is writeable by group
Premature end of script headers: index.php

This check is silly, as cPanel uses user private groups. All the resources on google will tell you something like do a chmod g-w or change umask to 022. But the whole point of user private groups is to have the files group writable by default. So we need to change suphp behaviour. But unfortunately the config file is not anywhere where you would look for it, and searching for "configure suphp cpanel" leads to nothing. So the key information is that it can be found at /opt/suphp/etc/suphp.conf. These changes will do:

--- /opt/suphp/etc/suphp.conf.orig 2014-01-13 15:21:07.000000000 -0500
+++ /opt/suphp/etc/suphp.conf 2014-01-13 15:22:04.000000000 -0500
@@ -28,9 +28,9 @@
 
 
 ; Security options
-allow_file_group_writeable=false
+allow_file_group_writeable=true
 allow_file_others_writeable=false
-allow_directory_group_writeable=false
+allow_directory_group_writeable=true
 allow_directory_others_writeable=false
 
 

Correctly install xdebug on cPanel

There is a lot of conflicting and incomplete information about this around so let me write how to correctly install xdebug on a cPanel system.

First we need to actually install the xdebug php extension. This will of course not work with yum because cPanel replaces the distro php with it's own source-compiled one. So we'll need to install it through cPanel. Go to WHM -> Software -> Module Installers -> PHP Pecl -> Manage. In the search box type "xdebug", for me it currently shows version 2.2.3. Click install. It looks that it actually configures php.ini correctly and if you verify it with phpinfo(), then it actually works.

But in reality it doesn't work properly. You'll get the following messages in the error_log:

PHP Warning:  Xdebug MUST be loaded as a Zend extension in Unknown on line 0

This is annoying and fills up the logs fast. So go to WHM -> Service Configuration -> PHP Configuration Editor -> Advanced Mode. Look for "extension", you'll find that the value is "xdebug.so", delete it.

The above will take care of the warning in the error_log, but xdebug will stop working. For it to work again look for "zend_extension", you'll see "xdebug.so" as a value there, so everything looks fine. But the trick is, that you need to use the full path here. You can find it on the install screen, for me it is /usr/local/lib/php/extensions/no-debug-non-zts-20100525/xdebug.so. Save the configuration and voila, xdebug works and there is no warning in the error_log.

PS: If you know how to do these from the command line, I'd appreciate it.

Installing sysbench on cPanel

I'm making a note of it because it was non-trivial and no google search led me to the solution.

The problem is that the sysbench package won't install on cPanel infested CentOS:

# yum install sysbench
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/metalink                                                                                                                                                                                            |  24 kB     00:00     
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
base                                                                                                                                                                                                     | 3.7 kB     00:00     
epel                                                                                                                                                                                                     | 4.2 kB     00:00     
epel/primary_db                                                                                                                                                                                          | 5.8 MB     00:00     
extras                                                                                                                                                                                                   | 3.4 kB     00:00     
updates                                                                                                                                                                                                  | 3.4 kB     00:00     
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package sysbench.x86_64 0:0.4.12-5.el6 will be installed
--> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Running transaction check
---> Package postgresql-libs.x86_64 0:8.4.18-1.el6_4 will be installed
---> Package sysbench.x86_64 0:0.4.12-5.el6 will be installed
--> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Finished Dependency Resolution
Error: Package: sysbench-0.4.12-5.el6.x86_64 (epel)
           Requires: libmysqlclient_r.so.16()(64bit)
Error: Package: sysbench-0.4.12-5.el6.x86_64 (epel)
           Requires: libmysqlclient_r.so.16(libmysqlclient_16)(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

The problem is that cPanel adds exclude=mysql* to /etc/yum.conf. But even when ignoring the excludes, installing sysbench won't work:

# yum install --disableexcludes=all sysbench
Loaded plugins: downloadonly, fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package sysbench.x86_64 0:0.4.12-5.el6 will be installed
--> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Running transaction check
---> Package mysql-libs.x86_64 0:5.1.71-1.el6 will be installed
---> Package postgresql-libs.x86_64 0:8.4.18-1.el6_4 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================
 Package                                                    Arch                                              Version                                                     Repository                                       Size
================================================================================================================================================================================================================================
Installing:
 sysbench                                                   x86_64                                            0.4.12-5.el6                                                epel                                             74 k
Installing for dependencies:
 mysql-libs                                                 x86_64                                            5.1.71-1.el6                                                base                                            1.2 M
 postgresql-libs                                            x86_64                                            8.4.18-1.el6_4                                              base                                            201 k

Transaction Summary
================================================================================================================================================================================================================================
Install       3 Package(s)

Total download size: 1.5 M
Installed size: 4.8 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): mysql-libs-5.1.71-1.el6.x86_64.rpm                                                                                                                                                                | 1.2 MB     00:00     
(2/3): postgresql-libs-8.4.18-1.el6_4.x86_64.rpm                                                                                                                                                         | 201 kB     00:00     
(3/3): sysbench-0.4.12-5.el6.x86_64.rpm                                                                                                                                                                  |  74 kB     00:00     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                           9.4 MB/s | 1.5 MB     00:00     
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Importing GPG key 0x0608B895:
 Userid : EPEL (6) 
 Package: epel-release-6-8.noarch (@epel/6.5)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test


Transaction Check Error:
  file /usr/share/mysql/charsets/README from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/Index.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/armscii8.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/ascii.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp1250.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp1251.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp1256.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp1257.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp850.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp852.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp866.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/dec8.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/geostd8.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/greek.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/hebrew.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/hp8.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/keybcs2.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/koi8r.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/koi8u.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/latin1.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/latin2.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/latin5.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/latin7.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/macce.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/macroman.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/swe7.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/czech/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/danish/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/dutch/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/english/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/estonian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/french/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/german/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/greek/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/hungarian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/italian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/japanese/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/korean/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/norwegian-ny/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/norwegian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/polish/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/portuguese/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/romanian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/russian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/serbian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/slovak/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/spanish/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/swedish/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/ukrainian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64

Error Summary
-------------

Ok, let's try to install without dependencies:

# yumdownloader sysbench
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
sysbench-0.4.12-5.el6.x86_64.rpm                                                                                                                                                                         |  74 kB     00:00     

# rpm -ivh sysbench-0.4.12-5.el6.x86_64.rpm 
warning: sysbench-0.4.12-5.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
error: Failed dependencies:
 libmysqlclient_r.so.16()(64bit) is needed by sysbench-0.4.12-5.el6.x86_64
 libmysqlclient_r.so.16(libmysqlclient_16)(64bit) is needed by sysbench-0.4.12-5.el6.x86_64
 libpq.so.5()(64bit) is needed by sysbench-0.4.12-5.el6.x86_64

# rpm -ivh --nodeps sysbench-0.4.12-5.el6.x86_64.rpm 
warning: sysbench-0.4.12-5.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing...                ########################################### [100%]
   1:sysbench               ########################################### [100%]

Great, we have sysbench installed now, but it doesn't work:

# sysbench
sysbench: error while loading shared libraries: libmysqlclient_r.so.16: cannot open shared object file: No such file or directory

And here comes the trick, quoting cPanel support:

I can't guarantee that this will work, however the file you need is located at /usr/lib64/libmysqlclient_r.so. You may try creating a symlink from the missing file to this file to see if that resolves this issue.

# ln -s /usr/lib64/libmysqlclient_r.so /usr/lib64/libmysqlclient_r.so.16

Please note that the version installed is /usr/lib64/libmysqlclient_r.so.18 while you need /usr/lib64/libmysqlclient_r.so.16 so there is a chance that this may not work with this library, however if it will work the above should do it.

We are not done yet though, because after symlinking it still doesn't work:

# ln -s /usr/lib64/libmysqlclient_r.so /usr/lib64/libmysqlclient_r.so.16
# sysbench
sysbench: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory

But that's easy to solve:

root@three [~]# yum whatprovides libpq.so.5
Loaded plugins: downloadonly, fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
postgresql-libs-8.4.18-1.el6_4.i686 : The shared libraries required for any PostgreSQL clients
Repo        : base
Matched from:
Other       : libpq.so.5



root@three [~]# yum install postgresql-libs
Loaded plugins: downloadonly, fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package postgresql-libs.x86_64 0:8.4.18-1.el6_4 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================
 Package                                                    Arch                                              Version                                                     Repository                                       Size
================================================================================================================================================================================================================================
Installing:
 postgresql-libs                                            x86_64                                            8.4.18-1.el6_4                                              base                                            201 k

Transaction Summary
================================================================================================================================================================================================================================
Install       1 Package(s)

Total size: 201 k
Installed size: 624 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : postgresql-libs-8.4.18-1.el6_4.x86_64                                                                                                                                                                        1/1 
  Verifying  : postgresql-libs-8.4.18-1.el6_4.x86_64                                                                                                                                                                        1/1 

Installed:
  postgresql-libs.x86_64 0:8.4.18-1.el6_4                                                                                                                                                                                       

Complete!

And now we are finally happy. There is still a warning on stderr, but seems to work fine (I didn't test the mysql benchmarks):

# sysbench --version
sysbench: /usr/lib64/libmysqlclient_r.so.16: no version information available (required by sysbench)
sysbench 0.4.12

gentoo: htpasswd: BCRYPT algorithm not supported on this platform

I'm making a note of it, as a google search didn't find the straightforward solution. In case you get the error "htpasswd: BCRYPT algorithm not supported on this platform" when using htpasswd -B on gentoo. The solution is to recompile apr-util with the openssl USE flag. Unfortunately it's switched off by default.

echo -e 'dev-libs/apr-util\topenssl\t# htpasswd: BCRYPT algorithm not supported on this platform' >> /etc/portage/package.use

How to install open source vmware tools on Ubuntu lucid

I did not find a simple and straightforward solution anywhere, so here it is:

apt-get install --no-install-recommends open-vm-source module-assistant
module-assistant prepare open-vm
module-assistant auto-install open-vm
apt-get install --no-install-recommends open-vm-tools

This approach has a nice side effect: when you update your kernel, the modules gets recompiled too.

Maximize rdesktop on startup

On my netbook, I use different displays, and I always had a difficulty when choosing the geometry option. So I wrote this as /usr/local/bin/rdesktop

#!/bin/bash

# Determine the maximum window size:
declare -i w="$(xprop -root | awk -F'[, ]' '/^_NET_WORKAREA/ { print $7 }')"
declare -i h="$(xprop -root | awk -F'[, ]' '/^_NET_WORKAREA/ { print $9 }')"
command -p rdesktop -D -g "${w}x${h}" "$@"

OpenSolaris guest on Linux KVM

I want a ZFS fileserver for my home network. Something similar to "A Home Fileserver using ZFS". The main difference is I cannot afford a dedicated ZFS server so I have to use virtualization to run Solaris. On linux the first choice is to use KVM, and I also found there is an OpenSolaris JeOS, which is a minimal OpenSolaris installation to use as a guest OS. So download, unpack and start it. Well, it's not as easy as it sounds.

The first problem is that early in the boot process the Solaris guest does a kernel panic. Luckily there is a workaround: you have to upgrade the OS. But since the kernel panics, you have to use VirtualBox to start your guest, upgrade from there, and then you can move back the image to KVM, and start your Solaris guest.

Second problem comes here, Solaris doesn't have a driver for virtio-net, and the only other gigabit nic in KVM is e1000. Solaris does have a driver for that, but unfortunately something is broken and the guest can only send packets, but for some reason cannot receive traffic. So, to have a working network connection you have to use the rtl8139 nic emulation which is 100Mbit, and that's really slow for a file server. But I decided to live with it, until it gets fixed (either in solaris or in qemu-kvm).

Here comes the third problem: Solaris doesn't have a driver for virtio storage, nor for the scsi card KVM emulates. So the only other option is to use IDE storage, but that is limited to 4 devices. I want to use ZFS on 8 HDDs plus there has to be a root zpool, so IDE is not enough. The same situation is described in this post: KVM, SCSI & OpenSolaris.

Now we have two options. Use ide only for the root device, and use iSCSI for the ZFS HDDs. When you take into account that you can have only 100Mbit network interfaces this is really bad. Sure, I could dedicate a virtual NIC for every HDD, but this would only unnecessarily complicate things.

The second option is to ditch KVM and use VirtualBox for OpenSolaris. This would not have the limitations described above, because under VirtualBox the gigabit network works fine, and there is also an option to emulate a SATA controller for the guest, this way I can give the guest 8 HDDs. And this is what I'm going to do next weekend.

So, the result from this weekend is: OpenSolaris - Bud 1:0
:(

The difference between rc_need, rc_use, rc_before and rc_after in gentoo

While trying to figure out why libvirtd starts before drbd (which is not good for me, because i have virtual machines on drbd, and autostarting virtual machines does not work this way) i wanted to know what is the difference between gentoo init script dependency keywords "need" "use" "before" and "after". Unfortunately i found no documentation about it anywhere, so i had to find it out myself:

• rc_need: A service which uses another service will only run if the another service is running. For example nfs needs portmap, so if you start nfs, then portmap will start automatically before it. If you stop portmap, nfs will be stopped before it. If you restart portmap, nfs will be stopped before, and started after portmap.
• rc_use: A service which uses another service will be started after the another service during bootup, and stopped before the another service during shutdown. Example: sshd uses logger and net, so sshd will be started before logger starts, and stopped before logger stops. But if you restart the logger it will not restart the ssh daemon.
• rc_after: The service will be started after another service during bootup, but during shutdown it does not need to be stopped before the other service stops. Example: nfs will start after quota.
• rc_before: The service will be started before another service during bootup. Example: iptables will start before network comes up.

I do not know what is the effect of rc_after and rc_before during shutdown.

Update: I found the documentation for baselayout-2: man 8 runscript

Slowing down ssh brute-force attempts

Once I was attending a presentation about exim. And I heard a good configuration idea: when we decide that an incoming mail is a spam, and we will not receive it, we do not send an error back to the client right away. We delay the error message for a long time (for example 60 seconds). Why is this useful? We already know the client wants to send junk mail, by keeping the connection open we are slowing his rate of sending down, he won't move on to try other servers so soon.

SSH brute force attacks are common, and everybody can see them in their logs. I wanted to apply the same principle, to slow down ssh brute force attacks. After googling around and not finding a solution I tried to find it myself. The following will be gentoo specific, I haven't tried this on other distributions yet.

Somewhere I read that this should be done in pam, which sounds reasonable. So I tried searching for "pam delay", and found pam_delay.so module, but it's not available in gentoo. But i got a hunch:

grep delay /etc/pam.d/*
/etc/pam.d/samba:auth required pam_smbpass.so nodelay
/etc/pam.d/samba:password required pam_smbpass.so nodelay smbconf=/etc/samba/smb.conf

Dead end, but:

locate delay | grep pam
/lib/security/pam_faildelay.so
/usr/share/doc/pam-1.0.4/modules/README.pam_faildelay.bz2
/usr/share/man/man3/pam_fail_delay.3.bz2
/usr/share/man/man8/pam_faildelay.8.bz2

Sounds good. Read the man page. So into which file to put it? My first idea was to put it into /etc/pam.d/system-remote-login, but this is a hard link to system-local-login, and I wanted to leave that alone. It leaves us with /etc/pam.d/sshd which now looks like this:

# set fail delay to 60 sec:
auth optional pam_faildelay.so delay=60000000
auth include system-remote-login
account include system-remote-login
password include system-remote-login
session include system-remote-login

Note that I only added the auth optional line, the include system-remote-login lines were already there.