Backups with duply: the root file system

I like duply, but unfortunately there is not much in the documentation about best practices or howtos. So here is the first post in my duply series: how to back up the root file system.

The obvious approach is to set SOURCE=/ in the duply conf file and we are done. The problem is there are lots of other filesystems mounted under root, like /sys, /proc, /dev, /run, etc. So we could come up with a long list of excludes and try it that way and this setup would still be wrong and fragile.

The correct way to do this is to bind mount the root filesystem somewhere else, and back up that mountpoint. This way we shall get the root, the whole root and nothing but the root. So create a pre script like this:

# This script shouldn't have chmod +x because in older duply version we don't have access to variables like $SOURCE

set -o pipefail
set -o nounset
set -o errexit

# Bind mount:
if ! mountpoint -q "$SOURCE"; then
 mount "$REAL_SOURCE" "$SOURCE" -o bind
 mountpoint "$SOURCE" || exit $?
fi

Now add a new variable into the duply conf file and change the SOURCE variable to the bind mountpoint. I like to use /mnt/root-bind:

# The duply pre script will bind mount $REAL_SOURCE under $SOURCE
REAL_SOURCE="/" 
SOURCE="/mnt/root-bind"

One more thing to take care of is now we will have to make sure duply doesn't start the backup in case the pre script fails. So instead of starting the backup with

duply $profile backup

we'll have to do it like this:

duply $profile pre_and_bkp

That's it, enjoy your root filesystem backups.

Coming up next: how to run duply from cron, how to encrypt and sign your backups, how to backup btrfs, lvm, mysql and svn repos.

lsmcli with netapp on ubuntu

Skimming through the RHEL 7.1 Release Notes on Storage I noticed libStorageMgmt, and got curious. Netapp provides an oncommand system manager rpm, which when extracted works on ubuntu too, but it starts a web application on localhost and then opens a browser, and that's how you manage the storage. Yuck! So let's try this libStorageMgmt stuff:

$ suod apt-get install libstoragemgmtd libstoragemgmt-tools

$ lsmcli
Traceback (most recent call last):
  File "/usr/bin/lsmcli", line 19, in 
    import lsm.cmdline
  File "/usr/lib/python2.7/dist-packages/lsm/__init__.py", line 23, in 
    from smis import Smis
  File "/usr/lib/python2.7/dist-packages/lsm/smis.py", line 20, in 
    import pywbem
ImportError: No module named pywbem

$ sudo apt-get install python-pywbem

$ sudo service libstoragemgmtd start
$ sudo gpasswd -a paul.tobias libstoragemgmt

$ lsmcli --list=SYSTEMS
ID         | Name     | Status
-----------+----------+-------
1918101635 | netapp01 | OK    

$ lsmcli --list POOLS
ID                                   | Name  | Total space    | Free space    | System ID 
-------------------------------------+-------+----------------+---------------+-----------
c6b01446-7146-11e3-a01e-123478563412 | aggr0 | 11322462519296 | 3456236142592 | 1918101635

$ lsmcli --list POOLS -H
ID                                   | Name  | Total space | Free space | System ID 
-------------------------------------+-------+-------------+------------+-----------
c6b01446-7146-11e3-a01e-123478563412 | aggr0 | 10.30 TiB   | 3.14 TiB   | 1918101635

$ lsmcli --list FS -H
ID                                   | Name                      | Total space | Free space | Pool ID                             
-------------------------------------+---------------------------+-------------+------------+-------------------------------------
c7b367ea-7146-11e3-a01e-123478563412 | vol0                      | 171.00 GiB  | 164.84 GiB | c6b01446-7146-11e3-a01e-123478563412
7ff4e8db-bb3b-11e3-821a-123478563412 | vms                       | 4.75 TiB    | 1.63 TiB   | c6b01446-7146-11e3-a01e-123478563412
04d28a88-12f9-425a-8149-5b70d1d9951d | brush_backup_restore_test | 2.50 TiB    | 2.46 TiB   | c6b01446-7146-11e3-a01e-123478563412
2f228ee9-4b09-4ce2-9e05-ca5bcdbbcc7c | brush_extra               | 1.00 TiB    | 63.63 GiB  | c6b01446-7146-11e3-a01e-123478563412
5548a33d-d0f7-4dd1-ac49-3ca07f5d76a9 | test                      | 1.24 GiB    | 1.23 GiB   | c6b01446-7146-11e3-a01e-123478563412

$ lsmcli --list EXPORTS
Key            | Value                                                  
---------------+--------------------------------------------------------
ID             | 998e7bde67701ad96ebb14cbb5927c90                       
File system ID | 7ff4e8db-bb3b-11e3-821a-123478563412                   
Export Path    | /vol/vms                                               
Authentication | sys                                                    
Root           | 172.28.50.100, 172.28.50.63, 172.28.50.62, 172.28.50.61
Read/Write     | *                                                      
ReadOnly       | None                                                   
Anon UID       | None                                                   
Anon GID       | None                                                   
Options        | None                                                   
ID             | f962aa0dc595c52f676bc3d52b8e9c92                       
File system ID | c7b367ea-7146-11e3-a01e-123478563412                   
Export Path    | /vol/vol0                                              
Authentication | sys                                                    
Root           | 172.28.50.21                                           
Read/Write     | 172.28.50.21                                           
ReadOnly       | None                                                   
Anon UID       | None                                                   
Anon GID       | None                                                   
Options        | None                                                   
ID             | f962aa0dc595c52f676bc3d52b8e9c92                       
File system ID | c7b367ea-7146-11e3-a01e-123478563412                   
Export Path    | /vol/vol0/home                                         
Authentication | sys                                                    
Root           | 172.28.50.21                                           
Read/Write     | *                                                      
ReadOnly       | None                                                   
Anon UID       | None                                                   
Anon GID       | None                                                   
Options        | None                                                   
ID             | 8aa21057ed71d0c1b579db8ea1f3e5ae                       
File system ID | 2f228ee9-4b09-4ce2-9e05-ca5bcdbbcc7c                   
Export Path    | /vol/brush_extra                                       
Authentication | sys                                                    
Root           | 172.28.5000.21, 172.28.50.100                          
Read/Write     | *                                                      
ReadOnly       | None                                                   
Anon UID       | None                                                   
Anon GID       | None                                                   
Options        | None                                                   
ID             | ca24797f40f9b55a2c6aa8174f1f0fee                       
File system ID | 04d28a88-12f9-425a-8149-5b70d1d9951d                   
Export Path    | /vol/brush_backup_restore_test                         
Authentication | sys                                                    
Root           | 172.28.50.100                                          
Read/Write     | 172.28.50.100                                          
ReadOnly       | None                                                   
Anon UID       | None                                                   
Anon GID       | None                                                   
Options        | None                                                   

$ lsmcli --create-fs=test --size 1G --pool c6b01446-7146-11e3-a01e-123478563412
ID                                   | Name | Total space | Free space | Pool ID                             
-------------------------------------+------+-------------+------------+-------------------------------------
5548a33d-d0f7-4dd1-ac49-3ca07f5d76a9 | test |  1326075904 | 1325924352 | c6b01446-7146-11e3-a01e-123478563412

Sweet! I can finally forget that oncommand system manager and use the command line to manage netapp.

scp protocol error: mtime.sec not present

After setting up a chroot sftp as described here https://wiki.archlinux.org/index.php/SFTP-chroot, I ran into a problem. Trying to scp fails with the following error:

protocol error: mtime.sec not present

There was no google result for this error, so I'm documenting it, maybe it will come handy for others. The best thing to do in this case is to run scp -v. That gave me a bit more information:

Sink: This service allows sftp connections only.
protocol error: mtime.sec not present

And really, silly me, trying scp where I should use sftp. And indeed, with sftp it works.

"Boot failed: Could not read from CDROM (code 0003)" on ganeti

I couldn't find the right pages with a google search for the following error while booting a virtual machine on ganeti:

Boot failed: Could not read from CDROM (code 0003)

The virtual machine was started to boot from a cdrom like this:

gnt-instance start -H kvm:boot_order=cdrom -H cdrom_image_path=/boot/systemrescuecd-x86-4.0.0.iso $INSTANCE

The problem is that ganeti by default uses the same value for cdrom_disk_type than what's set for disk_type. This means if you set disk_type=paravirtual, then you'll get the following as a parameter to kvm:

-drive file=/boot/systemrescuecd-x86-4.0.0.iso,format=raw,media=cdrom,if=virtio

and this fails to boot because of the if=virtio part. It seems the kvm bios can't boot from virtio cdrom. The best way to fix this is to change cdrom_disk_type from default to scsi or ide:

gnt-cluster modify -H kvm:cdrom_disk_type=scsi

File "something.php" is writeable by group on cPanel

On a cPanel system, if your php files are group writable, then you'll get HTTP 500 Internal Server Errors, and the following in /usr/local/apache/logs/error_log (not in the error_log in the documentroot):

SoftException in Application.cpp:256: File "/home/netfecom/subdomains/store/index.php" is writeable by group
Premature end of script headers: index.php

This check is silly, as cPanel uses user private groups. All the resources on google will tell you something like do a chmod g-w or change umask to 022. But the whole point of user private groups is to have the files group writable by default. So we need to change suphp behaviour. But unfortunately the config file is not anywhere where you would look for it, and searching for "configure suphp cpanel" leads to nothing. So the key information is that it can be found at /opt/suphp/etc/suphp.conf. These changes will do:

--- /opt/suphp/etc/suphp.conf.orig 2014-01-13 15:21:07.000000000 -0500
+++ /opt/suphp/etc/suphp.conf 2014-01-13 15:22:04.000000000 -0500
@@ -28,9 +28,9 @@
 
 
 ; Security options
-allow_file_group_writeable=false
+allow_file_group_writeable=true
 allow_file_others_writeable=false
-allow_directory_group_writeable=false
+allow_directory_group_writeable=true
 allow_directory_others_writeable=false
 
 

Correctly install xdebug on cPanel

There is a lot of conflicting and incomplete information about this around so let me write how to correctly install xdebug on a cPanel system.

First we need to actually install the xdebug php extension. This will of course not work with yum because cPanel replaces the distro php with it's own source-compiled one. So we'll need to install it through cPanel. Go to WHM -> Software -> Module Installers -> PHP Pecl -> Manage. In the search box type "xdebug", for me it currently shows version 2.2.3. Click install. It looks that it actually configures php.ini correctly and if you verify it with phpinfo(), then it actually works.

But in reality it doesn't work properly. You'll get the following messages in the error_log:

PHP Warning:  Xdebug MUST be loaded as a Zend extension in Unknown on line 0

This is annoying and fills up the logs fast. So go to WHM -> Service Configuration -> PHP Configuration Editor -> Advanced Mode. Look for "extension", you'll find that the value is "xdebug.so", delete it.

The above will take care of the warning in the error_log, but xdebug will stop working. For it to work again look for "zend_extension", you'll see "xdebug.so" as a value there, so everything looks fine. But the trick is, that you need to use the full path here. You can find it on the install screen, for me it is /usr/local/lib/php/extensions/no-debug-non-zts-20100525/xdebug.so. Save the configuration and voila, xdebug works and there is no warning in the error_log.

PS: If you know how to do these from the command line, I'd appreciate it.

Installing sysbench on cPanel

I'm making a note of it because it was non-trivial and no google search led me to the solution.

The problem is that the sysbench package won't install on cPanel infested CentOS:

# yum install sysbench
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/metalink                                                                                                                                                                                            |  24 kB     00:00     
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
base                                                                                                                                                                                                     | 3.7 kB     00:00     
epel                                                                                                                                                                                                     | 4.2 kB     00:00     
epel/primary_db                                                                                                                                                                                          | 5.8 MB     00:00     
extras                                                                                                                                                                                                   | 3.4 kB     00:00     
updates                                                                                                                                                                                                  | 3.4 kB     00:00     
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package sysbench.x86_64 0:0.4.12-5.el6 will be installed
--> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Running transaction check
---> Package postgresql-libs.x86_64 0:8.4.18-1.el6_4 will be installed
---> Package sysbench.x86_64 0:0.4.12-5.el6 will be installed
--> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Finished Dependency Resolution
Error: Package: sysbench-0.4.12-5.el6.x86_64 (epel)
           Requires: libmysqlclient_r.so.16()(64bit)
Error: Package: sysbench-0.4.12-5.el6.x86_64 (epel)
           Requires: libmysqlclient_r.so.16(libmysqlclient_16)(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

The problem is that cPanel adds exclude=mysql* to /etc/yum.conf. But even when ignoring the excludes, installing sysbench won't work:

# yum install --disableexcludes=all sysbench
Loaded plugins: downloadonly, fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package sysbench.x86_64 0:0.4.12-5.el6 will be installed
--> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: sysbench-0.4.12-5.el6.x86_64
--> Running transaction check
---> Package mysql-libs.x86_64 0:5.1.71-1.el6 will be installed
---> Package postgresql-libs.x86_64 0:8.4.18-1.el6_4 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================
 Package                                                    Arch                                              Version                                                     Repository                                       Size
================================================================================================================================================================================================================================
Installing:
 sysbench                                                   x86_64                                            0.4.12-5.el6                                                epel                                             74 k
Installing for dependencies:
 mysql-libs                                                 x86_64                                            5.1.71-1.el6                                                base                                            1.2 M
 postgresql-libs                                            x86_64                                            8.4.18-1.el6_4                                              base                                            201 k

Transaction Summary
================================================================================================================================================================================================================================
Install       3 Package(s)

Total download size: 1.5 M
Installed size: 4.8 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): mysql-libs-5.1.71-1.el6.x86_64.rpm                                                                                                                                                                | 1.2 MB     00:00     
(2/3): postgresql-libs-8.4.18-1.el6_4.x86_64.rpm                                                                                                                                                         | 201 kB     00:00     
(3/3): sysbench-0.4.12-5.el6.x86_64.rpm                                                                                                                                                                  |  74 kB     00:00     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                           9.4 MB/s | 1.5 MB     00:00     
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Importing GPG key 0x0608B895:
 Userid : EPEL (6) 
 Package: epel-release-6-8.noarch (@epel/6.5)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test


Transaction Check Error:
  file /usr/share/mysql/charsets/README from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/Index.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/armscii8.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/ascii.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp1250.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp1251.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp1256.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp1257.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp850.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp852.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/cp866.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/dec8.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/geostd8.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/greek.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/hebrew.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/hp8.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/keybcs2.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/koi8r.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/koi8u.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/latin1.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/latin2.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/latin5.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/latin7.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/macce.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/macroman.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/charsets/swe7.xml from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/czech/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/danish/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/dutch/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/english/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/estonian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/french/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/german/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/greek/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/hungarian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/italian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/japanese/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/korean/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/norwegian-ny/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/norwegian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/polish/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/portuguese/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/romanian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/russian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/serbian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/slovak/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/spanish/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/swedish/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64
  file /usr/share/mysql/ukrainian/errmsg.sys from install of mysql-libs-5.1.71-1.el6.x86_64 conflicts with file from package MySQL55-server-5.5.34-2.cp1136.x86_64

Error Summary
-------------

Ok, let's try to install without dependencies:

# yumdownloader sysbench
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
sysbench-0.4.12-5.el6.x86_64.rpm                                                                                                                                                                         |  74 kB     00:00     

# rpm -ivh sysbench-0.4.12-5.el6.x86_64.rpm 
warning: sysbench-0.4.12-5.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
error: Failed dependencies:
 libmysqlclient_r.so.16()(64bit) is needed by sysbench-0.4.12-5.el6.x86_64
 libmysqlclient_r.so.16(libmysqlclient_16)(64bit) is needed by sysbench-0.4.12-5.el6.x86_64
 libpq.so.5()(64bit) is needed by sysbench-0.4.12-5.el6.x86_64

# rpm -ivh --nodeps sysbench-0.4.12-5.el6.x86_64.rpm 
warning: sysbench-0.4.12-5.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing...                ########################################### [100%]
   1:sysbench               ########################################### [100%]

Great, we have sysbench installed now, but it doesn't work:

# sysbench
sysbench: error while loading shared libraries: libmysqlclient_r.so.16: cannot open shared object file: No such file or directory

And here comes the trick, quoting cPanel support:

I can't guarantee that this will work, however the file you need is located at /usr/lib64/libmysqlclient_r.so. You may try creating a symlink from the missing file to this file to see if that resolves this issue.

# ln -s /usr/lib64/libmysqlclient_r.so /usr/lib64/libmysqlclient_r.so.16

Please note that the version installed is /usr/lib64/libmysqlclient_r.so.18 while you need /usr/lib64/libmysqlclient_r.so.16 so there is a chance that this may not work with this library, however if it will work the above should do it.

We are not done yet though, because after symlinking it still doesn't work:

# ln -s /usr/lib64/libmysqlclient_r.so /usr/lib64/libmysqlclient_r.so.16
# sysbench
sysbench: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory

But that's easy to solve:

root@three [~]# yum whatprovides libpq.so.5
Loaded plugins: downloadonly, fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
postgresql-libs-8.4.18-1.el6_4.i686 : The shared libraries required for any PostgreSQL clients
Repo        : base
Matched from:
Other       : libpq.so.5



root@three [~]# yum install postgresql-libs
Loaded plugins: downloadonly, fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.nluug.nl
 * epel: ftp.nluug.nl
 * extras: ftp.nluug.nl
 * updates: ftp.nluug.nl
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package postgresql-libs.x86_64 0:8.4.18-1.el6_4 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================
 Package                                                    Arch                                              Version                                                     Repository                                       Size
================================================================================================================================================================================================================================
Installing:
 postgresql-libs                                            x86_64                                            8.4.18-1.el6_4                                              base                                            201 k

Transaction Summary
================================================================================================================================================================================================================================
Install       1 Package(s)

Total size: 201 k
Installed size: 624 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : postgresql-libs-8.4.18-1.el6_4.x86_64                                                                                                                                                                        1/1 
  Verifying  : postgresql-libs-8.4.18-1.el6_4.x86_64                                                                                                                                                                        1/1 

Installed:
  postgresql-libs.x86_64 0:8.4.18-1.el6_4                                                                                                                                                                                       

Complete!

And now we are finally happy. There is still a warning on stderr, but seems to work fine (I didn't test the mysql benchmarks):

# sysbench --version
sysbench: /usr/lib64/libmysqlclient_r.so.16: no version information available (required by sysbench)
sysbench 0.4.12

gentoo: htpasswd: BCRYPT algorithm not supported on this platform

I'm making a note of it, as a google search didn't find the straightforward solution. In case you get the error "htpasswd: BCRYPT algorithm not supported on this platform" when using htpasswd -B on gentoo. The solution is to recompile apr-util with the openssl USE flag. Unfortunately it's switched off by default.

echo -e 'dev-libs/apr-util\topenssl\t# htpasswd: BCRYPT algorithm not supported on this platform' >> /etc/portage/package.use

Automatically connect to any unsecured Wi-Fi AP from Android

For a long time I wanted my phone to just connect to any SSID which has no password set. I'm travelling often and don't have mobile internet most of the time, and I want my phone to sync emails and stuff whenever I get to the vicinity of a publicly available Wi-Fi signal without touching my phone. Google didn't help, but I finally figured it out accidentally when searching for a solution to extract cleartext Wi-Fi passwords from my phone.

You will need to edit the file /data/misc/wifi/wpa_supplicant.conf and add the following lines:

# This is a network block that connects to any unsecured access point.
# We give it a low priority so any defined blocks are preferred.
network={
        key_mgmt=NONE
        priority=-9
}

I recommend switching off Wi-Fi first, because this file gets updated automatically. There is a drawback though, on the AP list, you won't see which SSID you are connected to, and QuickSettings will say "Connected to (null)". But it works.

PS: You need a rooted phone for this, the wpa_supplicant.conf is not writable (and not even readable) otherwise.

How to install sun java 5 on Ubuntu 10.04 Lucid

We sometimes have requests to install the sun java 5 JDK on recent Ubuntu distributions. The problem is that Ubuntu 9.04 Jauty was the last release of Ubuntu to include sun java5 packages. So here is how I do it:

First, download the required packages:

wget http://de.archive.ubuntu.com/ubuntu/pool/multiverse/s/sun-java5/{ia32-sun-java5-bin_1.5.0-22-0ubuntu0.8.04_amd64.deb,sun-java5-bin_1.5.0-22-0ubuntu0.8.04_amd64.deb,sun-java5-demo_1.5.0-22-0ubuntu0.8.04_amd64.deb,sun-java5-jdk_1.5.0-22-0ubuntu0.8.04_amd64.deb,sun-java5-jre_1.5.0-22-0ubuntu0.8.04_all.deb}


Now, install them with dpkg -i:

dpkg -i ia32-sun-java5-bin_1.5.0-22-0ubuntu0.8.04_amd64.deb sun-java5-bin_1.5.0-22-0ubuntu0.8.04_amd64.deb sun-java5-demo_1.5.0-22-0ubuntu0.8.04_amd64.deb sun-java5-jdk_1.5.0-22-0ubuntu0.8.04_amd64.deb sun-java5-jre_1.5.0-22-0ubuntu0.8.04_all.deb


You will get unresolved dependency problems, but we are going to fix them:

aptitude full-upgrade


Keep pressing n until aptitude doesn't want to remove or downgrade any packages, only install. The solution will look like this:

The following actions will resolve these dependencies:

Install the following packages:
ia32-libs [2.7ubuntu25 (lucid)]
lib32asound2 [1.0.22-0ubuntu7 (lucid)]
lib32bz2-1.0 [1.0.5-4 (lucid)]
lib32gcc1 [1:4.4.3-4ubuntu5 (lucid)]
lib32ncurses5 [5.7+20090803-2ubuntu3 (lucid)]
lib32stdc++6 [4.4.3-4ubuntu5 (lucid)]
lib32v4l-0 [0.6.4-1ubuntu1 (lucid)]
lib32z1 [1:1.2.3.3.dfsg-15ubuntu1 (lucid)]
libc6-i386 [2.11.1-0ubuntu7.1 (lucid-updates, lucid-security)]
odbcinst [2.2.11-21 (lucid)]
odbcinst1debian1 [2.2.11-21 (lucid)]
unixodbc [2.2.11-21 (lucid)]

Score is -58


After accepting the solution and installing the missing packages, you will have both 32 and 64bit sun java5 installed:

# update-alternatives --list java
/usr/lib/jvm/ia32-java-1.5.0-sun/jre/bin/java
/usr/lib/jvm/java-1.5.0-sun/jre/bin/java
/usr/lib/jvm/java-6-openjdk/jre/bin/java


And you can use one of these values, as JAVA_HOME:

# java 1.5 32bit:
JAVA_HOME="/usr/lib/jvm/ia32-java-1.5.0-sun"
# java 1.5 64bit:
JAVA_HOME="/usr/lib/jvm/java-1.5.0-sun"
# java 1.6 64bit:
JAVA_HOME="/usr/lib/jvm/java-6-sun"


How to install open source vmware tools on Ubuntu lucid

I did not find a simple and straightforward solution anywhere, so here it is:

apt-get install --no-install-recommends open-vm-source module-assistant
module-assistant prepare open-vm
module-assistant auto-install open-vm
apt-get install --no-install-recommends open-vm-tools

This approach has a nice side effect: when you update your kernel, the modules gets recompiled too.

Maximize rdesktop on startup

On my netbook, I use different displays, and I always had a difficulty when choosing the geometry option. So I wrote this as /usr/local/bin/rdesktop

#!/bin/bash

# Determine the maximum window size:
declare -i w="$(xprop -root | awk -F'[, ]' '/^_NET_WORKAREA/ { print $7 }')"
declare -i h="$(xprop -root | awk -F'[, ]' '/^_NET_WORKAREA/ { print $9 }')"
command -p rdesktop -D -g "${w}x${h}" "$@"

OpenSolaris guest on Linux KVM

I want a ZFS fileserver for my home network. Something similar to "A Home Fileserver using ZFS". The main difference is I cannot afford a dedicated ZFS server so I have to use virtualization to run Solaris. On linux the first choice is to use KVM, and I also found there is an OpenSolaris JeOS, which is a minimal OpenSolaris installation to use as a guest OS. So download, unpack and start it. Well, it's not as easy as it sounds.

The first problem is that early in the boot process the Solaris guest does a kernel panic. Luckily there is a workaround: you have to upgrade the OS. But since the kernel panics, you have to use VirtualBox to start your guest, upgrade from there, and then you can move back the image to KVM, and start your Solaris guest.

Second problem comes here, Solaris doesn't have a driver for virtio-net, and the only other gigabit nic in KVM is e1000. Solaris does have a driver for that, but unfortunately something is broken and the guest can only send packets, but for some reason cannot receive traffic. So, to have a working network connection you have to use the rtl8139 nic emulation which is 100Mbit, and that's really slow for a file server. But I decided to live with it, until it gets fixed (either in solaris or in qemu-kvm).

Here comes the third problem: Solaris doesn't have a driver for virtio storage, nor for the scsi card KVM emulates. So the only other option is to use IDE storage, but that is limited to 4 devices. I want to use ZFS on 8 HDDs plus there has to be a root zpool, so IDE is not enough. The same situation is described in this post: KVM, SCSI & OpenSolaris.

Now we have two options. Use ide only for the root device, and use iSCSI for the ZFS HDDs. When you take into account that you can have only 100Mbit network interfaces this is really bad. Sure, I could dedicate a virtual NIC for every HDD, but this would only unnecessarily complicate things.

The second option is to ditch KVM and use VirtualBox for OpenSolaris. This would not have the limitations described above, because under VirtualBox the gigabit network works fine, and there is also an option to emulate a SATA controller for the guest, this way I can give the guest 8 HDDs. And this is what I'm going to do next weekend.

So, the result from this weekend is: OpenSolaris - Bud 1:0
:(

The difference between rc_need, rc_use, rc_before and rc_after in gentoo

While trying to figure out why libvirtd starts before drbd (which is not good for me, because i have virtual machines on drbd, and autostarting virtual machines does not work this way) i wanted to know what is the difference between gentoo init script dependency keywords "need" "use" "before" and "after". Unfortunately i found no documentation about it anywhere, so i had to find it out myself:

• rc_need: A service which uses another service will only run if the another service is running. For example nfs needs portmap, so if you start nfs, then portmap will start automatically before it. If you stop portmap, nfs will be stopped before it. If you restart portmap, nfs will be stopped before, and started after portmap.
• rc_use: A service which uses another service will be started after the another service during bootup, and stopped before the another service during shutdown. Example: sshd uses logger and net, so sshd will be started before logger starts, and stopped before logger stops. But if you restart the logger it will not restart the ssh daemon.
• rc_after: The service will be started after another service during bootup, but during shutdown it does not need to be stopped before the other service stops. Example: nfs will start after quota.
• rc_before: The service will be started before another service during bootup. Example: iptables will start before network comes up.

I do not know what is the effect of rc_after and rc_before during shutdown.

Update: I found the documentation for baselayout-2: man 8 runscript

Enabling KSM on gentoo

Update: linux-headers-2.6.32 is now available in gentoo, i have updated the post to use that package, instead of patched 2.6.30. The patched linux-headers-2.6.30 is still available in the ebuild bundle if somebody needs it.

Last week I finally created some virtual machines on my home server (virt-manager is a pain to work with compared to VirtualBox), and I wanted to make sure Kernel Samepage Merging is working. I have kernel 2.6.32, CONFIG_KSM enabled in kernel, and qemu-kvm has support for it. But KSM didn't work, because /sys/kernel/mm/ksm/pages_sharing always returned 0. It turns out, that to get it working, linux-headers and also glibc needs to support KSM. KSM support appeared in 2.6.32, so you need linux-headers-2.6.32, which is now available in portage, see gentoo bug #297755. So let's make KSM work on gentoo:

First download into your local overlay two ebuilds I made for you:

• sys-libs/glibc-2.11-r1 with KSM patch from piavlo
• app-emulation/ksm package has an initscript to enable ksm, and ksmtuned daemon, all from fedora qemu rpm


cd /var/tmp/
wget -O gentoo-ksm-ebuilds.tbz2 http://sites.google.com/site/buddsite/gentoo-ksm-ebuilds.tbz2
tar xjf gentoo-ksm-ebuilds.tbz2 -C /usr/local/portage/
(Make sure you have something like PORTDIR_OVERLAY="/usr/local/portage" in your /etc/make.conf)

Unmask the packages we want to emerge (binutils is a dependency for glibc):
echo '~sys-kernel/gentoo-sources-2.6.32' >> /etc/portage/package.keywords
echo '~sys-kernel/linux-headers-2.6.32' >> /etc/portage/package.keywords
echo '=sys-libs/glibc-2.11-r1' >> /etc/portage/package.keywords
echo '=sys-devel/binutils-2.20' >> /etc/portage/package.keywords
echo 'app-emulation/ksm' >> /etc/portage/package.keywords

Now emerge linux-headers and glibc with the KSM patch:
emerge -v1 linux-headers glibc

After this, you need to recompile qemu-kvm too. Plus emerge the ksm ebuild, because it has a nice initscript to enable ksm:
emerge -v1 qemu-kvm ksm

Now enable KSM:
/etc/init.d/ksm start

Start some virtual machines, and examine how much memory are you saving with KSM:
echo "KSM pages shared: $(( $(cat /sys/kernel/mm/ksm/pages_sharing) * $(getconf PAGESIZE) / 1024 / 1024 ))Mb"

You can also try out the ksmtuned daemon from fedora:
/etc/init.d/ksmtuned start

Using proxy auto config to get around the "squid does not know IPv6 yet" limitation

I like making stuff which just works from the users perspective, and they dont have to care about how it works. But i do not like transparent proxies, diverting HTTP traffic without the applications knowing about it just does not seem to be right. But I like the idea of proxy autoconfiguration in web browsers. I did set it up a long time ago, with squid proxy, but there were some problems with it:

• When you go to an address with no DNS records associated with, squid displays an error page. When not using a proxy, browsers redirect you to a search page, or display a much nicer error. (I'm really not into redesigning squid error pages.)


• IPv6. Squid does not know about IPv6 yet.

Well, actually squid does support IPv6, in version 3.1, but as of this writing it is still RC. The most important part of the Internet is the Web, so i'm reluctant to use RC. But we could use the PAC file, which is just a javascript code to not use proxy for IPv6 sites. I wanted to make this work when I first created the proxy autoconfig file, but I could not find a good way to test how it works, so my script always returned the same string: "PROXY proxy:3128; DIRECT". This means: use the proxy server at proxy:3128, and if that fails, use direct connection. Yesterday I stumbled upon pactester, so I tought I'll give it a try.

I wanted the following funcionality:

• If the hostname is not resolvable, use direct.


• If the hostname resolves only to IPv4 addresses, use proxy


• If the hostname resolves only to IPv6 addresses, use direct


• If the hostname resolves to both IPv4 and IPv6 address, use direct.

First, since I use gentoo, I wanted to create an ebuild for pactester. The packaging of pactester is a bit ugly, you'll get 1Mb of bundled dependencies and 15kb of useful code. I did put it into /usr/local/portage/net-proxy/pactester/.

The wpad.dat file should have a function FindProxyForURL(url, host), and newer Internet Exploer also supports a function FindProxyForURLEx(url, host). Here are some things to know:

• Internet Explorer 8 calls the function FindProxyForURLEx.


• pactester and Google Chrome are using the FindProxyForURL function


• There are additional functions which you can use in your script, see pac_utils.js


• There are even more additional functions for you to use in FindProxyForURLEx, see proxy_resolver_v8.cc


• in pactester the function dnsResolveEx() is not available


• Google Chrome and Internet Explorer gives different results when calling dnsResolve()

So with a little experimentation I wrote the following proxy autoconfiguration script: wpad.dat

I found the following links useful:
http://kate-ward.blogspot.com/2008/11/im-up-and-running-with-ipv6.html
http://users.telenet.be/mydotcom/library/network/pac.htm

Accessing services behind NAT on public address

We have a gateway with haproxy+tproxy. In the haproxy.cfg we have transparent proxying:

source 0.0.0.0 usesrc clientip


Behind the gateway there are several NAT-ed hosts, acting as backends for haproxy. Everythong works fine, but when we want to access a website served by haproxy from the NAT-ed hosts, we get an 503 error, after a timeout.

The problem is the following:
Let there be host A, with ip address 172.21.0.2. This hosts wants to connect to the site hosted on B (172.21.0.3). This website has an address of B.example.com, which resolves to the public IP address on the gateway. The gateway receives a packet from A, and forwards it to the backend server B, but because of the tproxy setup, haproxy keeps the source IP address of A. Now B receives a packet from A and sends a response to A's IP address. But because they are on the same subnet, the packet will reach A directly and haproxy won't see it. A now received a packet from B (172.21.0.3), but he is expecting a packet from the public interface ip address, so he discards it and keep trying to connect. The gateway cannot see a reply from B either, so after a timeout haproxy sends back an 503 error reply to A.

After some trial and error I remembered a presentation from Kadlecsik Jozsef (in hungarian), where he also talks about how to address this problem. He talks about several solutions, and the NETMAP one sounds good for our setup.

So on the gateway we issue the following command:

iptables -t nat -I POSTROUTING -o eth1 -s 172.21.0.0/16 -j NETMAP --to 172.31.0.0/16

And voila! Everything works.

Some explanation about the command: The eth1 interface is the LAN interface, the 172.21.0.0/16 is the LAN subnet, and the 172.31.0.0/16 is an imaginary unused subnet. Packets originating from the LAN and going back to the same LAN will now have a source IP address from the 172.31.x.x range. And because LAN host don't have a route for this subnet, they will route their answer back through the default gateway, which will map the imaginary addresses back to their original.

But the same iptables command with the NETMAP target can be used on any other NAT-ed LAN. This way port forwarding from the public address can be used inside from the LAN too.

Update: There is a slight issue with the above command. Packets originating from the gateway machine will have their IP addresses NETMAP-ed too, which is not pretty. But we can fix that too:

iptables -t nat -I POSTROUTING -m addrtype ! --src-type LOCAL -o eth1 -s 172.21.0.0/16 -j NETMAP --to 172.31.0.0/16


Slowing down ssh brute-force attempts

Once I was attending a presentation about exim. And I heard a good configuration idea: when we decide that an incoming mail is a spam, and we will not receive it, we do not send an error back to the client right away. We delay the error message for a long time (for example 60 seconds). Why is this useful? We already know the client wants to send junk mail, by keeping the connection open we are slowing his rate of sending down, he won't move on to try other servers so soon.

SSH brute force attacks are common, and everybody can see them in their logs. I wanted to apply the same principle, to slow down ssh brute force attacks. After googling around and not finding a solution I tried to find it myself. The following will be gentoo specific, I haven't tried this on other distributions yet.

Somewhere I read that this should be done in pam, which sounds reasonable. So I tried searching for "pam delay", and found pam_delay.so module, but it's not available in gentoo. But i got a hunch:

grep delay /etc/pam.d/*
/etc/pam.d/samba:auth required pam_smbpass.so nodelay
/etc/pam.d/samba:password required pam_smbpass.so nodelay smbconf=/etc/samba/smb.conf

Dead end, but:

locate delay | grep pam
/lib/security/pam_faildelay.so
/usr/share/doc/pam-1.0.4/modules/README.pam_faildelay.bz2
/usr/share/man/man3/pam_fail_delay.3.bz2
/usr/share/man/man8/pam_faildelay.8.bz2

Sounds good. Read the man page. So into which file to put it? My first idea was to put it into /etc/pam.d/system-remote-login, but this is a hard link to system-local-login, and I wanted to leave that alone. It leaves us with /etc/pam.d/sshd which now looks like this:

# set fail delay to 60 sec:
auth optional pam_faildelay.so delay=60000000
auth include system-remote-login
account include system-remote-login
password include system-remote-login
session include system-remote-login

Note that I only added the auth optional line, the include system-remote-login lines were already there.

Introduction Post

I'm a (mostly) linux system administrator. There are some things I cannot find an answer for by googling, but I figure them out myself. So I decided to write them down so others can find it. It will be mainly a technical blog. Please excuse my english, I'm not a native english speaker, and feel free to correct any mistakes I make.