Compiling Mozilla’s SpiderMonkey 1.8.5 and building an RPM out of it for CentOS 6.x

Hey Squirrels,

We recently came across a situation where we wanted to install CouchDB on CentOS 6.x

As you can imagine from the title of this post, one of the requirements was to install SpiderMonkey 1.8.5 – “Mozilla’s JavaScript engine written in C/C++“.

Here’s how we compiled it and made an RPM package out of it – you can find this package in our Squirrel5 RPM repository here.

1. Create a target directory – we’ll use this as a target for the files that will make up our RPM package.

mkdir /target

2. Get the prerequisites to compile the package:

yum -y install make gcc gcc-c++ perl zip rpm-build

3. Download, configure, compile the source and install it to our target directory

cd /tmp
tar -xzf js185-1.0.0.tar.gz
cd js-1.8.5/js/src/
./configure --prefix=/target
time make -j`nproc`
# this took about 5 minutes on our 2 core VPS
make install

4. Now the source is compiled and installed under /target – let’s go take a look:

cd /target/lib
ls -al
[root@squirrel5test lib]# ls -al
total 9040
drwxr-xr-x 3 root root    4096 Dec 12 05:28 .
drwxr-xr-x 5 root root    4096 Dec 12 05:28 ..
-rwxr-xr-x 1 root root 5571078 Dec 12 05:24 libmozjs185-1.0.a
lrwxrwxrwx 1 root root      30 Dec 12 05:28 -> /target/lib/
lrwxrwxrwx 1 root root      32 Dec 12 05:28 -> /target/lib/
-rwxr-xr-x 1 root root 3667244 Dec 12 05:24
drwxr-xr-x 2 root root    4096 Dec 12 05:28 pkgconfig

5. See those symlinks? they’re not going to work ! if we pack this up the symlinks will keep pointing to a /target directory which will no longer exist. We’ll delete the symlinks and replace them with relative ones:

rm -f
rm -f
#recreate the symlinks
ln -s
ln -s

So now the output of ls -al should look like this:

[root@squirrel5test lib]# ls -al
total 9040
drwxr-xr-x 3 root root    4096 Dec 12 06:19 .
drwxr-xr-x 5 root root    4096 Dec 12 06:18 ..
-rwxr-xr-x 1 root root 5571078 Dec 12 06:18 libmozjs185-1.0.a
lrwxrwxrwx 1 root root      18 Dec 12 06:19 ->
lrwxrwxrwx 1 root root      20 Dec 12 06:19 ->
-rwxr-xr-x 1 root root 3667244 Dec 12 06:18
drwxr-xr-x 2 root root    4096 Dec 12 06:18 pkgconfig

6. Now install Ruby and fpm – we need those to create the RPM package:

yum -y install
yum -y install ruby
gem install fpm --no-ri --no-rdoc

7. Use fpm to create an RPM out of the current directory:

cd /target
fpm --verbose -v 0.1 -n libmozjs185-dev -s dir -t rpm .=/usr/

You’ll see output like this:

Executing(%clean): /bin/sh -e /tmp/rpm-tmp.UIOsJQ {:level=>:info}
Created package {:path=>"libmozjs185-dev-0.1-1.x86_64.rpm"}

8. You can install the RPM package by typing:

yum -y install /target/libmozjs185-dev-0.1-1.x86_64.rpm



An introduction to lsyncd

Hello squirrels,

Today we’re going to do a gentle introduction to lsyncd. lsyncd is a file synchronizer. Many folks confuse lsyncd with it’s cousin rsync. The difference is that rsync is a one-off solution (unless you automate it via cron etc.). lsyncd will use inotify to detect changes to files and will launch rsync for you. That’s right – lsyncd uses rsync.

Here’s how to set it up:

1. Get the lsyncd dependencies (one of them is rsync so we install the IUS repository to get the most recent version:

yum install -y
yum install -y rsync31u
yum install -y cmake lua-devel gcc gcc-c++ openssh-clients

2. Get the lsyncd source code – at the time of writing the latest version was 2.1.6:

cd /tmp
curl -LO
tar -xzf release-2.1.6.tar.gz
cd lsyncd-release-2.1.6/

3. Now compile lsyncd:

cmake .
make install

Also copy the manual page in place in case you need it:

cp doc/lsyncd.1 /usr/local/share/man/man1/

This will install lsyncd under /usr/local/bin.

4. Check the version:

lsyncd -version
Version: 2.1.6

5. Create an SSH key pair to use specifically for lsyncd (optional but good idea)

ssh-keygen -f ~/.ssh/squirrel5-lsyncd -N ''

6. Also add the key in your authorized_keys file so that you can still ssh in after the sync is done

cat /root/.ssh/ >> /root/.ssh/authorized_keys

7. Now add a config file under /usr/local/etc/lsyncd.conf:

In a previous post we showed how to use rsync to migrate CentOS from one machine to another. Let’s do the exact same thing using rsync – for reference here’s the rsync command we used:

time /usr/bin/rsync -aHAXv --delete-after --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/etc/sysconfig/network-scripts/*"} / root@

This is what the exact same thing would look like in lsyncd:

settings {
	log = all,
        logfile    = "/var/log/lsyncd.log",
        statusFile = "/tmp/lsyncd.status",
        pidfile    = "/var/run/",
        nodaemon   = false,

        exclude={ "dev/*", "proc/*", "sys/*","tmp/*","run/*","mnt/*"
                ,"media/*", "lost+found", "etc/sysconfig/network-scripts/*" },
        rsync = {
                compress = false,
                checksum = true,
                 _extra = {"-e", "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /root/.ssh/squirrel5-lsyncd -l root"}

As you can see the above includes the excluded directories and SSH options such as which private key to use and which user to connect as.

8. Start lsyncd by pointing it to the configuration file and setting the -nodaemon option so we can see all of it’s output:

lsyncd /usr/local/etc/lsyncd.conf -nodaemon


If you get an error like this:

06:05:41 Error: Terminating since out of inotify watches.
Consider increasing /proc/sys/fs/inotify/max_user_watches

Check what the number is currently:

cat /proc/sys/fs/inotify/max_user_watches


sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

set it to 15 million:

sysctl fs.inotify.max_user_watches=15000000

9. We’re going to control lsyncd using monit – install it with:

yum install -y monit
chkconfig monit on && chkconfig --list monit

10. Add a very simple configuration for lsyncd in /etc/monit.d/lsyncd:

check process lsyncd pidfile /var/run/
 start program = "/usr/local/bin/lsyncd /usr/local/etc/lsyncd.conf"
 stop program  = "/bin/kill `/bin/cat /var/run/`"

11. Start lsyncd:

monit start lsyncd

12. Monitor it’s status:

monit status lsyncd
The Monit daemon 5.14 uptime: 4m 

Process 'lsyncd'
  status                            Running
  monitoring status                 Monitored
  pid                               1929
  parent pid                        1
  uid                               0
  effective uid                     0
  gid                               0
  uptime                            1m 
  children                          2
  memory                            2.3 MB
  memory total                      6.1 MB
  memory percent                    0.2%
  memory percent total              0.6%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Sat, 03 Dec 2016 20:05:27


If the output you get looks like:

monit status lsyncd
The Monit daemon 5.14 uptime: 2m 

Process 'lsyncd'
  status                            Initializing
  monitoring status                 Initializing
  data collected                    Sat, 03 Dec 2016 20:01:08

Don’t worry about it – give monit a couple more minutes and it will get the status 🙂


Creating an SSL Certificate Authority (CA), signing certificates and authenticating using a client certificate

Greetings Squirrel army,

We’ve been looking into writing a quick post on SSL Certificate Authorities for quite some time, so here’s our blog post on how to get this done. We’re also going to take a look at client authentication using a certificate.

Please note that this post is based on the incredible post from the DataCenter Lords called “Creating Your Own SSL Certificate Authority (and Dumping Self Signed Certs)” – make sure to visit that one for more details on how the authentication works.

1. Create a directory that we’ll use to store our Root certificates and enter the directory

mkdir -pv /squirrel5-CA-root
cd /squirrel5-CA-root

2. Create a key for the CA – (Protip: NEVER SHARE THIS!)

openssl genrsa -out rootCA-NEVER_SHARE.key 2048

3. Use the key to create your CA certificate – this one (rootCA.pem) can be shared to the workstations you want to trust your new CA

openssl req -x509 -new -nodes -key rootCA-NEVER_SHARE.key -sha256 -days 1024 -out rootCA.pem

In the prompt that you’ll get the only line that really matters is the Common Name (CN):

"Common Name (eg, your name or your server's hostname) []"

In this example we’ll answer everything with “Enter” except the Common Name which we will name “Squirrel5-CA

4. Create a private key for one of your servers:

openssl genrsa -out server1.key 2048

5. Create a Certificate Signing Request (CSR) for your first server:

openssl req -new -key server1.key -out server1.csr

Just like above, the only line that matters here is the “Common Name” – the common name must be a FQDN or an IP address, in this example we’ll use

6. And now for the most important bit – use the rootCA-NEVER_SHARE.key to SIGN the new certificate and also tell it what CA to use the “-CA rootCA.pem” bit:

openssl x509 -req -in server1.csr -CA rootCA.pem -CAkey rootCA-NEVER_SHARE.key -CAcreateserial -out server1.crt -days 500 -sha256

For server1 we now have the following files, let’s take a look:

server1.crt <--- server1 cert (never share) 
server1.csr <--- certificate signing request - public
server1.key <--- your private key (never share)

7. Make a PEM out of the server1 certs:

cat server1.key server1.crt > server1.pem

The server1.pem goes to the server (In Hiawatha this means TLScertFile = /root/server1.pem)

8. Install the Squirrel5 repository and install Hiawatha:

yum -y install
yum -y install hiawatha

9. Configure Hiawatha

This is the Hiawatha config (/etc/hiawatha/hiawatha.conf) we’re using in this example (we’ve stripped out anything not necessary on purpose) :

Binding {
        Port = 443
        # The next line uses the PEM file we created in step 7
        TLScertFile = /squirrel5-CA-root/server1.pem 

Hostname =
WebsiteRoot = /var/www/hiawatha
StartFile = index.html

VirtualHost {
        Hostname =
        WebsiteRoot = /var/www
        RequireTLS = yes
        AccessLogfile = /var/log/server1.log
        ErrorLogfile = /var/log/server1-error.log

So what is the above doing? The above tells Hiawatha that:

“The site (Hostname = needs to be accessed over TLS (RequireTLS = yes)”

Let’s also throw a sample HTML page so we can get some output:

echo "<html><title>the squirrels...are watching</title></html>" > /var/www/index.html

10. Restart Hiawatha to apply the changes:

/etc/init.d/hiawatha restart

11. Now test with curl:


I got this error:

curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here:

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

This is because our server doesn’t have our new CA key in it’s standard location (/etc/pki/ca-trust/source/anchors/) – let’s point curl manually to the CA cert according to the instructions from that error message:

curl -I --cacert /squirrel5-CA-root/rootCA.pem


curl --cacert /squirrel5-CA-root/rootCA.pem
<html><title>the squirrels...are watching</title></html>

Awesome that works!

12. OK, but how do I make this permanent so I don’t have to tell curl every time?

Copy your PEM to /etc/pki/ca-trust/source/anchors/

cp /squirrel5-CA-root/rootCA.pem /etc/pki/ca-trust/source/anchors/

And update the system’s CAs:

update-ca-trust extract

I got the following message:

update-ca-trust: Warning: The dynamic CA configuration feature is in the disabled state

Enable the dynamic CA configuration (note: this command has no output) :

update-ca-trust force-enable

Try accessing the site again, but this time don’t specify the CA:

<html><title>the squirrels...are watching</title></html>

13. Very cool, but what about authentication?

Good point! We can edit Hiawatha’s configuration so that it demands that you provide the client’s certificate – here’s how:

  • Edit the Hiawatha configuration (/etc/hiawatha/hiawatha.conf) and edit the Binding section:
Binding {
        Port = 443
        # The next line uses the PEM file we created in step 7
        TLScertFile = /squirrel5-CA-root/server1.pem
        # Require a client certificate that has been signed by the following CA
        RequiredCA  = /squirrel5-CA-root/rootCA.pem
  • Restart Hiawatha:
/etc/init.d/hiawatha restart
  • Try connecting again as before:
curl -I


curl -I
curl: (35) NSS: client certificate not found (nickname not specified)
  • Try again, but this time offer the client certificate:
curl --cert /squirrel5-CA-root/server1.pem
<html><title>the squirrels...are watching</title></html>

There we go – authenticated to the server by providing the client certificate.


Collectd, Graphite and Grafana on CentOS 6.x

Hi Squirrels,

Monitoring dashboards are the cool thing nowadays and their leader is without doubt Grafana and for good reason – it’s easy to install, user intuitive and it’s beautiful.

However, Grafana doesn’t work on it’s own – it needs a data source to read data from. It’s compatible with multiple but the most popular seems to be Graphite.

In this post we’re going to look into what it took to install Graphite and Grafana on CentOS 6.x and then feed data to Graphite from collectd. To make things easier on us we’re going to use our own created collectd RPM.

1. Install Python 2.7 for CentOS using the IUS (Inline with Upstream Stable) repository

yum install -y
yum install -y python27 python27-devel python27-pip

2. Install a bunch of other dependencies to make our life easier:

yum install -y git gcc libffi-devel autoconf bitmap-fonts cairo cairo-devel openldap-devel nc

3. Install Carbon – Carbon is the listener daemon of Graphite – it receives the metrics and stores them in a database

cd /tmp
tar -xzf 0.9.15.tar.gz
rm -f 0.9.15.tar.gz 
cd carbon-0.9.15/
pip2.7 install -r requirements.txt
python2.7 install

4. Now get the Graphite source and install it’s dependencies:

cd /tmp
tar -xzf 0.9.15.tar.gz
rm -f 0.9.15.tar.gz
cd graphite-web-0.9.15/
pip2.7 install -r requirements.txt
pip2.7 install python-ldap

5. Make sure that all dependencies are met:

Test with:

python2.7 ./ 

Expected output:

All necessary dependencies are met.
All optional dependencies are met.

6. Install Graphite:

python2.7 install

7. Let’s begin configuring Graphite – first move the templates into their expected locations:

cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
cp /opt/graphite/webapp/graphite/ /opt/graphite/webapp/graphite/
cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi

8. Edit /opt/graphite/webapp/graphite/ and find the line:

#TIME_ZONE = 'America/Los_Angeles'

change it to:

TIME_ZONE = 'America/Los_Angeles'

Then also find the line:


change it to:

SECRET_KEY = 'random_password'

Where random_password can be a password you have generated with something like openssl rand -hex 10.

9. Setup the database:

cd /opt/graphite/webapp/graphite
python2.7 syncdb

You’ll get some questions, the first one is:

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no):

Answer: yes
Next question:

Username (leave blank to use 'root'):

I took the default, root – just press enter.
Next Question:

E-mail address:

You need to give a valid e-mail address, for our example I used
One more question:


You’ll need to enter a password twice.

When done you should see output like this:

Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

10. Now we can start Carbon:

/opt/graphite/bin/ start

You should get output like this:

Starting carbon-cache (instance a)

You can check that carbon is listening on TCP port 2003 with netstat -ntlp | grep 2003

netstat -ntlp | grep 2003
tcp        0      0      *                   LISTEN      1769/python2.7 

11. Setup Apache with mod_wsgi for Python 2.7

Since Graphite is a Django based web application, Apache needs to be installed with mod_wsgi. Here comes the tricky part – you need to make sure that you install the mod_wsgi for Python 2.7 – the one in the CentOS repositories is for Python 2.6 and if you install that a lot of things are going to break.

To do it the right way, we need to install mod_wsgi through pip:

yum -y install httpd httpd-devel
pip2.7 install mod_wsgi
chown -R apache:apache /opt/graphite
cp /opt/graphite/examples/example-graphite-vhost.conf  /etc/httpd/conf.d/graphite.conf

Note: If you see this in the apache logs you got the wrong version of mod_wsgi and the wrong version of Python being loaded by Apache:

grep -i python /var/log/httpd/*
[Mon Sep 12 03:45:54 2016] [notice] Apache/2.2.15 (Unix) DAV/2 mod_wsgi/3.2 &amp;amp;amp;lt;span style="color: #ff0000;"&amp;amp;amp;gt;Python/2.6.6&amp;amp;amp;lt;/span&amp;amp;amp;gt; configured -- resuming normal operations

Now check the Apache module listing:

apachectl -M
httpd: Syntax error on line 221 of /etc/httpd/conf/httpd.conf: Syntax error on line 15 of /etc/httpd/conf.d/graphite.conf: &amp;amp;amp;lt;span style="color: #ff0000;"&amp;amp;amp;gt;Cannot load /etc/httpd/modules/ into server: /etc/httpd/modules/ cannot open shared object file: No such file or directory;

The above error means that Apache did not find the module where it expected it to. That’s an easy fix:

cp /usr/lib64/python2.7/site-packages/mod_wsgi/server/ /etc/httpd/modules
ln -s /etc/httpd/modules/ /etc/httpd/modules/
pip2.7 uninstall -y whitenoise

Now restart Apache:

/etc/init.d/httpd restart

and check your Apache logs again:

/var/log/httpd/error_log:[Mon Sep 12 03:59:04 2016] [notice] Apache/2.2.15 (Unix) DAV/2 mod_wsgi/4.5.6 &amp;amp;amp;lt;span style="color: #ff0000;"&amp;amp;amp;gt;Python/2.7.12&amp;amp;amp;lt;/span&amp;amp;amp;gt; configured -- resuming normal operations

See how now it says Python/2.7.12 ? That’s what we were aiming for.

12. Check the Graphite logs and make sure there are no errors

In my case I saw the following

tail -n 30 /opt/graphite/storage/log/webapp/error.log | grep DatabaseError
[Tue Sep 13 03:36:48 2016] [error] DatabaseError: attempt to write a readonly database
[Tue Sep 13 03:36:48 2016] [error] &amp;amp;amp;lt;span style="color: #ff0000;"&amp;amp;amp;gt;DatabaseError: attempt to write a readonly database&amp;amp;amp;lt;/span&amp;amp;amp;gt;

If you get the error: “DatabaseError: attempt to write a readonly database”

Then you need to edit /opt/graphite/webapp/graphite/

Find this section:

  'default': {
    'NAME': '',

change it to:

  'default': {
    'NAME': '/opt/graphite/storage/graphite.db',

And restart Apache again:

/etc/init.d/httpd restart

13. Access Graphite at http://yourip

Graphite will now be listening on port 80 – please note that this example isn’t meant for production and Graphite is totally unsecured here – you won’t even be prompted for a password.

The graphite web interface looks something like this:


We’re not really going to work with the Graphite interface at all – let’s move right ahead to the next steps to install collectd.

14. Install the Squirrel5 repository:

yum -y install

15. Install collectd

yum -y install collectd collectd-rrdtool

16. Configure collectd

Edit the file /etc/collectd.conf and change the line:

#Hostname    "localhost"


Hostname    ""

Then find the line:

#LoadPlugin write_graphite

change it to:

LoadPlugin write_graphite

Now tell collectd where to send it’s metrics:

find the section:

#<Plugin write_graphite>
#  <Node "example">
#    Host "localhost"
#    Port "2003"
#    Protocol "tcp"
#    LogSendErrors true
#    Prefix "collectd"
#    Postfix "collectd"
#    StoreRates true
#    AlwaysAppendDS false
#    EscapeCharacter "_"
#  </Node>

change it to:

<Plugin write_graphite>
  <Node "squirrel5">
    Host "localhost"
    Port "2003"
    Protocol "tcp"
    LogSendErrors true
    StoreRates false
    AlwaysAppendDS false
    EscapeCharacter "_"

17. Start collectd

/etc/init.d/collectd start

18. Add the Grafana repository

cat &amp;gt; /etc/yum.repos.d/grafana.repo &amp;lt;&amp;lt; EOL

19. Install and start Grafana

yum -y install grafana
chkconfig --add grafana-server
/etc/init.d/grafana-server start

20. Make sure that Grafana is running

Check that it’s listening on port 3000:

netstat -ntlp | grep 3000
tcp        0      0 :::3000                     :::*                        LISTEN      3599/grafana-server

21. Access Grafana

Go to http://yourip:3000 and login with username ‘admin‘, password ‘admin

22. Add a ‘Data Source’

Click on the grafana icon and then on ‘Data Sources.


You should now be seeing the ‘Data Sources‘ menu – click on ‘Add data source


You should now be seeing the ‘Edit data source’ Configuration.

Set it as follows:

Name: Squirrel-Graphite, tick on ‘Default‘, Type should be set to ‘Graphite

Then on the http settings, you need to direct it to http://localhost:80 and set it to ‘proxy‘ access.

Finally click on ‘Save & Test’


23. Add a new dashboard:

Grafana is all about Dashboards. Click on the Grafana icon to add one and then on ‘Dashboards‘ and then on ‘New


24. Once you have your new dashboard, we need to add at least one graph to it.

Click on the green icon on the left, then ‘Add Panel‘ and ‘Graph


  • Click on “select metric“, Select “testing_squirrel5_com
  • Click on “select metric“, Select “memory
  • Click on “select metric“, Select “memory used
  • Finally click on the ‘X‘ to be done!


25. Extra points – set the graphs to automatically refresh every 5 seconds and the graph duration to 1 hour.


Follow steps 24-25 to add as many graphs as you like!


Migrating CentOS from VirtualBox (LVM) to a cloud server (without LVM)

Hi squirrels,

Recently we were faced with the requirement of moving a number of CentOS servers partitioned with LVM to the cloud. There are a lot of tutorials and documentation on the internet on how to do this but we found most of them to not be working or not really applying to our case.

For example, there were a few articles discussing using dd backups to do a block by block transfer of the server over the internet. This seemed reasonable but unfortunately some cloud providers VPS’ do not have LVM partitioned disks and others that do, do not allow attaching multiple drives at the same time (unlike EBS volumes at AWS).

So to summarize, here’s the lsblk output from both servers – both servers were running CentOS, ServerA was running 6.8 but the cloud server was running 6.7.

ServerA = source server, partitioned with LVM
ServerB = target server, no LVM

# this is from serverA - the source server
# note the LVM partitions
NAME                         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                           11:0    1 1024M  0 rom  
sda                            8:0    0    8G  0 disk 
├─sda1                         8:1    0  500M  0 part /boot
└─sda2                         8:2    0  7.5G  0 part 
  ├─vg_centos-lv_root (dm-0) 253:0    0  6.7G  0 lvm  /
  └─vg_centos-lv_swap (dm-1) 253:1    0  816M  0 lvm  [SWAP]
# this is from ServerB - the destination server
# note that there are no LVM partitions
sda      8:0    0  80G  0 disk 
└─sda1   8:1    0  80G  0 part /

As you can see you cannot use dd for the above unless you are also ready to export and import the partition table over as well. This cannot be done without booting from rescue media or a secondary disk as dd will nuke your target filesystem in the process.

Enter rsync.

It looks like this was previously done with rsync by the nice guys over at ArchLinux:

Unfortunately the instructions there didn’t quite work for us for CentOS since that example is talking about Grub2, but CentOS 6.x is still on legacy grub.

Here’s how we did it:

1. Save ServerB’s (target) /etc/grub.cfg and /etc/fstab:

ServerB’s /etc/grub.cfg:

# grub.conf generated by anaconda
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You do not have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /, eg.
#          root (hd0,0)
#          kernel /boot/vmlinuz-version ro root=/dev/sda1
#          initrd /boot/initrd-[generic-]version.img
title CentOS (2.6.32-573.18.1.el6.x86_64)
	root (hd0,0)
	kernel /boot/vmlinuz-2.6.32-573.18.1.el6.x86_64 ro root=UUID=ca3c1585-34a7-4280-b18f-dcd5f61831b0 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM console=ttyS0 console=tty0
	initrd /boot/initramfs-2.6.32-573.18.1.el6.x86_64.img

ServerB’s /etc/fstab:

# /etc/fstab
# Created by anaconda on Wed Jun  3 03:44:43 2015
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
UUID=ca3c1585-34a7-4280-b18f-dcd5f61831b0 /                       ext4    defaults,noatime        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

2. Prepare rsync (skip if you are using password authentication for SSH)

Add your ssh key to ~/.ssh/config. There’s other ways to do this as well but this is the easiest one to tell rsync to use a private key.

Host # replace with ServerB's (target server) IP
	User root    # your SSH username for ServerB
	IdentityFile /root/id_rsa # your SSH private key

3. Prepare rsync on the target (remember: rsync needs to be installed on both the source and target)

yum -y install rsync

NOTE: if you are getting the error:

bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: remote command not found (code 127) at io.c(600) [sender=3.0.6]

this means that rsync is not installed on the REMOTE side.

4. rsync across

time /usr/bin/rsync -aHAXv --delete-after --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/etc/sysconfig/network-scripts/*"} / root@

why are we excluding the above directories?

/dev – special or device files – we want to keep the ORIGINAL ones, do not overwrite those
/proc – a process information pseudo-file system
/sys – hardware hierarchy
/tmp – temporary files
/run – udev runtime data
/mnt – generic mount point
/media – generic mount point for removable media
/lost+found – recovered corrupt files
/etc/sysconfig/network-scripts/ – we want to keep the original networking configuration since we are migrating to the cloud

5. “Fix” /etc/fstab on the target

I quoted “fix” because in this case the “fix” is to simply reuse your old /etc/fstab with a single but important change:

# /etc/fstab
# Created by anaconda on Wed Jun  3 03:44:43 2015
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
# the next line is the only one we changed
/dev/sda1	 	/                       ext4    defaults,noatime        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

Note how the line: UUID=ca3c1585-34a7-4280-b18f-dcd5f61831b0 / ext4 is gone? it’s replaced with /dev/sda1 / ext4 defaults,noatime 1 1

6. Fix /etc/grub.conf

For the new /etc/grub.conf we need:

* ServerA’s (source) /etc/grub.conf – this has been copied over by rsync
* ServerB’s (targets) paths – in this case this means to prefix everything with /boot


title CentOS 6 (2.6.32-642.el6.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-642.el6.x86_64 ro root=UUID=ca3c1585-34a7-4280-b18f-dcd5f61831b0 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM console=ttyS0 console=tty0
initrd /boot/initramfs-2.6.32-642.el6.x86_64.img

7. One last touch – re-install Grub.

If you used the same rsync command as I did above, you’ll notice that /boot was not excluded. This means that ServerB has ServerA’s stage1 and stage2. Note that those are DIFFERENT than /etc/grub.conf.

To re-install grub, go to ServerB (target) and do the following:

Start grub:


Re-install the stage1 and stage2 to (hd0) – that’s the grub notation for the first physical disk. Replace as needed if you have more than one disks.

setup (hd0)

This is the output we got:

setup (hd0)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/e2fs_stage1_5" exists... yes
Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... 27 sectors are embedded.
Running "install /boot/grub/stage1 (hd0) (hd0)1+27 p (hd0,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded

Exit grub:


You should now be ready to go!


Building an RPM package for collectd

Hello Squirrels,

We figured we would save folks some time and put together a quick blog post with the steps needed to create an RPM package for collectd for CentOS 6.x out of the latest sources.

This has been partly based on a few sources we found online so please look at the bottom of this post if you are interested in finding out more about those.

For those who are not familiar with it, collectd, is a daemon for collecting system statistics (cpu, memory etc). The highlights are that it’s built in plain C and as such it’s really fast and lightweight and it also comes with a ton of plugins. It can also be used as a building block for building Grafana dashboards – but more on this later 🙂

1. Get the latest source code – in this example we are going to build an RPM out of collectd 5.5.2.

cd /tmp
wget -c

2. Untar and enter the source directory

tar -xf collectd-5.5.2.tar.bz2
cd collectd-5.5.2

3. Install EPEL, GCC and the RPM build tools:

yum install -y rpm-build yum-utils rpmdevtools epel-release gcc

4. Create the /root/rpmbuild/SOURCES/ since rpmbuild expects it and copy the source code there

mkdir -pv /root/rpmbuild/SOURCES/
cp /tmp/collectd*.bz2 /root/rpmbuild/SOURCES/

5. Change the version number to 5.5.2 (the spec file in my case was for 5.5.1)

sed -i -- 's/5.5.1/5.5.2/g' /tmp/collectd-5.5.2/contrib/redhat/collectd.spec

6. Get any remaining dependencies with yum:

yum-builddep -y /tmp/collectd-5.5.2/contrib/redhat/collectd.spec

7. Build the RPM:

rpmbuild -bb /tmp/collectd-5.5.2/contrib/redhat/collectd.spec

8. The packages are now under /root/rpmbuild/RPMS/x86_64. To install locally do this:

cd /root/rpmbuild/RPMS/x86_64
yum localinstall collectd-* libcollectdclient-*

9. Set collectd to chkconfig to ‘on’ so that it can start on boot automatically:

chkconfig collectd on

10. Before starting the collectd service make sure that you change and uncomment the “#Hostname” line in the /etc/collectd.conf configuration file

11. Finally start collectd:

/etc/init.d/collectd start


Nagios with Hiawatha and PHP-FPM 5.6

Hola ardillas!

This week we’ll look at systems administration gone off the beaten path. We’re all Hiawatha fanatics here at Squirrel5, so when the time came to setup Nagios we were faced with a scary prospect:

“Nagios on CentOS has Apache (httpd) and php as a dependency.”

It’s sad but true – here’s how to confirm what the Nagios dependencies are on CentOS:

yum -y install epel-release
repoquery --requires nagios | egrep -i "httpd|php"

So yes, our life would be much easier if we went with Apache since Apache is mostly preconfigured for it via the CentOS packages, but where’s the fun in that? We want Nagios to work with Hiawatha!

Here’s how we did it:

  1. Make sure the EPEL repo is installed:
yum -y install epel-release

2. Install Nagios and plugins:

yum -y install nagios nagios-plugins nagios-plugins-all

3. Install the REMI repository so we can get PHP-FPM 5.6:

yum -y install

4. Install the Squirrel5 RPM repository to get Hiawatha:

yum -y install

5. Install Hiawatha 10:

yum -y install hiawatha

6. Add a Hiawatha user:

useradd hiawatha -s /sbin/nologin

Very important note: Nagios does not like changing the user it runs under, this is by default ‘nagios’ but we are going to change it anyway 🙂

7. Fix permissions so that Nagios runs under the ‘hiawatha‘ user:

mkdir -pv /var/run/nagios
sed -i -- 's/NagiosUser=nagios/NagiosUser=hiawatha/g' /etc/init.d/nagios
sed -i -- 's/NagiosGroup=nagios/NagiosGroup=hiawatha/g' /etc/init.d/nagios
sed -i -- 's#NagiosRunFile=/var/run/' /etc/init.d/nagios
sed -i -- 's/nagios_user=nagios/nagios_user=hiawatha/g' /etc/nagios/nagios.cfg
sed -i -- 's/nagios_group=nagios/nagios_group=hiawatha/g' /etc/nagios/nagios.cfg
sed -i -- 's#lock_file=/var/run/' /etc/nagios/nagios.cfg
chown -R hiawatha:hiawatha /usr/lib64/nagios 
chown -R hiawatha:hiawatha /var/lib/php
chown -R hiawatha:hiawatha /var/log/nagios &amp;amp;amp;nbsp;
chown -R hiawatha:hiawatha /var/spool/nagios
chown -R hiawatha:hiawatha /var/run/nagios
chown -R hiawatha:hiawatha /etc/nagios
chown -R hiawatha:hiawatha /usr/share/nagios

8. Create a directory for the Hiawatha VirtualHost configurations:

mkdir /etc/hiawatha/conf.d

9. Tell Hiawatha to run under the ‘hiawatha‘ user, recognize the ‘.cgi‘ extension and include files under the /etc/hiawatha/conf.d directory:

echo "ServerId = hiawatha:hiawatha" >> /etc/hiawatha/hiawatha.conf
echo "CGIextension = cgi" >> /etc/hiawatha/hiawatha.conf
echo "Include /etc/hiawatha/conf.d" >>  /etc/hiawatha/hiawatha.conf

10. Now let’s add a VirtualHost for Nagios – we’ll call the file /etc/hiawatha/conf.d/nagios.conf

VirtualHost {
Hostname = $your_ip
WebsiteRoot = /usr/share/nagios/html
StartFile = index.php

AccessLogfile = /var/log/hiawatha/nagios-access.log
ErrorLogfile = /var/log/hiawatha/nagios-error.log
ExecuteCGI = yes
TimeForCGI = 5
UseFastCGI = PHP5
PasswordFile = basic:/etc/hiawatha/users
Alias = /nagios:/usr/share/nagios/html
Alias = /nagios/cgi-bin:/usr/lib64/nagios/cgi-bin

RequireTLS = yes

11. Now let’s install PHP-FPM 5.6 from the Remi repo along with a bunch of PHP extensions:

yum -y install php56 php56-php-gd php56-php-pecl-dom-varimport php56-php-pecl-jsonc php56-php-xml php56-php-pecl-zip php56-php-pecl-crypto php56-php-mcrypt php56-php-intl php56-php-mysqlnd php56-php-fpm

12. Add a PHP-FPM configuration – this needs to go under /opt/remi/php56/root/etc/php-fpm.d/nagios.conf:

; Start a new pool named 'nagios'.
listen = /var/lib/hiawatha/php-fcgi-nagios.sock
user = hiawatha
group = hiawatha

pm = static
pm.max_children = 2
pm.start_servers = 2
pm.min_spare_servers = 3
pm.max_spare_servers = 5
slowlog = /var/log/php-fpm/nagios-slow.log
php_admin_value[error_log] = /var/log/php-fpm/nagios-error.log
php_admin_flag[log_errors] = on

; Set session path to a directory owned by process user
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php

13. Move the example PHP-FPM configuration out of the way:

mv /opt/remi/php56/root/etc/php-fpm.d/www.conf /opt/remi/php56/root/etc/php-fpm.d/www.conf-backup

14. Cool – now tell Hiawatha, how to talk to PHP-FPM – add the file /etc/hiawatha/conf.d/php.conf:

FastCGIserver {
FastCGIid = PHP5
ConnectTo = /var/lib/hiawatha/php-fcgi-nagios.sock
Extension = php

15. Add an SSL configuration for Hiawatha /etc/hiawatha/conf.d/ssl.conf:

Binding {
Port = 443
TLScertFile = /etc/ssl/serverkey.pem

16. Create an SSL self signed certificate:

cd /etc/ssl/
openssl req -subj '/CN=$replace_with_your_ip$/C=US' -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 -keyout serverkey.pem -out server.crt
cat server.crt >> serverkey.pem
rm -f server.crt
chmod 400 serverkey.pem

17. Let’s start verifying configurations to make sure everything’s good to go – start with Hiawatha – type:

service hiawatha check

Sample output:

service hiawatha check
Configuration check via Wigwam...
Using /etc/hiawatha
Reading hiawatha.conf
Reading /etc/hiawatha/conf.d/nagios.conf
Reading /etc/hiawatha/conf.d/php.conf
Reading /etc/hiawatha/conf.d/ssl.conf
No non-fatal errors found in the Hiawatha configuration.

Configuration check via Hiawatha...
Using /etc/hiawatha
Reading hiawatha.conf
Reading /etc/hiawatha/conf.d/nagios.conf
Reading /etc/hiawatha/conf.d/php.conf
Reading /etc/hiawatha/conf.d/ssl.conf
Reading mimetype.conf
Configuration OK.

18. Let’s check PHP-FPM next, type:

service php56-php-fpm configtest

Sample output:

service php56-php-fpm configtest
[30-Aug-2016 04:29:30] NOTICE: configuration file /opt/remi/php56/root/etc/php-fpm.conf test is successful

19. Now let’s test Nagios – type:

service nagios checkconfig

Sample output:

service nagios checkconfig
Running configuration check... OK.

20. Let’s add an authorized user for Hiawatha – this is going to be the username and password you’ll need to use to access the web-based interface:

htpasswd -cb /etc/hiawatha/users nagiosadmin obey_the_acorn

Where ‘nagiosadmin‘ is the username and ‘obey_the_acorn‘ is the password.

You should get output like this:

Adding password for user squirrel_supreme

NOTE: If you change the username from ‘nagiosadmin‘ to anything else, make sure to also make this change in the /etc/nagios/cgi.cfg and to restart nagios after making the change.

21. Now start the services:


service nagios start

Sample output:

service nagios start
Starting nagios: done.


service hiawatha start

Sample output:

service hiawatha start
Starting webserver: Hiawatha


service php56-php-fpm start

service php56-php-fpm start
Starting php-fpm: [ OK ]

Now you should be able to access the Nagios web-based interface at the IP of your host – for example: You’ll get a warning for using a self-signed certificate (this is of course normal) and then you’ll be prompted for your username and password.

This is what you should be looking at now:

Nagios home screen
Nagios home screen



Compiling Ruby from source and building an RPM from it using FPM

We all love Ruby here at Squirrel5. We love everything about it … except installing it.

What’s the big deal with installing Ruby you’ll ask? Well, let’s start from the top.

If you are using CentOS for pretty much everything like we do, you probably know that the most recent version of Ruby you
can find in the repositories is for CentOS 6.x.

Things are better with CentOS 7.x as you can get Ruby directly from the repositories.

Ruby 2.0.0 is probably good enough for most cases, but 1.8.7 is not. How do we go about installing a newer version of Ruby on CentOS 6.x then?

There’s a few ways to install a more modern version of Ruby – you can use RVM, Rbenv, or even the RedHat Software Collections.

Our preferred way is to compile Ruby from source once and then make an RPM package out of it.

Just like installing Ruby, there’s more than one ways to build an RPM package. We like using FPM to create a Ruby RPM, which of course requires Ruby, so we have a chicken and egg situation going.

For this guide we fired up a test VPS using – we recommend using a VPS/server with at least 2 CPU cores so that the compiling goes much faster.

1. Install Ruby 1.9.3 or newer so that FPM can work – I’m going to use RVM to install it:

gpg2 --keyserver hkp:// --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
curl -sSL | bash -s stable --ruby
source /usr/local/rvm/scripts/rvm

2. Check what version of Ruby did RVM install:

rvm list 

rvm rubies

=* ruby-2.3.0 [ x86_64 ]

# =&gt; - current
# =* - current &amp;&amp; default
#&nbsp; * - default

3. Also check by asking Ruby itself:

ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux]

4. Let’s install some other dependencies so we don’t run into trouble later:

yum groupinstall Development tools -y
yum -y install openssl-devel libffi-devel readline-devel gdbm-devel libyaml-devel tcl-devel tk-devel

5. All good – now let’s install the FPM gem:

gem install fpm --no-ri --no-rdoc

6. Now visit and get the source of Ruby that you like – in our case we’re going to download Ruby 2.3.1:

cd /tmp

7. Untar it:

tar -xzf ruby-2.3.1.tar.gz

8. Enter the directory of the source code:

cd ruby-2.3.1

9. Now let’s configure Ruby:

./configure --prefix=/usr

Note the –prefix=/usr — this means that the compiled ruby will expect to live under /usr (which is the standard Ruby location)

This one should only take a couple of minutes.

10. Compile Ruby:

time make -j2

In the above command, replace the -j2 with the number of cores your system has – in our case, we got 2:

The above took just under 4 minutes on our test server.

11. Now the tricky part – we’re going to install the version of Ruby we just compiled in a different directory, so that we can package it up with ease:

make install DESTDIR=/ruby-compiled-from-source

12. Test that the install worked:

/ruby-compiled-from-source/usr/bin/ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]

Important note: although Ruby seems to be working, if you try to do any actual work you might get a dependency warning. This is because we have compiled ruby by telling it to live under /usr but thrn we installed it under /ruby-compiled-from-source 🙂

13. Package it up with FPM:

cd /ruby-compiled-from-source/usr/
fpm --verbose -v 2.3.1 -n ruby-squirrel5 -d 'libyaml' -s dir -t rpm .=/usr

What the above command does is:

  • It will create an RPM type package from the files in the current directory (/ruby-compiled-from-source/usr)
  • It will set the package version to 2.3.1
  • It will set the RPM name to ruby-squirrel5 and append the version – so you’ll get ruby-squirrel5-2.3.1-1.x86_64.rpm
  • It will set the package ‘libyaml’ as an RPM dependency

When this is done you should see output like this:

Wrote: /tmp/package-rpm-build20160816-9601-1dz4ld2/RPMS/x86_64/ruby-squirrel5-2.3.1-1.x86_64.rpm {:level=>:info}
Executing(%clean): /bin/sh -e /tmp/rpm-tmp.Ybvumo {:level=>:info}
Created package {:path=>"ruby-squirrel5-2.3.1-1.x86_64.rpm"}

14. Now let’s uninstall the previous versions of Ruby and install the new one:

rvm remove ruby-2.3.0
rvm implode #this will remove RVM!

15. Now install the new one:

yum install /ruby-compiled-from-source/usr/ruby-squirrel5-2.3.1-1.x86_64.rpm

16. Test it:

[root@rubytest usr]# ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
[root@rubytest usr]# which ruby

So now you can create a yum repository and distribute the Ruby RPM to servers through yum or any other way you like!


Sending notifications to Jabber through Ruby – a simple example

Hello Squirrels,

We were recently investigating possible solutions to notify our team members when an alert from our monitoring system comes in.

Most of our team was already using Skype so that seemed like the most natural solution. Unfortunately it turns out, that the most “robust” way of sending messages programmatically to Skype in 2016 is via Sevabot but it requires quite an indirect approach to send those messages. We ended up not even testing this solution as having a dependency to software that requires a graphical environment running seems like overkill for us.

We then looked into using Jabber. A disadvantage with Jabber is of course that we’d need to setup our own Jabber server and maintain it, but this turned out to be trivial in comparison to making Skype work for us (We’ll discuss the Jabber setup in a future post).

The next problem was actually sending the messages. We sure found a lot of ready scripts out there claiming of being able to do the job, but once we tested them they simply would not work.

This prompted us to write our own script to get this done using Ruby. The script uses the xmpp4r gem to interface with Jabber.

Here’s what the script looks like:

#!/bin/env ruby
# Install the gem with:
# gem install xmpp4r --no-ri --no-rdoc

require 'timeout'
require 'xmpp4r'
include Jabber

Timeout::timeout(3) {
 password = "the_supa_password"
 message = (
 destination = ARGV[0]
 Jabber::debug = true

 client =

 msg = Message::new(destination, message)

The above will send a notification from “” to whoever you specify on the terminal. Example:

echo "hey did you see those squirrels across the street?!?" | /usr/local/bin/jabber_send.rb

The above example will send the message “hey did you see those squirrels across the street?!?” to the user