Category Archives: collectd

Collectd, Graphite and Grafana on CentOS 6.x

Hi Squirrels,

Monitoring dashboards are the cool thing nowadays and their leader is without doubt Grafana and for good reason – it’s easy to install, user intuitive and it’s beautiful.

However, Grafana doesn’t work on it’s own – it needs a data source to read data from. It’s compatible with multiple but the most popular seems to be Graphite.

In this post we’re going to look into what it took to install Graphite and Grafana on CentOS 6.x and then feed data to Graphite from collectd. To make things easier on us we’re going to use our own created collectd RPM.

1. Install Python 2.7 for CentOS using the IUS (Inline with Upstream Stable) repository

yum install -y
yum install -y python27 python27-devel python27-pip

2. Install a bunch of other dependencies to make our life easier:

yum install -y git gcc libffi-devel autoconf bitmap-fonts cairo cairo-devel openldap-devel nc

3. Install Carbon – Carbon is the listener daemon of Graphite – it receives the metrics and stores them in a database

cd /tmp
tar -xzf 0.9.15.tar.gz
rm -f 0.9.15.tar.gz 
cd carbon-0.9.15/
pip2.7 install -r requirements.txt
python2.7 install

4. Now get the Graphite source and install it’s dependencies:

cd /tmp
tar -xzf 0.9.15.tar.gz
rm -f 0.9.15.tar.gz
cd graphite-web-0.9.15/
pip2.7 install -r requirements.txt
pip2.7 install python-ldap

5. Make sure that all dependencies are met:

Test with:

python2.7 ./ 

Expected output:

All necessary dependencies are met.
All optional dependencies are met.

6. Install Graphite:

python2.7 install

7. Let’s begin configuring Graphite – first move the templates into their expected locations:

cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
cp /opt/graphite/webapp/graphite/ /opt/graphite/webapp/graphite/
cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi

8. Edit /opt/graphite/webapp/graphite/ and find the line:

#TIME_ZONE = 'America/Los_Angeles'

change it to:

TIME_ZONE = 'America/Los_Angeles'

Then also find the line:


change it to:

SECRET_KEY = 'random_password'

Where random_password can be a password you have generated with something like openssl rand -hex 10.

9. Setup the database:

cd /opt/graphite/webapp/graphite
python2.7 syncdb

You’ll get some questions, the first one is:

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no):

Answer: yes
Next question:

Username (leave blank to use 'root'):

I took the default, root – just press enter.
Next Question:

E-mail address:

You need to give a valid e-mail address, for our example I used
One more question:


You’ll need to enter a password twice.

When done you should see output like this:

Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

10. Now we can start Carbon:

/opt/graphite/bin/ start

You should get output like this:

Starting carbon-cache (instance a)

You can check that carbon is listening on TCP port 2003 with netstat -ntlp | grep 2003

netstat -ntlp | grep 2003
tcp        0      0      *                   LISTEN      1769/python2.7 

11. Setup Apache with mod_wsgi for Python 2.7

Since Graphite is a Django based web application, Apache needs to be installed with mod_wsgi. Here comes the tricky part – you need to make sure that you install the mod_wsgi for Python 2.7 – the one in the CentOS repositories is for Python 2.6 and if you install that a lot of things are going to break.

To do it the right way, we need to install mod_wsgi through pip:

yum -y install httpd httpd-devel
pip2.7 install mod_wsgi
chown -R apache:apache /opt/graphite
cp /opt/graphite/examples/example-graphite-vhost.conf  /etc/httpd/conf.d/graphite.conf

Note: If you see this in the apache logs you got the wrong version of mod_wsgi and the wrong version of Python being loaded by Apache:

grep -i python /var/log/httpd/*
[Mon Sep 12 03:45:54 2016] [notice] Apache/2.2.15 (Unix) DAV/2 mod_wsgi/3.2 <span style="color: #ff0000;">Python/2.6.6</span> configured -- resuming normal operations

Now check the Apache module listing:

apachectl -M
httpd: Syntax error on line 221 of /etc/httpd/conf/httpd.conf: Syntax error on line 15 of /etc/httpd/conf.d/graphite.conf: <span style="color: #ff0000;">Cannot load /etc/httpd/modules/ into server: /etc/httpd/modules/ cannot open shared object file: No such file or directory;

The above error means that Apache did not find the module where it expected it to. That’s an easy fix:

cp /usr/lib64/python2.7/site-packages/mod_wsgi/server/ /etc/httpd/modules
ln -s /etc/httpd/modules/ /etc/httpd/modules/
pip2.7 uninstall -y whitenoise

Now restart Apache:

/etc/init.d/httpd restart

and check your Apache logs again:

/var/log/httpd/error_log:[Mon Sep 12 03:59:04 2016] [notice] Apache/2.2.15 (Unix) DAV/2 mod_wsgi/4.5.6 <span style="color: #ff0000;">Python/2.7.12</span> configured -- resuming normal operations

See how now it says Python/2.7.12 ? That’s what we were aiming for.

12. Check the Graphite logs and make sure there are no errors

In my case I saw the following

tail -n 30 /opt/graphite/storage/log/webapp/error.log | grep DatabaseError
[Tue Sep 13 03:36:48 2016] [error] DatabaseError: attempt to write a readonly database
[Tue Sep 13 03:36:48 2016] [error] <span style="color: #ff0000;">DatabaseError: attempt to write a readonly database</span>

If you get the error: “DatabaseError: attempt to write a readonly database”

Then you need to edit /opt/graphite/webapp/graphite/

Find this section:

  'default': {
    'NAME': '',

change it to:

  'default': {
    'NAME': '/opt/graphite/storage/graphite.db',

And restart Apache again:

/etc/init.d/httpd restart

13. Access Graphite at http://yourip

Graphite will now be listening on port 80 – please note that this example isn’t meant for production and Graphite is totally unsecured here – you won’t even be prompted for a password.

The graphite web interface looks something like this:


We’re not really going to work with the Graphite interface at all – let’s move right ahead to the next steps to install collectd.

14. Install the Squirrel5 repository:

yum -y install

15. Install collectd

yum -y install collectd collectd-rrdtool

16. Configure collectd

Edit the file /etc/collectd.conf and change the line:

#Hostname    "localhost"


Hostname    ""

Then find the line:

#LoadPlugin write_graphite

change it to:

LoadPlugin write_graphite

Now tell collectd where to send it’s metrics:

find the section:

#<Plugin write_graphite>
#  <Node "example">
#    Host "localhost"
#    Port "2003"
#    Protocol "tcp"
#    LogSendErrors true
#    Prefix "collectd"
#    Postfix "collectd"
#    StoreRates true
#    AlwaysAppendDS false
#    EscapeCharacter "_"
#  </Node>

change it to:

<Plugin write_graphite>
  <Node "squirrel5">
    Host "localhost"
    Port "2003"
    Protocol "tcp"
    LogSendErrors true
    StoreRates false
    AlwaysAppendDS false
    EscapeCharacter "_"

17. Start collectd

/etc/init.d/collectd start

18. Add the Grafana repository

cat &amp;gt; /etc/yum.repos.d/grafana.repo &amp;lt;&amp;lt; EOL

19. Install and start Grafana

yum -y install grafana
chkconfig --add grafana-server
/etc/init.d/grafana-server start

20. Make sure that Grafana is running

Check that it’s listening on port 3000:

netstat -ntlp | grep 3000
tcp        0      0 :::3000                     :::*                        LISTEN      3599/grafana-server

21. Access Grafana

Go to http://yourip:3000 and login with username ‘admin‘, password ‘admin

22. Add a ‘Data Source’

Click on the grafana icon and then on ‘Data Sources.


You should now be seeing the ‘Data Sources‘ menu – click on ‘Add data source


You should now be seeing the ‘Edit data source’ Configuration.

Set it as follows:

Name: Squirrel-Graphite, tick on ‘Default‘, Type should be set to ‘Graphite

Then on the http settings, you need to direct it to http://localhost:80 and set it to ‘proxy‘ access.

Finally click on ‘Save & Test’


23. Add a new dashboard:

Grafana is all about Dashboards. Click on the Grafana icon to add one and then on ‘Dashboards‘ and then on ‘New


24. Once you have your new dashboard, we need to add at least one graph to it.

Click on the green icon on the left, then ‘Add Panel‘ and ‘Graph


  • Click on “select metric“,┬áSelect “testing_squirrel5_com
  • Click on “select metric“, Select “memory
  • Click on “select metric“, Select “memory used
  • Finally click on the ‘X‘ to be done!


25. Extra points – set the graphs to automatically refresh every 5 seconds and the graph duration to 1 hour.


Follow steps 24-25 to add as many graphs as you like!


Building an RPM package for collectd

Hello Squirrels,

We figured we would save folks some time and put together a quick blog post with the steps needed to create an RPM package for collectd for CentOS 6.x out of the latest sources.

This has been partly based on a few sources we found online so please look at the bottom of this post if you are interested in finding out more about those.

For those who are not familiar with it, collectd, is a daemon for collecting system statistics (cpu, memory etc). The highlights are that it’s built in plain C and as such it’s really fast and lightweight and it also comes with a ton of plugins. It can also be used as a building block for building Grafana dashboards – but more on this later ­čÖé

1. Get the latest source code – in this example we are going to build an RPM out of collectd 5.5.2.

cd /tmp
wget -c

2. Untar and enter the source directory

tar -xf collectd-5.5.2.tar.bz2
cd collectd-5.5.2

3. Install EPEL, GCC and the RPM build tools:

yum install -y rpm-build yum-utils rpmdevtools epel-release gcc

4. Create the /root/rpmbuild/SOURCES/ since rpmbuild expects it and copy the source code there

mkdir -pv /root/rpmbuild/SOURCES/
cp /tmp/collectd*.bz2 /root/rpmbuild/SOURCES/

5. Change the version number to 5.5.2 (the spec file in my case was for 5.5.1)

sed -i -- 's/5.5.1/5.5.2/g' /tmp/collectd-5.5.2/contrib/redhat/collectd.spec

6. Get any remaining dependencies with yum:

yum-builddep -y /tmp/collectd-5.5.2/contrib/redhat/collectd.spec

7. Build the RPM:

rpmbuild -bb /tmp/collectd-5.5.2/contrib/redhat/collectd.spec

8. The packages are now under /root/rpmbuild/RPMS/x86_64. To install locally do this:

cd /root/rpmbuild/RPMS/x86_64
yum localinstall collectd-* libcollectdclient-*

9. Set collectd to chkconfig to ‘on’ so that it can start on boot automatically:

chkconfig collectd on

10. Before starting the collectd service make sure that you change and uncomment the “#Hostname” line in the /etc/collectd.conf configuration file

11. Finally start collectd:

/etc/init.d/collectd start