Category Archives: apache

Collectd, Graphite and Grafana on CentOS 6.x

Hi Squirrels,

Monitoring dashboards are the cool thing nowadays and their leader is without doubt Grafana and for good reason – it’s easy to install, user intuitive and it’s beautiful.

However, Grafana doesn’t work on it’s own – it needs a data source to read data from. It’s compatible with multiple but the most popular seems to be Graphite.

In this post we’re going to look into what it took to install Graphite and Grafana on CentOS 6.x and then feed data to Graphite from collectd. To make things easier on us we’re going to use our own created collectd RPM.

1. Install Python 2.7 for CentOS using the IUS (Inline with Upstream Stable) repository

yum install -y https://dl.iuscommunity.org/pub/ius/stable/CentOS/6/x86_64/ius-release-1.0-14.ius.centos6.noarch.rpm
yum install -y python27 python27-devel python27-pip

2. Install a bunch of other dependencies to make our life easier:

yum install -y git gcc libffi-devel autoconf bitmap-fonts cairo cairo-devel openldap-devel nc

3. Install Carbon – Carbon is the listener daemon of Graphite – it receives the metrics and stores them in a database

cd /tmp
wget https://github.com/graphite-project/carbon/archive/0.9.15.tar.gz
tar -xzf 0.9.15.tar.gz
rm -f 0.9.15.tar.gz 
cd carbon-0.9.15/
pip2.7 install -r requirements.txt
python2.7 setup.py install

4. Now get the Graphite source and install it’s dependencies:

cd /tmp
wget https://github.com/graphite-project/graphite-web/archive/0.9.15.tar.gz
tar -xzf 0.9.15.tar.gz
rm -f 0.9.15.tar.gz
cd graphite-web-0.9.15/
pip2.7 install -r requirements.txt
pip2.7 install python-ldap

5. Make sure that all dependencies are met:

Test with:

python2.7 ./check-dependencies.py 

Expected output:

All necessary dependencies are met.
All optional dependencies are met.

6. Install Graphite:

python2.7 setup.py install

7. Let’s begin configuring Graphite – first move the templates into their expected locations:

cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi

8. Edit /opt/graphite/webapp/graphite/local_settings.py and find the line:

#TIME_ZONE = 'America/Los_Angeles'

change it to:

TIME_ZONE = 'America/Los_Angeles'

Then also find the line:

#SECRET_KEY = 'UNSAFE_DEFAULT'

change it to:

SECRET_KEY = 'random_password'

Where random_password can be a password you have generated with something like openssl rand -hex 10.

9. Setup the database:

cd /opt/graphite/webapp/graphite
python2.7 manage.py syncdb

You’ll get some questions, the first one is:

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no):

Answer: yes
Next question:

Username (leave blank to use 'root'):

I took the default, root – just press enter.
Next Question:

E-mail address:

You need to give a valid e-mail address, for our example I used support@squirrel5.com.
One more question:

Password:

You’ll need to enter a password twice.

When done you should see output like this:

Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

10. Now we can start Carbon:

/opt/graphite/bin/carbon-cache.py start

You should get output like this:

Starting carbon-cache (instance a)

You can check that carbon is listening on TCP port 2003 with netstat -ntlp | grep 2003

netstat -ntlp | grep 2003
tcp        0      0 0.0.0.0:2003                0.0.0.0:*                   LISTEN      1769/python2.7 

11. Setup Apache with mod_wsgi for Python 2.7

Since Graphite is a Django based web application, Apache needs to be installed with mod_wsgi. Here comes the tricky part – you need to make sure that you install the mod_wsgi for Python 2.7 – the one in the CentOS repositories is for Python 2.6 and if you install that a lot of things are going to break.

To do it the right way, we need to install mod_wsgi through pip:

yum -y install httpd httpd-devel
pip2.7 install mod_wsgi
chown -R apache:apache /opt/graphite
cp /opt/graphite/examples/example-graphite-vhost.conf  /etc/httpd/conf.d/graphite.conf

Note: If you see this in the apache logs you got the wrong version of mod_wsgi and the wrong version of Python being loaded by Apache:

grep -i python /var/log/httpd/*
[Mon Sep 12 03:45:54 2016] [notice] Apache/2.2.15 (Unix) DAV/2 mod_wsgi/3.2 <span style="color: #ff0000;">Python/2.6.6</span> configured -- resuming normal operations

Now check the Apache module listing:

apachectl -M
httpd: Syntax error on line 221 of /etc/httpd/conf/httpd.conf: Syntax error on line 15 of /etc/httpd/conf.d/graphite.conf: <span style="color: #ff0000;">Cannot load /etc/httpd/modules/mod_wsgi.so into server: /etc/httpd/modules/mod_wsgi.so: cannot open shared object file: No such file or directory;

The above error means that Apache did not find the module mod_wsgi.so where it expected it to. That’s an easy fix:

cp /usr/lib64/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so /etc/httpd/modules
ln -s /etc/httpd/modules/mod_wsgi-py27.so /etc/httpd/modules/mod_wsgi.so
pip2.7 uninstall -y whitenoise

Now restart Apache:

/etc/init.d/httpd restart

and check your Apache logs again:

/var/log/httpd/error_log:[Mon Sep 12 03:59:04 2016] [notice] Apache/2.2.15 (Unix) DAV/2 mod_wsgi/4.5.6 <span style="color: #ff0000;">Python/2.7.12</span> configured -- resuming normal operations

See how now it says Python/2.7.12 ? That’s what we were aiming for.

12. Check the Graphite logs and make sure there are no errors

In my case I saw the following

tail -n 30 /opt/graphite/storage/log/webapp/error.log | grep DatabaseError
[Tue Sep 13 03:36:48 2016] [error] DatabaseError: attempt to write a readonly database
[Tue Sep 13 03:36:48 2016] [error] <span style="color: #ff0000;">DatabaseError: attempt to write a readonly database</span>

If you get the error: “DatabaseError: attempt to write a readonly database”

Then you need to edit /opt/graphite/webapp/graphite/settings.py.

Find this section:

DATABASES = {
  'default': {
    'NAME': '',

change it to:

DATABASES = {
  'default': {
    'NAME': '/opt/graphite/storage/graphite.db',

And restart Apache again:

/etc/init.d/httpd restart

13. Access Graphite at http://yourip

Graphite will now be listening on port 80 – please note that this example isn’t meant for production and Graphite is totally unsecured here – you won’t even be prompted for a password.

The graphite web interface looks something like this:

graphite01

We’re not really going to work with the Graphite interface at all – let’s move right ahead to the next steps to install collectd.

14. Install the Squirrel5 repository:

yum -y install http://rpm.squirrel5.com/squirrel5-repo-1.0-1.x86_64.rpm

15. Install collectd

yum -y install collectd collectd-rrdtool

16. Configure collectd

Edit the file /etc/collectd.conf and change the line:

#Hostname    "localhost"

to:

Hostname    "testing.squirrel5.com"

Then find the line:

#LoadPlugin write_graphite

change it to:

LoadPlugin write_graphite

Now tell collectd where to send it’s metrics:

find the section:

#<Plugin write_graphite>
#  <Node "example">
#    Host "localhost"
#    Port "2003"
#    Protocol "tcp"
#    LogSendErrors true
#    Prefix "collectd"
#    Postfix "collectd"
#    StoreRates true
#    AlwaysAppendDS false
#    EscapeCharacter "_"
#  </Node>
#</Plugin>

change it to:

<Plugin write_graphite>
  <Node "squirrel5">
    Host "localhost"
    Port "2003"
    Protocol "tcp"
    LogSendErrors true
    StoreRates false
    AlwaysAppendDS false
    EscapeCharacter "_"
  </Node>
</Plugin>

17. Start collectd

/etc/init.d/collectd start

18. Add the Grafana repository

cat &amp;gt; /etc/yum.repos.d/grafana.repo &amp;lt;&amp;lt; EOL
[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/6/\$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOL

19. Install and start Grafana

yum -y install grafana
chkconfig --add grafana-server
/etc/init.d/grafana-server start

20. Make sure that Grafana is running

Check that it’s listening on port 3000:

netstat -ntlp | grep 3000
tcp        0      0 :::3000                     :::*                        LISTEN      3599/grafana-server

21. Access Grafana

Go to http://yourip:3000 and login with username ‘admin‘, password ‘admin

22. Add a ‘Data Source’

Click on the grafana icon and then on ‘Data Sources.

grafana01

You should now be seeing the ‘Data Sources‘ menu – click on ‘Add data source

grafana02

You should now be seeing the ‘Edit data source’ Configuration.

Set it as follows:

Name: Squirrel-Graphite, tick on ‘Default‘, Type should be set to ‘Graphite

Then on the http settings, you need to direct it to http://localhost:80 and set it to ‘proxy‘ access.

Finally click on ‘Save & Test’

grafana03

23. Add a new dashboard:

Grafana is all about Dashboards. Click on the Grafana icon to add one and then on ‘Dashboards‘ and then on ‘New

grafana04

24. Once you have your new dashboard, we need to add at least one graph to it.

Click on the green icon on the left, then ‘Add Panel‘ and ‘Graph

grafana05

  • Click on “select metric“,┬áSelect “testing_squirrel5_com
  • Click on “select metric“, Select “memory
  • Click on “select metric“, Select “memory used
  • Finally click on the ‘X‘ to be done!

grafana06

25. Extra points – set the graphs to automatically refresh every 5 seconds and the graph duration to 1 hour.

grafana07

Follow steps 24-25 to add as many graphs as you like!

Sources: