Category Archives: Hiawatha

Creating an SSL Certificate Authority (CA), signing certificates and authenticating using a client certificate

Greetings Squirrel army,

We’ve been looking into writing a quick post on SSL Certificate Authorities for quite some time, so here’s our blog post on how to get this done. We’re also going to take a look at client authentication using a certificate.

Please note that this post is based on the incredible post from the DataCenter Lords called “Creating Your Own SSL Certificate Authority (and Dumping Self Signed Certs)” – make sure to visit that one for more details on how the authentication works.

1. Create a directory that we’ll use to store our Root certificates and enter the directory

mkdir -pv /squirrel5-CA-root
cd /squirrel5-CA-root

2. Create a key for the CA – (Protip: NEVER SHARE THIS!)

openssl genrsa -out rootCA-NEVER_SHARE.key 2048

3. Use the key to create your CA certificate – this one (rootCA.pem) can be shared to the workstations you want to trust your new CA

openssl req -x509 -new -nodes -key rootCA-NEVER_SHARE.key -sha256 -days 1024 -out rootCA.pem

In the prompt that you’ll get the only line that really matters is the Common Name (CN):

"Common Name (eg, your name or your server's hostname) []"

In this example we’ll answer everything with “Enter” except the Common Name which we will name “Squirrel5-CA

4. Create a private key for one of your servers:

openssl genrsa -out server1.key 2048

5. Create a Certificate Signing Request (CSR) for your first server:

openssl req -new -key server1.key -out server1.csr

Just like above, the only line that matters here is the “Common Name” – the common name must be a FQDN or an IP address, in this example we’ll use

6. And now for the most important bit – use the rootCA-NEVER_SHARE.key to SIGN the new certificate and also tell it what CA to use the “-CA rootCA.pem” bit:

openssl x509 -req -in server1.csr -CA rootCA.pem -CAkey rootCA-NEVER_SHARE.key -CAcreateserial -out server1.crt -days 500 -sha256

For server1 we now have the following files, let’s take a look:

server1.crt <--- server1 cert (never share) 
server1.csr <--- certificate signing request - public
server1.key <--- your private key (never share)

7. Make a PEM out of the server1 certs:

cat server1.key server1.crt > server1.pem

The server1.pem goes to the server (In Hiawatha this means TLScertFile = /root/server1.pem)

8. Install the Squirrel5 repository and install Hiawatha:

yum -y install
yum -y install hiawatha

9. Configure Hiawatha

This is the Hiawatha config (/etc/hiawatha/hiawatha.conf) we’re using in this example (we’ve stripped out anything not necessary on purpose) :

Binding {
        Port = 443
        # The next line uses the PEM file we created in step 7
        TLScertFile = /squirrel5-CA-root/server1.pem 

Hostname =
WebsiteRoot = /var/www/hiawatha
StartFile = index.html

VirtualHost {
        Hostname =
        WebsiteRoot = /var/www
        RequireTLS = yes
        AccessLogfile = /var/log/server1.log
        ErrorLogfile = /var/log/server1-error.log

So what is the above doing? The above tells Hiawatha that:

“The site (Hostname = needs to be accessed over TLS (RequireTLS = yes)”

Let’s also throw a sample HTML page so we can get some output:

echo "<html><title>the squirrels...are watching</title></html>" > /var/www/index.html

10. Restart Hiawatha to apply the changes:

/etc/init.d/hiawatha restart

11. Now test with curl:


I got this error:

curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here:

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

This is because our server doesn’t have our new CA key in it’s standard location (/etc/pki/ca-trust/source/anchors/) – let’s point curl manually to the CA cert according to the instructions from that error message:

curl -I --cacert /squirrel5-CA-root/rootCA.pem


curl --cacert /squirrel5-CA-root/rootCA.pem
<html><title>the squirrels...are watching</title></html>

Awesome that works!

12. OK, but how do I make this permanent so I don’t have to tell curl every time?

Copy your PEM to /etc/pki/ca-trust/source/anchors/

cp /squirrel5-CA-root/rootCA.pem /etc/pki/ca-trust/source/anchors/

And update the system’s CAs:

update-ca-trust extract

I got the following message:

update-ca-trust: Warning: The dynamic CA configuration feature is in the disabled state

Enable the dynamic CA configuration (note: this command has no output) :

update-ca-trust force-enable

Try accessing the site again, but this time don’t specify the CA:

<html><title>the squirrels...are watching</title></html>

13. Very cool, but what about authentication?

Good point! We can edit Hiawatha’s configuration so that it demands that you provide the client’s certificate – here’s how:

  • Edit the Hiawatha configuration (/etc/hiawatha/hiawatha.conf) and edit the Binding section:
Binding {
        Port = 443
        # The next line uses the PEM file we created in step 7
        TLScertFile = /squirrel5-CA-root/server1.pem
        # Require a client certificate that has been signed by the following CA
        RequiredCA  = /squirrel5-CA-root/rootCA.pem
  • Restart Hiawatha:
/etc/init.d/hiawatha restart
  • Try connecting again as before:
curl -I


curl -I
curl: (35) NSS: client certificate not found (nickname not specified)
  • Try again, but this time offer the client certificate:
curl --cert /squirrel5-CA-root/server1.pem
<html><title>the squirrels...are watching</title></html>

There we go – authenticated to the server by providing the client certificate.


Nagios with Hiawatha and PHP-FPM 5.6

Hola ardillas!

This week we’ll look at systems administration gone off the beaten path. We’re all Hiawatha fanatics here at Squirrel5, so when the time came to setup Nagios we were faced with a scary prospect:

“Nagios on CentOS has Apache (httpd) and php as a dependency.”

It’s sad but true – here’s how to confirm what the Nagios dependencies are on CentOS:

yum -y install epel-release
repoquery --requires nagios | egrep -i "httpd|php"

So yes, our life would be much easier if we went with Apache since Apache is mostly preconfigured for it via the CentOS packages, but where’s the fun in that? We want Nagios to work with Hiawatha!

Here’s how we did it:

  1. Make sure the EPEL repo is installed:
yum -y install epel-release

2. Install Nagios and plugins:

yum -y install nagios nagios-plugins nagios-plugins-all

3. Install the REMI repository so we can get PHP-FPM 5.6:

yum -y install

4. Install the Squirrel5 RPM repository to get Hiawatha:

yum -y install

5. Install Hiawatha 10:

yum -y install hiawatha

6. Add a Hiawatha user:

useradd hiawatha -s /sbin/nologin

Very important note: Nagios does not like changing the user it runs under, this is by default ‘nagios’ but we are going to change it anyway 🙂

7. Fix permissions so that Nagios runs under the ‘hiawatha‘ user:

mkdir -pv /var/run/nagios
sed -i -- 's/NagiosUser=nagios/NagiosUser=hiawatha/g' /etc/init.d/nagios
sed -i -- 's/NagiosGroup=nagios/NagiosGroup=hiawatha/g' /etc/init.d/nagios
sed -i -- 's#NagiosRunFile=/var/run/' /etc/init.d/nagios
sed -i -- 's/nagios_user=nagios/nagios_user=hiawatha/g' /etc/nagios/nagios.cfg
sed -i -- 's/nagios_group=nagios/nagios_group=hiawatha/g' /etc/nagios/nagios.cfg
sed -i -- 's#lock_file=/var/run/' /etc/nagios/nagios.cfg
chown -R hiawatha:hiawatha /usr/lib64/nagios 
chown -R hiawatha:hiawatha /var/lib/php
chown -R hiawatha:hiawatha /var/log/nagios &amp;amp;amp;nbsp;
chown -R hiawatha:hiawatha /var/spool/nagios
chown -R hiawatha:hiawatha /var/run/nagios
chown -R hiawatha:hiawatha /etc/nagios
chown -R hiawatha:hiawatha /usr/share/nagios

8. Create a directory for the Hiawatha VirtualHost configurations:

mkdir /etc/hiawatha/conf.d

9. Tell Hiawatha to run under the ‘hiawatha‘ user, recognize the ‘.cgi‘ extension and include files under the /etc/hiawatha/conf.d directory:

echo "ServerId = hiawatha:hiawatha" >> /etc/hiawatha/hiawatha.conf
echo "CGIextension = cgi" >> /etc/hiawatha/hiawatha.conf
echo "Include /etc/hiawatha/conf.d" >>  /etc/hiawatha/hiawatha.conf

10. Now let’s add a VirtualHost for Nagios – we’ll call the file /etc/hiawatha/conf.d/nagios.conf

VirtualHost {
Hostname = $your_ip
WebsiteRoot = /usr/share/nagios/html
StartFile = index.php

AccessLogfile = /var/log/hiawatha/nagios-access.log
ErrorLogfile = /var/log/hiawatha/nagios-error.log
ExecuteCGI = yes
TimeForCGI = 5
UseFastCGI = PHP5
PasswordFile = basic:/etc/hiawatha/users
Alias = /nagios:/usr/share/nagios/html
Alias = /nagios/cgi-bin:/usr/lib64/nagios/cgi-bin

RequireTLS = yes

11. Now let’s install PHP-FPM 5.6 from the Remi repo along with a bunch of PHP extensions:

yum -y install php56 php56-php-gd php56-php-pecl-dom-varimport php56-php-pecl-jsonc php56-php-xml php56-php-pecl-zip php56-php-pecl-crypto php56-php-mcrypt php56-php-intl php56-php-mysqlnd php56-php-fpm

12. Add a PHP-FPM configuration – this needs to go under /opt/remi/php56/root/etc/php-fpm.d/nagios.conf:

; Start a new pool named 'nagios'.
listen = /var/lib/hiawatha/php-fcgi-nagios.sock
user = hiawatha
group = hiawatha

pm = static
pm.max_children = 2
pm.start_servers = 2
pm.min_spare_servers = 3
pm.max_spare_servers = 5
slowlog = /var/log/php-fpm/nagios-slow.log
php_admin_value[error_log] = /var/log/php-fpm/nagios-error.log
php_admin_flag[log_errors] = on

; Set session path to a directory owned by process user
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php

13. Move the example PHP-FPM configuration out of the way:

mv /opt/remi/php56/root/etc/php-fpm.d/www.conf /opt/remi/php56/root/etc/php-fpm.d/www.conf-backup

14. Cool – now tell Hiawatha, how to talk to PHP-FPM – add the file /etc/hiawatha/conf.d/php.conf:

FastCGIserver {
FastCGIid = PHP5
ConnectTo = /var/lib/hiawatha/php-fcgi-nagios.sock
Extension = php

15. Add an SSL configuration for Hiawatha /etc/hiawatha/conf.d/ssl.conf:

Binding {
Port = 443
TLScertFile = /etc/ssl/serverkey.pem

16. Create an SSL self signed certificate:

cd /etc/ssl/
openssl req -subj '/CN=$replace_with_your_ip$/C=US' -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 -keyout serverkey.pem -out server.crt
cat server.crt >> serverkey.pem
rm -f server.crt
chmod 400 serverkey.pem

17. Let’s start verifying configurations to make sure everything’s good to go – start with Hiawatha – type:

service hiawatha check

Sample output:

service hiawatha check
Configuration check via Wigwam...
Using /etc/hiawatha
Reading hiawatha.conf
Reading /etc/hiawatha/conf.d/nagios.conf
Reading /etc/hiawatha/conf.d/php.conf
Reading /etc/hiawatha/conf.d/ssl.conf
No non-fatal errors found in the Hiawatha configuration.

Configuration check via Hiawatha...
Using /etc/hiawatha
Reading hiawatha.conf
Reading /etc/hiawatha/conf.d/nagios.conf
Reading /etc/hiawatha/conf.d/php.conf
Reading /etc/hiawatha/conf.d/ssl.conf
Reading mimetype.conf
Configuration OK.

18. Let’s check PHP-FPM next, type:

service php56-php-fpm configtest

Sample output:

service php56-php-fpm configtest
[30-Aug-2016 04:29:30] NOTICE: configuration file /opt/remi/php56/root/etc/php-fpm.conf test is successful

19. Now let’s test Nagios – type:

service nagios checkconfig

Sample output:

service nagios checkconfig
Running configuration check... OK.

20. Let’s add an authorized user for Hiawatha – this is going to be the username and password you’ll need to use to access the web-based interface:

htpasswd -cb /etc/hiawatha/users nagiosadmin obey_the_acorn

Where ‘nagiosadmin‘ is the username and ‘obey_the_acorn‘ is the password.

You should get output like this:

Adding password for user squirrel_supreme

NOTE: If you change the username from ‘nagiosadmin‘ to anything else, make sure to also make this change in the /etc/nagios/cgi.cfg and to restart nagios after making the change.

21. Now start the services:


service nagios start

Sample output:

service nagios start
Starting nagios: done.


service hiawatha start

Sample output:

service hiawatha start
Starting webserver: Hiawatha


service php56-php-fpm start

service php56-php-fpm start
Starting php-fpm: [ OK ]

Now you should be able to access the Nagios web-based interface at the IP of your host – for example: You’ll get a warning for using a self-signed certificate (this is of course normal) and then you’ll be prompted for your username and password.

This is what you should be looking at now:

Nagios home screen
Nagios home screen