Monitoring UrbanCode Deployments with Docker, Graphite, Grafana, collectd and Chef! (Part 3: The Chef Cookbook)

In Part 2 we examined the UCD processes that provisioned the collectd cookbook onto the UrbanCode Deploy Agent hosts. In this blog post, we'll take a closer look at this cookbook.

When we use Chef to install collectd, we are using Chef "Solo" or local Chef. Local Chef assumes everything (ie. the Cookbooks) that it needs is available locally so it won't need to contact a Chef server to pull down Cookbook dependencies. You will see that the git repository contains not only the collectd Chef Cookbook but also the dependencies. The collectd cookbook I am using is a slightly modified version of this one: https://github.com/hectcastro/chef-collectd

When you execute chef in local mode, you supply in a configuration file; a node file. This node file contains all the recipes we want to execute as well as all the properties needed by those recipes.

We have two configuration files, one for collectd as a server and the other for when collectd installed as a client. The configuration files are created dynamically by UrbanCode Deploy as part of the generic process and are modified on-the-fly with some UrbanCode Process Request properties.

Recall in the Generic Process in Part 2 that there were two steps we glossed over:
Let's look at the collectd server step first, it creates the Chef node configuration file with the following contents:

{
  "run_list": [ "recipe[collectd::default]","recipe[collectd::attribute_driven]" ],
  "collectd": {
    "dir": "${p:collectd_dir}",
    "graphite_ipaddress": "graphite",
    "plugins": {
      "aggregation" : {
         "template" : "aggregation.conf.erb"
      },
      "cpu" : {
      },
      "disk" : {
      },
      "df" : {
        "config" : {
         "FSType" : [ "proc", "sysfs", "fusectl", "debugfs", "devtmpfs", "devpts", "tmpfs", "cgroup" ],
         "IgnoreSelected" : true
        }
      },
      "entropy" : {
      },
      "interface" : {
        "config" : { "Interface" : "lo", "IgnoreSelected" : true }
      },
      "irq" : {
      },
      "java" : {
        "template" : "${p:java_monitoring_template}"
      },
      "load" : {
      },
      "memory" : {
      },
      "network" : {
        "template" : "network.conf.erb",
        "config" : {
          "host" : "0.0.0.0",
          "listen" : {
              "SecurityLevel" : "Encrypt",
              "AuthFile" : "${p:collectd_dir}/etc/auth_file"
          }N
        }
      },
      "processes" : {
        "config" : {
         "ProcessMatch" : [ "UrbanCode Deploy Server\", \".*java.*UDeployServer",
                            "UrbanCode Deploy Agent Monitor\" \".*java.*air-monitor.jar.*",
                            "UrbanCode Deploy Agent Worker\" \".*java.*com.urbancode.air.agent.AgentWorker"]
        }
      },
      "swap" : {
      },
      "syslog" : {
        "config" : {
          "LogLevel" : "info"
        }
      },
      "tcpconns" : {
        "config" : {
          "ListeningPorts" : false,
          "LocalPort" : [ 7918, 8443, 8080, 43, 80]
        }
      },
      "users" : {
      },
      "write_graphite" : {
        "config" : {
      "Host" : "${p:collectd_server}",
      "Protocol" : "tcp",
      "LogSendErrors" : true,
          "Prefix" : "collectd.",
      "StoreRates" : true,
          "AlwaysAppendDS" : false,
          "EscapeCharacter": "_"
        }
      }
    }
  }
} 

Most of this JSON content is actually the collectd configuration, in particular the contents of the "collectd" object. The "run_list" array, contains the recipes we want to execute. One of them is the default collectd installation, collectd::default, this pulls down the collect source code and all the packages needed to build and run collectd. The other recipe is called collectd::attribute_driven which reads the "collectd" object and constructs the series of collectd configuration files. Notice that we substitute in UrbanCode Deploy properties into the file, the ${p:} tokens.

Once the collectd::attribute_driven recipe is complete the configuration looks something like this:

${p:collectd_dir}/etc
├── auth_file (generated by a previous step)
├── collectd.conf
└── conf.d
    ├── aggregation.conf
    ├── cpu.conf
    ├── df.conf
    ├── disk.conf
    ├── entropy.conf
    ├── interface.conf
    ├── irq.conf
    ├── java.conf
    ├── load.conf
    ├── memory.conf
    ├── network.conf
    ├── processes.conf
    ├── swap.conf
    ├── syslog.conf
    ├── tcpconns.conf
    ├── users.conf
    └── write_graphite.conf

Each object within the "collectd" object in the JSON file corresponding to a configuration file which is included as part of the global collectd configuration using an include attribute in the collectd.conf file. This file that is read directly by the collectd process.

Include "${p:collectd_dir}/etc/conf.d/*.conf"

Each configuration file pertains to the collectd plugin configuration specified by the object name. So, for instance, the "cpu" JSON object generated the cpu.conf file which contains the options for the collectd cpu plugin.

Each collectd plugin has its own configuration specification. The simple ones that are only one level deep are handled generically by the recipe using JSON attributes in the "config" object that are translated from JSON to the collectd configuration file. So taking the syslog plugin example:

      "syslog" : {
        "config" : {
          "LogLevel" : "info"
        }
      }

This creates a syslog.conf file with the following contents:

LoadPlugin "syslog"
<Plugin "syslog">  
  LogLevel "info
</Plugin>

The default configuration template also supports the write_graphite plugin,which is more then one level deep, but for anything more complicated then one level deep one has to supply a configuration template,in the "template" attribute.

One interesting template is the one for Java JMX. If you remember in Part 1, one of the attributes we pass into the Generic Process, is the Java template. This value gets passed into the configuration via ${p:java_monitoring_template} property binding.
You can examine the two templates now, the one for Java is here:

https://hub.jazz.net/git/kuschel/monitorucd/contents/master/cookbooks/collectd/templates/default/java.conf.erb

and the one for Tomcat here:

https://hub.jazz.net/git/kuschel/monitorucd/contents/master/cookbooks/collectd/templates/default/tomcat.conf.erb

The difference is that in one case, we monitor Tomcat JMX mbeans and in the other case we do not. Notice that there are also bind variables passed into the template:

Host "<%= node["collectd"]["name"] %>"

This way we can actually pass in UrbanCode Deploy properties directly into the templates.

There are a few other examples, aggregation.conf.erb, network.conf.erb and we also have one for mysql.conf.erb. If we wanted to include mysql monitoring for the server using the mysql plugin we could customize this template and add:

      "mysql" : {
        "template" : "mysql.conf.emb"
      }

Be aware that mysql monitoring requires that mysql be set to binary logging. In your my.cnf you'll need to add:

log-bin=/var/lib/mysql/log-bin.log
binlog_format=row

Templates can be very powerful, look at the network.conf.erb template as an example. It is able to generate the network configuration for both the collectd server and client based on attributes passed alone.

Definitely check other the collectd plugins, there is one for Oracle, hypervisors and a load of others. New templates can be created as needed and added to the cookbook.

Let's take a look at the client Chef node configuration file generated by the client step:

{
  "run_list": [ "recipe[collectd::default]","recipe[collectd::attribute_driven]" ],
  "collectd": {
    "dir": "${p:collectd_dir}", 
    "plugins": {
      "aggregation" : {
    "template" : "aggregation.conf.erb"
      },
      "cpu" : {
      },
      "disk" : {
      },
      "df" : {
        "config" : {
         "FSType" : [ "proc", "sysfs", "fusectl", "debugfs", "devtmpfs", "devpts", "tmpfs", "cgroup" ],
         "IgnoreSelected" : true
        }
      },
      "entropy" : {
      },
      "interface" : {
        "config" : { "Interface" : "lo", "IgnoreSelected" : true }
      },
      "irq" : {
      },
      "java" : {
    "template" : "${p:java_monitoring_template}"
      },
      "load" : {
      },
      "memory" : {
      },
      "network" : {
    "template" : "network.conf.erb",
     "config" : {
          "host" : "${p:collectd_server}",
      "server" : {
          "SecurityLevel" : "Encrypt",
              "Username" : "${p:collectd_username}",
              "Password" :"${p:collectd_password}"
      }
        }
      },
      "ping" : {
        "config" : {
          "Host" : "${p:collectd_server}"
        }
      },
      "processes" : {
        "config" : {
         "ProcessMatch" : [ "UrbanCode Deploy Server\", \".*java.*UDeployServer",
                            "UrbanCode Deploy Agent Monitor\" \".*java.*air-monitor.jar.*",
                            "UrbanCode Deploy Agent Worker\" \".*java.*com.urbancode.air.agent.AgentWorker"]
        }
      },
      "swap" : {
      },
      "syslog" : {
        "config" : {
          "LogLevel" : "info"
        }
      },
      "users" : {
      }
    }
  }
}

It's very similar to the server except we have a different network configuration, ,the collectd client connects to the collectd server, and we include the ping plugin and remove the tcpconns plugin. As the client does not write to graphite directly the write_graphite plugin section is also removed.

That's about it. Some other small modifications were made to the cookbook to upgrade the collectd version and install some JVM specific libraries into the system path. Feel free to check out the cookbook, pull it and modify it for your topology.

Next is Part 4, the fun stuff, metrics and visualization.