One of the issues I ran into right away while deploying the monitoring solution described in "Monitoring UrbanCode Deployments with Docker, Graphite, Grafana, collectd and Chef!" into an internal topology managed by UrbanCode Deploy was that each of the agent host machines had quirks and issues that required me to constantly tweak the monitoring install process. Fixing yum and apt-get repositories, removing conflicts, installing unexpectedly missing libraries, conflicting JDKs.
The reason for this is that each machine was installed by different people who installed the operating systems and the UrbanCode Deploy Agent in different ways with different options. It would of been great if all nodes were consistent, it would of made my life much easier.
It was at this point that my colleague Michael told me that I should create a Blueprint in UrbanCode Deploy for the topology I want to deploy the monitoring solution into for testing. Here's Michael doing a quick demo of UrbanCode Deploy Blueprint Designer (aka UrbanCode Deploy with Patterns):
Fantastic, I can create a blueprint of the desired topology, add a monitoring component to the nodes that I wish to have monitored and presto! Here is what the blueprint looks like in UrbanCode Deploy Blueprint Designer:
I created three nodes with three different operating systems just to show off that this solution works on different Operating Systems. (it also works on RHEL 7 but I thought adding another node would be overdoing it a little as well as cramming my already overcrowded RSA sketches).
This blueprint is actually a Heat Orchestration Template (HOT). You can see the source code here: https://hub.jazz.net/git/kuschel/monitorucd/contents/master/Monitoring/Monitoring.yaml
So if we modify the original Installation in Monitoring UrbanCode Deployments with Docker, Graphite, Grafana, collectd and Chef! Part 1, it would look something like this:
|(Click to Enlarge)|
You'll also see the IBM::UrbanCode::SoftwareDeploy::UCD, IBM::UrbanCode::SoftwareConfig::UCD and IBM::UrbanCode::ResourceTree resource types which allow the Heat engine to deploy create resources in UrbanCode Deploy and ensure that component processes are executed are installed into the virtual machines, once the UrbanCode Deploy agents are installed and started.
Ok, let's take a time out and talk a little about how this all works. First what's Heat?
Heat is an Orchestration engine that is able to call cloud provider API's, and other necessary APIs to actualize the resources that are specified in yaml into a cloud environment. Heat is part of the OpenStack project so it "natively" supports OpenStack Clouds but can also work with Amazon Web Services, IBM SoftLayer and any other cloud provider that is compliant with the OpenStack interfaces required to create virtual machines, the virtual networks, etc.
In addition, Heat can be extended with other resource types like those for UrbanCode Deploy Components that allows them to be deployed into Environments provisioned by OpenStack via Heat using the Heat Orchestration Template (HOT) specified during a provisioning.
The UrbanCode Deploy Blueprint Designer provides a kick ass visual editor and a simple way to drag drop UrbanCode Deploy Components into Heat Orchestration Templates (HOT). It also provides the ability to connect to a cloud provider (OpenStack, AWS and IBM SoftLayer are currently supported) and deploy the HOT.
You can monitor the deployment progress.
Oh, it also uses Git as a source for the HOTs (yaml) so that makes it super easy to version and share blueprints.
Ok, let's go over the steps on how to install it. I assume you have UrbanCode Deploy installed and configured with UrbanCode Deploy Blueprint Designer and connected to an OpenStack Cloud. You can set up a quick cloud using DevStack.
- You'll also need to install the chef plugin from here: https://developer.ibm.com/urbancode/plugin/chef.
- Import the appliactionfrom IBM BlueMix DevOps Service Git found here:
Import it from the Applications tab:
Use the default options in the import dialog. After, you should now see it listed in Applications as "Monitored". There will also be a new component in the Components tab called Monitoring
I have made the git repository public so the component is already configured to to to the IBM BlueMix DevOps Service Git and pull the recipe periodically and create a new version, you may change this behaviour in Basic Settings by unchecking the Import Versions Automatically setting.
- You'll have to fix up the imported process a little as I had to remove the encrypted fields to allow easier import.
Go to Components->Monitoring->Processes-Install and edit the Install Collectd Step
In the Collectd Password field put. You will see bullets, that's ok. Copy/Paste (and No Spaces!):
- We need a metrics collector to store the metrics and a graphing engine to visualize them. We'll be using a Docker image of Graphite/Grafana/Collectd I put together. You will need to ability to build run a docker container either using boot2docker or the native support available in Linux
I have put the image up on the public docker registry as bkuschel/graphite-grafana-collectd but you can also build it from the Dockerfile in IBM BlueMix DebOps Services's Git at https://hub.jazz.net/git/kuschel/monitorucd/contents/master/DockerWithCollectd/Dockerfile
- To get the image run:
docker pull bkuschel/graphite-grafana-collect
Now run the image and bind the ports 80, 2003 and udp port 2 from the docker container to the hosts ports.
docker run -p 80:80 -p 2003:2003 -p 25826:25826/udp -t bkuschel/graphite-grafana-collectd
You can also mount file volumes to the container that contains the collector's database, if you wish that to be persisted. Each time you restart the container, it contains a fresh database. This has its advantages for testing. You can also specify other configurations beyond what are provided as defaults. Look at the Dockerfile for the volumes.
- You'll need to connect the UrbanCode Blueprint Designer to Git by adding https://hub.jazz.net/git/kuschel/monitorucd to the repositories
- You should now see Monitoring in the list of Blue prints on the UrbanCode Deploy Blueprint Designer Home Page. Click on it to open the blueprint.
I am not going to cover the UrbanCode Component Processes as they are essentially the same the ones I described in Monitoring UrbanCode Deployments with Docker, Graphite, Grafana, collectd and Chef! (Part 2: The UCD Process) and Interlude #2: UCD Monitoring with Windows Performance Monitor and JMXTrans. The processes have been reworked to be executable using the application/component processes rather then solely from Resource Processes (generic). I also added some steps that do fix of typical problems in OpenStack images, such as fixing the repository and a workaround for a host name issue causing JMX not to bind properly.
The blueprint is also rudimentary and it may need to be tweaked to conform to the specific cloud set up in your environment. I created three virtual machines for Operating System images I happed to have available on my OpenStack, hooked them together on the private network and gave them external IPs so that I can access them. They all have the Monitoring Component added to them and should be deployed into the Monitored Application.
Once you've fixed everything up, make sure you select a cloud and then click Provision...:
|(Click to Enlarge)|
It will now ask for launch configuration parameters, again, many of these will be specific to you environment but you should be able to leave everything as is.
If you bound the Docker container to other different ports you'll have to change the port numbers for graphite (2003) and Docker (25826).
You will need to set the Admin Pass to something recognizable, it's the Windows Administrator Password. You may or may not need this depending on how your windows image is set up. I needed it.
The Monitoring / Server is the Public IP address of your Docker host running the bkuschel/graphite-grafana-collectd image. The Monitoring / Password is the one the is built into the Docker image. You will need to modify the Docker image to either not hard code this value or build a new image with a different password.
- The Monitoring.yaml (originating from Git) in UrbanCode Deploy Blueprint is passed to the heat engine on provisioning, with all parameters bound.
- The heat engine creates an UrbanCode Deploy Environment in the application specified in yaml (this can be changed)
- The UrbanCode Deploy Environment is mapped to the UrbanCode Deploy Component as specified in the yaml resource
- It also creates UrbanCode Deploy resources that will be used to represent the UrbanCode Deploy agents once they come online
- The agent resources are mapped to the environment.
- Heat interacts with the cloud provider (OpenStack in this case) to deploy the virtual machines specified in the yaml.
- The virtual machines are created and the agents installed as part of virtual machine intialization ("user data").
- Once the agents come online the component process is run
- The component process will be run for each resource mapped to the environment
- The component process runs the generic process Install_collectd_process (or Install_perfmon_process for Windows) on each agent.
- The agent installs collectd or GraphitePowershellFunctions via chef and performs other process steps as required to get the monitoring solution deployed.
The progress can be monitored in UrbanCode Deploy Blueprint Designer:
|(Click to Enlarge)|
Once the process is finished the new topology should look something like this:
|(Click to Enlarge)|
That should be it, give it a shot. Once you got it working the results are quite impressive. Here are some Grafana performance dashboards for CPU and Heap based on the environment I deployed using this method. The three Monitoring_Monitoring_<ip> correspond to <UrbanCode Deploy Application>_<UrbanCode Deploy Component>_<UrbanCode Agent Host Name>:
|(Click to Enlarge)|
Hers are some related topics that I am considering for future posts:
- More on HOT with UrbanCode Blueprint Designer
- More on OpenStack
- General Installation, other topics
- Ceilometer, Monasca and Kiloeyes
- Docker in OpenStack
- More Grafana Dashboards