Monday, 10 August 2015

Monitoring UrbanCode Deployments with Docker, Graphite, Grafana, collectd and Chef! (Part 1)

Monitoring an UrbanCode Deploy server (sometimes more in HA setups) and it's agents requires keeping track of resource utilization multiple environments, the UrbanCode deployment server(s) itself and the linkages between (ie. the network).

Typical resources include:

  • CPU
  • Memory
  • Java Heap
  • Threads
  • Disk
  • Network
  • Virtual environment (hypervisor)

In addition to resource utilization, log files should also be monitored for abnormal activity and traffic. There are commercial offerings which do these types of things but since UrbanCode Deploy itself is a deployment solution, it can be used to deliver monitoring to nodes. All that's is needed is monitoring agents and a collector and a means to configure and connect it all together.

In this post I'll demonstrate a quick bootstrap solution for system and JVM resource monitoring using UrbanCode Deploy. It will provide an "out of the box" monitoring dashboard solution, Grafana from data stored in Graphite (running in a Docker container) of metrics collected by collectd that installed on nodes using a Chef recipe that's deployed through UrbanCode Deploy. The end result looking something like this:

Fig. 1 Monitoring Topology
Fig. 1 Monitoring Topology

For the time being this solution is solely for a Linux environments (RHEL, Ubuntu and variants) but this solution can be adapted to other OS's as many of the components have counterparts for Windows, AIX and other OS's.

So how do we get there? Well, one approach is to set it up manually, quite an operation if you have 1000s of agents, so we'll need to do better.

First, the assets need to be installed.

Fig. 2 Installing the Solution
We will need:

  1. An UrbanCode server with a few agents. You'll also need to install the chef plugin from here: https://developer.ibm.com/urbancode/plugin/chef
  2. I also created a plugin with groovy that adds 2 additional steps for components. One step gets the latest version for a component, and the other step gets an ID for a version in the component. You can see the source code here, it's a good example of how to create a custom plugin. It's quite simple.
    Plugin:
    http://www.boriskuschel.com/downloads/ComponentPlus.zip
    Source
  3. Import a component from IBM BlueMix DevOps Service Git found here:
    https://hub.jazz.net/git/kuschel/monitorucd/contents/master/Collectd+Chef+Cookbook.json
    Import it from the Components tab:

    You should now see it listed:


    The component is preconfigured to connect to 
    IBM BlueMix DevOps Service Git and pull the recipe periodically and create a new version, you may change this behaviour in Basic Settings by unchecking the Import Versions Automatically setting.
    All you need to do now is supply a BlueMix username and password in the component properties page. You may need to enter a jazz.net username, if applicable (without a domain).

  4. Now you need to import a generic process (the top level Processes tab. Not the component!) that will be used to deploy the latest version of the component deployment package onto agent nodes. This process is kept in IBM BlueMix DevOps Services's Git and can be found here: https://hub.jazz.net/git/kuschel/monitorucd/contents/master/Install_collectd.json


    Or, you can quickly import this into UrbanCode Deploy by using curl:
    curl -k -X POST -F file=@Install_collectd.json https://<user>:<pass>@<ucd host>/rest/process/import
    
    NOTE: I noticed that after importing the generic template the versionName step in the Generic Import_collectd process design (design tab) had three bullets "•", this needs to be updated to ensure that the Secure Property Value field is blank. If it's not, the fetching of the latest version will fail when version is not specified.
    
  5. We need a metrics collector to store the metrics and a graphing engine to visualize them. We'll be using a Docker image of Graphite/Grafana I put together. You will need to ability to build run a docker container either using boot2docker or the native support available in Linux
    I have put the image up on the public docker registry as bkuschel/graphite-grafana but you can also build it from the Dockerfile in IBM BlueMix DebOps Services's Git at https://hub.jazz.net/git/kuschel/monitorucd/contents/master/Dockerfile
  6. To get the image run:

    docker pull bkuschel/graphite-grafana

    Now run the image and bind the ports 80 and 2003 from the docker container to the hosts ports.

    docker run -p 80:80 -p 2003:2003 -t bkuschel/graphite-grafana

    You can also mount file volumes to the container that contains the collector's database, if you wish that to be persisted. Each time you restart the container, it contains a fresh database. This has its advantages for testing. You can also specify other configurations beyond what are provided as defaults. Look at the Dockerfile for the volumes.

Once the solution is installed all that needs to be done is to execute the process on UrbanCode. Yes, it's that easy.

Go to the Process Tab in UrbanCode Deploy Server, Click on Run Next to the "Install_collectd" process.

A dialog will popup asking for a series of parameters. These will be explained in more depth in a later post regarding the Chef recipe I created. (You can find it here)
  • Component Name: Should be set to the name of the component we imported earlier
  • Version Name (Optional): You can specify the name of a specific version of the component to use, otherwise it will use the latest
  • Is this a collectd Server?: If you look at Fig. 1, you'll see that many collectd clients connect to a central collectd server. If this node is the central collectd, this should be checked. Generally, this should be the main agent in the UrbanCode Server, usually co-located with the server.
  • Collectd Install Directory: The default is good
  • Collectd Username: You can leave this as default. This username is the one used to encrypt traffice between collectd clients and servers.
  • Collectd Password: Set any password. This password is the one used to encrypt traffic between collectd clients and servers. It's a good idea  to encrypt this password with htpasswd utility before pasting it here. For example to set the admin password, the first parameter is the username, the second is the password. The output contains the username, a colon, then the encrypted password. Paste that value in this property:

    > htpasswd -bnm admin admin
    admin:$apr1$qSfx7.W2$xf/2k1mDHnksPXZlrU.b90
    
    
  • Collectd Server (client)/Graphite host (server): if "Is this a collectd Server?" is checked then this is the graphite server host, the host that is running the docker container. Otherwise this the collectd server host.
  • UCD Server: The installation directory of the UrbanCode server (ex. /opt/ibm_ucd/server) if this collectd is to be installed on a node with a server.
  • Java Monitoring Template: If UCD Server is set and is installed on tomcat, select tomcat.conf.erb, otherwise select java.conf.erb.
  • Resource: Select the agent that this process should be executed on. (the host.

Once you click Submit, this should happen:
Deploy Process
Fig 3. Deploy Process
At this point, all the collectd daemons should be started and collecting. Navigate to the docker host at http://<Docker host>/. You should see a tree with metrics, like this:

You can also navigate to Grafana at http://<Docker Host>/grafana. Note that the username and password for both Graphite and Grafana are admin/admin.

This is quite a mouthful for one blog post and there are so many aspects to cover such as:
  • The UrbanCode Deploy Process, how does it work?
  • The Chef Recipe.
  • Collectd Collection Options (and the nmon option!).
  • How to create useful graphs in Graphite and dashboards in Grafana.
I will cover these in subsequent postings. In the meantime, try to set it up and see how it goes. If you're lucky, you end up playing with some cool metrics and graphs in Graphite/Grafana.

67 comments:

  1. Replies
    1. Great Article
      Cloud Computing Projects


      Networking Projects

      Final Year Projects for CSE


      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. keep up the good work. this is an Assam post. this to helpful, i have reading here all post. i am impressed. thank you. this is our digital marketing training center. This is an online certificate course
    digital marketing training in bangalore | https://www.excelr.com/digital-marketing-training-in-bangalore

    ReplyDelete
  3. Thanks for the interesting blog that you have implemented here. Very helpful and innovative. Waiting for your next upcoming article.
    Java training in chennai
    Java training institute in chennai
    Java course in chennai
    Java training classes
    Java training
    Java programming classes
    core java coure
    <a

    ReplyDelete
  4. You are providing a post is very useful for develop my knowledge and I learn more info from your blog.
    Digital Marketing Course In Kolkata
    Web Design Course In Kolkata
    SEO Course In Kolkata

    ReplyDelete
  5. Very nice job... Thanks for sharing this amazing and educative blog post! ExcelR Pune Digital Marketing Course

    ReplyDelete
  6. شرکت تلکا هاست پیشرو در میزبانی انواع وب سایت و هم چنین دارای انواع هاست فوق ارزان با کنترل پنل سی پنل میباشد.حتما از سایت ما دیدن کنید و از قیمت های مناسب برای انواع سرویس های وب و ثبت انواع دامنه شگفت زده شوید.

    ReplyDelete
  7. Wow! This article is jam-packed full of useful information. The points made here are clear, concise, readable and poignant. I honestly like your writing style.
    Best Data Science training in Mumbai

    Data Science training in Mumbai

    ReplyDelete
  8. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.
    Data Science Institute in Bangalore

    ReplyDelete
  9. What a great article!. I am bookmarking it to read it over again after work. It seems like a very interesting topic to write about.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata
    SAP training institute Kolkata

    ReplyDelete
  10. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work




    Dot Net Training in Chennai | Dot Net Training in anna nagar | Dot Net Training in omr | Dot Net Training in porur | Dot Net Training in tambaram | Dot Net Training in velachery





    ReplyDelete
  11. It is perfect time to make some plans for the future and it is time to be happy. I've read this post and if I could I desire to suggest you some interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read more things about it!

    Data Science Course

    ReplyDelete
  12. I’m happy I located this blog! From time to time, students want to cognitive the keys of productive literary essays composing. Your first-class knowledge about this good post can become a proper basis for such people. nice one

    Data Science Training

    ReplyDelete
  13. The development of artificial intelligence (AI) has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning. Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. machine learning projects for final year In case you will succeed, you have to begin building machine learning projects in the near future.

    Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include projects into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Final Year Project Centers in Chennai even arrange a more significant compensation.


    Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account.

    ReplyDelete
  14. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.
    Data Science Institute in Bangalore

    ReplyDelete
  15. The development of artificial intelligence (AI) has propelled more programming architects, information scientists, and different experts to investigate the plausibility of a vocation in machine learning. Notwithstanding, a few newcomers will in general spotlight a lot on hypothesis and insufficient on commonsense application. machine learning projects for final year In case you will succeed, you have to begin building machine learning projects in the near future.

    Projects assist you with improving your applied ML skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include projects into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Final Year Project Centers in Chennai even arrange a more significant compensation.


    Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account.

    ReplyDelete
  16. I see some amazingly important and kept up to length of your strength searching for in your on the site.

    Data Science Course

    ReplyDelete
  17. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    Data Science Training Institute in Bangalore

    ReplyDelete
  18. I think about it is most required for making more on this get engaged.

    Data Science Training

    ReplyDelete
  19. Nice post found to be very impressive with a genuine content while going through this post. Thanks for sharing and keep posting such an informative content.

    360DigiTMG Digital Marketing Course

    ReplyDelete
  20. Wonderful blog found to be very impressive to come across such an awesome blog. I should really appreciate the blogger for the efforts they have put in to develop such an amazing content for all the curious readers who are very keen of being updated across every corner. Ultimately, this is an awesome experience for the readers. Anyways, thanks a lot and keep sharing the content in future too.

    360DigiTMG Tableau Course

    ReplyDelete
  21. Extraordinary blog went amazed with the content that they have developed in a very descriptive manner. This type of content surely ensures the participants to explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

    360DigiTMG Cloud Computing Course

    ReplyDelete
  22. Tremendous blog quite easy to grasp the subject since the content is very simple to understand. Obviously, this helps the participants to engage themselves in to the subject without much difficulty. Hope you further educate the readers in the same manner and keep sharing the content as always you do.

    360DigiTMG Data Science Course

    ReplyDelete
  23. I really enjoyed reading your blog filled with valuable information thank you.
    Data Analytics Course Online 360DigiTMG

    ReplyDelete
  24. Fantastic blog with very informative information, found valuable thanks for sharing
    typeerror nonetype object is not subscriptable

    ReplyDelete
  25. Comfortably, the post is really the freshest on that deserving topic. I harmonies with your conclusions and definitely will thirstily look forward to your next updates. cheap dedicated servers silicon valley

    ReplyDelete
  26. Top quality article with knowledgeable information, found very useful thanks for sharing waiting for next blog.
    Data Analytics Course Online 360DigiTMG

    ReplyDelete
  27. Fantastic blog extremely good well enjoyed with the incredible informative content which surely activates the learners to gain the enough knowledge. Which in turn makes the readers to explore themselves and involve deeply in to the subject. Wish you to dispatch the similar content successively in future as well.

    artificial intelligence course in bhilai

    ReplyDelete
  28. Really nice and interesting blog information shared was valuable and enjoyed reading this one. Keep posting. Thanks for sharing.
    Data Science Training in Hyderabad

    ReplyDelete
  29. Really wonderful blog completely enjoyed reading and learning to gain the vast knowledge. Eventually, this blog helps in developing certain skills which in turn helpful in implementing those skills. Thanking the blogger for delivering such a beautiful content and keep posting the contents in upcoming days.

    Data Science certification in Bhilai

    ReplyDelete
  30. Truly incredible blog found to be very impressive due to which the learners who ever go through it will try to explore themselves with the content to develop the skills to an extreme level. Eventually, thanking the blogger to come up with such an phenomenal content. Hope you aarrive with the similar content in future as well.

    Digital Marketing training in Bhilai

    ReplyDelete
  31. Tremendous blog quite easy to grasp the subject since the content is very simple to understand. Obviously, this helps the participants to engage themselves in to the subject without much difficulty. Hope you further educate the readers in the same manner and keep sharing the content as always you do.

    Data Science training in Raipur

    ReplyDelete
  32. Stupendous blog huge applause to the blogger and hoping you to come up with such an extraordinary content in future. Surely, this post will inspire many aspirants who are very keen in gaining the knowledge. Expecting many more contents with lot more curiosity further.

    Digital Marketing training in Raipur

    ReplyDelete
  33. Fantastic blog extremely good well enjoyed with the incredible informative content which surely activates the learners to gain the enough knowledge. Which in turn makes the readers to explore themselves and involve deeply in to the subject. Wish you to dispatch the similar content successively in future as well.

    ai course in raipur

    ReplyDelete
  34. Nice Information Your first-class knowledge of this great job can become a suitable foundation for these people. I did some research on the subject and found that almost everyone will agree with your blog.
    Cyber Security Course in Bangalore

    ReplyDelete
  35. Writing in style and getting good compliments on the article is hard enough, to be honest, but you did it so calmly and with such a great feeling and got the job done. This item is owned with style and I give it a nice compliment. Better!
    Cyber Security Training in Bangalore

    ReplyDelete
  36. It took a while to understand all the comments, but I really enjoyed the article. It turned out to be really helpful for me and I'm positive for all the reviewers here!. PMP Training in Hyderabad

    ReplyDelete