Monday, 10 August 2015

Monitoring UrbanCode Deployments with Docker, Graphite, Grafana, collectd and Chef! (Part 1)

Monitoring an UrbanCode Deploy server (sometimes more in HA setups) and it's agents requires keeping track of resource utilization multiple environments, the UrbanCode deployment server(s) itself and the linkages between (ie. the network).

Typical resources include:

  • CPU
  • Memory
  • Java Heap
  • Threads
  • Disk
  • Network
  • Virtual environment (hypervisor)

In addition to resource utilization, log files should also be monitored for abnormal activity and traffic. There are commercial offerings which do these types of things but since UrbanCode Deploy itself is a deployment solution, it can be used to deliver monitoring to nodes. All that's is needed is monitoring agents and a collector and a means to configure and connect it all together.

In this post I'll demonstrate a quick bootstrap solution for system and JVM resource monitoring using UrbanCode Deploy. It will provide an "out of the box" monitoring dashboard solution, Grafana from data stored in Graphite (running in a Docker container) of metrics collected by collectd that installed on nodes using a Chef recipe that's deployed through UrbanCode Deploy. The end result looking something like this:

Fig. 1 Monitoring Topology
Fig. 1 Monitoring Topology

For the time being this solution is solely for a Linux environments (RHEL, Ubuntu and variants) but this solution can be adapted to other OS's as many of the components have counterparts for Windows, AIX and other OS's.

So how do we get there? Well, one approach is to set it up manually, quite an operation if you have 1000s of agents, so we'll need to do better.

First, the assets need to be installed.

Fig. 2 Installing the Solution
We will need:

  1. An UrbanCode server with a few agents. You'll also need to install the chef plugin from here: https://developer.ibm.com/urbancode/plugin/chef
  2. I also created a plugin with groovy that adds 2 additional steps for components. One step gets the latest version for a component, and the other step gets an ID for a version in the component. You can see the source code here, it's a good example of how to create a custom plugin. It's quite simple.
    Plugin:
    http://www.boriskuschel.com/downloads/ComponentPlus.zip
    Source
  3. Import a component from IBM BlueMix DevOps Service Git found here:
    https://hub.jazz.net/git/kuschel/monitorucd/contents/master/Collectd+Chef+Cookbook.json
    Import it from the Components tab:

    You should now see it listed:


    The component is preconfigured to connect to 
    IBM BlueMix DevOps Service Git and pull the recipe periodically and create a new version, you may change this behaviour in Basic Settings by unchecking the Import Versions Automatically setting.
    All you need to do now is supply a BlueMix username and password in the component properties page. You may need to enter a jazz.net username, if applicable (without a domain).

  4. Now you need to import a generic process (the top level Processes tab. Not the component!) that will be used to deploy the latest version of the component deployment package onto agent nodes. This process is kept in IBM BlueMix DevOps Services's Git and can be found here: https://hub.jazz.net/git/kuschel/monitorucd/contents/master/Install_collectd.json


    Or, you can quickly import this into UrbanCode Deploy by using curl:
    curl -k -X POST -F file=@Install_collectd.json https://<user>:<pass>@<ucd host>/rest/process/import
    
    NOTE: I noticed that after importing the generic template the versionName step in the Generic Import_collectd process design (design tab) had three bullets "•", this needs to be updated to ensure that the Secure Property Value field is blank. If it's not, the fetching of the latest version will fail when version is not specified.
    
  5. We need a metrics collector to store the metrics and a graphing engine to visualize them. We'll be using a Docker image of Graphite/Grafana I put together. You will need to ability to build run a docker container either using boot2docker or the native support available in Linux
    I have put the image up on the public docker registry as bkuschel/graphite-grafana but you can also build it from the Dockerfile in IBM BlueMix DebOps Services's Git at https://hub.jazz.net/git/kuschel/monitorucd/contents/master/Dockerfile
  6. To get the image run:

    docker pull bkuschel/graphite-grafana

    Now run the image and bind the ports 80 and 2003 from the docker container to the hosts ports.

    docker run -p 80:80 -p 2003:2003 -t bkuschel/graphite-grafana

    You can also mount file volumes to the container that contains the collector's database, if you wish that to be persisted. Each time you restart the container, it contains a fresh database. This has its advantages for testing. You can also specify other configurations beyond what are provided as defaults. Look at the Dockerfile for the volumes.

Once the solution is installed all that needs to be done is to execute the process on UrbanCode. Yes, it's that easy.

Go to the Process Tab in UrbanCode Deploy Server, Click on Run Next to the "Install_collectd" process.

A dialog will popup asking for a series of parameters. These will be explained in more depth in a later post regarding the Chef recipe I created. (You can find it here)
  • Component Name: Should be set to the name of the component we imported earlier
  • Version Name (Optional): You can specify the name of a specific version of the component to use, otherwise it will use the latest
  • Is this a collectd Server?: If you look at Fig. 1, you'll see that many collectd clients connect to a central collectd server. If this node is the central collectd, this should be checked. Generally, this should be the main agent in the UrbanCode Server, usually co-located with the server.
  • Collectd Install Directory: The default is good
  • Collectd Username: You can leave this as default. This username is the one used to encrypt traffice between collectd clients and servers.
  • Collectd Password: Set any password. This password is the one used to encrypt traffic between collectd clients and servers. It's a good idea  to encrypt this password with htpasswd utility before pasting it here. For example to set the admin password, the first parameter is the username, the second is the password. The output contains the username, a colon, then the encrypted password. Paste that value in this property:

    > htpasswd -bnm admin admin
    admin:$apr1$qSfx7.W2$xf/2k1mDHnksPXZlrU.b90
    
    
  • Collectd Server (client)/Graphite host (server): if "Is this a collectd Server?" is checked then this is the graphite server host, the host that is running the docker container. Otherwise this the collectd server host.
  • UCD Server: The installation directory of the UrbanCode server (ex. /opt/ibm_ucd/server) if this collectd is to be installed on a node with a server.
  • Java Monitoring Template: If UCD Server is set and is installed on tomcat, select tomcat.conf.erb, otherwise select java.conf.erb.
  • Resource: Select the agent that this process should be executed on. (the host.

Once you click Submit, this should happen:
Deploy Process
Fig 3. Deploy Process
At this point, all the collectd daemons should be started and collecting. Navigate to the docker host at http://<Docker host>/. You should see a tree with metrics, like this:

You can also navigate to Grafana at http://<Docker Host>/grafana. Note that the username and password for both Graphite and Grafana are admin/admin.

This is quite a mouthful for one blog post and there are so many aspects to cover such as:
  • The UrbanCode Deploy Process, how does it work?
  • The Chef Recipe.
  • Collectd Collection Options (and the nmon option!).
  • How to create useful graphs in Graphite and dashboards in Grafana.
I will cover these in subsequent postings. In the meantime, try to set it up and see how it goes. If you're lucky, you end up playing with some cool metrics and graphs in Graphite/Grafana.

106 comments:

  1. Replies
    1. Great Article
      Cloud Computing Projects


      Networking Projects

      Final Year Projects for CSE


      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. What a great article!. I am bookmarking it to read it over again after work. It seems like a very interesting topic to write about.
    SAP training in Kolkata
    SAP training Kolkata
    Best SAP training in Kolkata
    SAP course in Kolkata
    SAP training institute Kolkata

    ReplyDelete
  3. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work




    Dot Net Training in Chennai | Dot Net Training in anna nagar | Dot Net Training in omr | Dot Net Training in porur | Dot Net Training in tambaram | Dot Net Training in velachery





    ReplyDelete
  4. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.
    Data Science Institute in Bangalore

    ReplyDelete
  5. I see some amazingly important and kept up to length of your strength searching for in your on the site.

    Data Science Course

    ReplyDelete
  6. Nice post found to be very impressive with a genuine content while going through this post. Thanks for sharing and keep posting such an informative content.

    360DigiTMG Digital Marketing Course

    ReplyDelete
  7. Wonderful blog found to be very impressive to come across such an awesome blog. I should really appreciate the blogger for the efforts they have put in to develop such an amazing content for all the curious readers who are very keen of being updated across every corner. Ultimately, this is an awesome experience for the readers. Anyways, thanks a lot and keep sharing the content in future too.

    360DigiTMG Tableau Course

    ReplyDelete
  8. Extraordinary blog went amazed with the content that they have developed in a very descriptive manner. This type of content surely ensures the participants to explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

    360DigiTMG Cloud Computing Course

    ReplyDelete
  9. Tremendous blog quite easy to grasp the subject since the content is very simple to understand. Obviously, this helps the participants to engage themselves in to the subject without much difficulty. Hope you further educate the readers in the same manner and keep sharing the content as always you do.

    360DigiTMG Data Science Course

    ReplyDelete
  10. I really enjoyed reading your blog filled with valuable information thank you.
    Data Analytics Course Online 360DigiTMG

    ReplyDelete
  11. Fantastic blog with very informative information, found valuable thanks for sharing
    typeerror nonetype object is not subscriptable

    ReplyDelete
  12. Comfortably, the post is really the freshest on that deserving topic. I harmonies with your conclusions and definitely will thirstily look forward to your next updates. cheap dedicated servers silicon valley

    ReplyDelete
  13. Really wonderful blog completely enjoyed reading and learning to gain the vast knowledge. Eventually, this blog helps in developing certain skills which in turn helpful in implementing those skills. Thanking the blogger for delivering such a beautiful content and keep posting the contents in upcoming days.

    Data Science certification in Bhilai

    ReplyDelete
  14. Truly incredible blog found to be very impressive due to which the learners who ever go through it will try to explore themselves with the content to develop the skills to an extreme level. Eventually, thanking the blogger to come up with such an phenomenal content. Hope you aarrive with the similar content in future as well.

    Digital Marketing training in Bhilai

    ReplyDelete
  15. Stupendous blog huge applause to the blogger and hoping you to come up with such an extraordinary content in future. Surely, this post will inspire many aspirants who are very keen in gaining the knowledge. Expecting many more contents with lot more curiosity further.

    Digital Marketing training in Raipur

    ReplyDelete
  16. Nice Information Your first-class knowledge of this great job can become a suitable foundation for these people. I did some research on the subject and found that almost everyone will agree with your blog.
    Cyber Security Course in Bangalore

    ReplyDelete
  17. Writing in style and getting good compliments on the article is hard enough, to be honest, but you did it so calmly and with such a great feeling and got the job done. This item is owned with style and I give it a nice compliment. Better!
    Cyber Security Training in Bangalore

    ReplyDelete
  18. So, when you decide to implement Salesforce for your business operations, then it is very important to hire any expert Salesforce interview questions and answers

    ReplyDelete
  19. They are produced by high level developers who will stand out for the creation of their polo dress. You will find Ron Lauren polo shirts in an exclusive range which includes private lessons for men and women.

    Business Analytics Course in Bangalore

    ReplyDelete
  20. It's like you understand the topic well, but forgot to include your readers. Maybe you should think about it from several angles.

    Data Analytics Course in Bangalore

    ReplyDelete
  21. Such trends have engendered a growing requirement to provide professionals with apt handling on huge info analysis. data science course syllabus

    ReplyDelete
  22. Stupendous blog huge applause to the blogger and hoping you to come up with such an extraordinary content in future. Surely, this post will inspire many aspirants who are very keen in gaining the knowledge. Expecting many more contents with lot more curiosity further.

    Data Science certification in Raipur

    ReplyDelete
  23. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  24. I have to search sites with relevant information ,This is a
    wonderful blog,These type of blog keeps the users interest in
    the website, i am impressed. thank you.
    Data Science Training in Bangalore

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. I have to search sites with relevant information ,This is a
    wonderful blog,These type of blog keeps the users interest in
    the website, i am impressed. thank you.
    Data Science Course in Bangalore

    ReplyDelete
  27. Great post! I really appreciate you and I like to more unique content about this title and keep updating here...
    Informatica Training in Bangalore
    Informatica Training in Chennai
    Informatica MDM Training in Chennai
    Informatica Course in Chennai

    ReplyDelete
  28. Happy to visit your blog, I am by all accounts forward to more solid articles and I figure we as a whole wish to thank such huge numbers of good articles, blog to impart to us.
    data science in malaysia

    ReplyDelete
  29. Very awesome!!! When I searched for this I found this website at the top of all blogs in search engines.
    Best Digital Marketing Courses in Hyderabad

    ReplyDelete
  30. I think I actually have never seen such blogs ever before that has complete things with all details that i would like. therefore kindly update this ever for us.
    data scientist certification

    ReplyDelete

  31. Very awesome!!! When I searched for this I found this website at the top of all blogs in search engines.


    business analytics course


    ReplyDelete
  32. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    data science training in bangalore

    ReplyDelete
  33. Kardinal Stick Siam - relx a great promotion. Express delivery in 3 hours.

    ufa football betting, casino, slots, lottery, direct website 1688, stable financial, 100% UFABET168.

    Online Baccarat FOXZ24 Easy to apply, fast, บาคาร่า deposit-withdraw 10 seconds with the system.

    Watch movies online sa-movie.com, watch new movies, series Netflix HD 4K ดูหนังออนไลน์, watch free movies on your mobile phone, Tablet, watch movies on the web.

    SEE4K Watch movies, watch movies, free series, load without interruption, sharp images in HD FullHD 4k, ดูหนังใหม่ all matters, all tastes, see anywhere, anytime, on mobile phones, tablets, computers.

    GangManga read manga, read manga, read manga online for free, fast loading, clear images in HD quality, อ่านการ์ตูน all titles, anywhere, anytime, on mobile, tablet, computer.

    Watch live football ผลบอลสด, watch football online, link to watch live football, watch football for free.

    ReplyDelete
  34. "Very Nice Blog!!!


    Please have a look about "
    data scientist course

    ReplyDelete
  35. This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.
    data scientist course in hyderabad

    ReplyDelete
  36. Really wonderful blog completely enjoyed reading and learning to gain the vast knowledge. Eventually, this blog helps in developing certain skills which in turn helpful in implementing those skills. Thanking the blogger for delivering such a beautiful content and keep posting the contents in upcoming days.

    data science course in bangalore with placement

    ReplyDelete
  37. This post is very simple to read and appreciate without leaving any details out. Great work!
    data science training

    ReplyDelete
  38. อีกทั้งเรายังให้บริการ เกมสล็อต ยิงปลา แทงบอลออนไลน์ รองรับทุกการใช้งานในอุปกรณ์ต่าง ๆ HTML5 คอมพิวเตอร์ แท็บเล็ต สมาทโฟน คาสิโนออนไลน์ และมือถือทุกรุ่น เล่นได้ตลอด 24ชม. ไม่ต้อง Downloads เกมส์ให้ยุ่งยาก ด้วยระบบที่เสถียรที่สุดในประเทศไทย

    ReplyDelete
  39. หาคุณกำลังหาเกมส์ออนไลน์ที่สามารถสร้างรายได้ให้กับคุณ เรามีเกมส์แนะนำ เกมยิงปลา รูปแบบใหม่เล่นง่ายบนมือถือ คาสิโนออนไลน์ บนคอม เล่นได้ทุกอุปกรณ์รองรับทุกเครื่องมือ มีให้เลือกเล่นหลายเกมส์ เล่นได้ทั่วโลกเพราะนี้คือเกมส์ออนไลน์แบบใหม่ เกมยิงปลา

    ReplyDelete
  40. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    machine learning training in aurangabad

    ReplyDelete
  41. Baccarat is actually money making and it's remarkable accessibility. Optimal In your case it's readily available that you will find pretty fascinating choices. And that is thought to be one thing that is really different And it's very something that is really prepared to hit with Pretty much the most wonderful, as well, is actually a really positive option. Furthermore, it's a really fascinating solution. It is a better way which can make money. Superbly prepar The number of best-earning baccarat will be the accessibility of generting the most money. Pretty much as practical is very well suited for you A substitute that could be assured. To a wide range of accessibility and performance And find out outstanding results also.บาคาร่า
    ufa
    ufabet
    แทงบอล
    แทงบอล
    แทงบอล

    ReplyDelete
  42. I must admit that your post is really interesting. I have spent a lot of my spare time reading your content. Thank you a lot!
    data scientist training and placement in hyderabad

    ReplyDelete
  43. I have voiced some of the posts on your website now, and I really like your blogging style. I added it to my list of favorite blogging sites and will be back soon ...

    Digital Marketing Training in Bangalore

    ReplyDelete
  44. I wanted to leave a little comment to support you and wish you the best of luck. We wish you the best of luck in all of your blogging endeavors.

    Artificial Intelligence Training in Bangalore

    ReplyDelete
  45. Incredibly conventional blog and articles. I am realy very happy to visit your blog. Directly I am found which I truly need. Thankful to you and keeping it together for your new post.
    data scientist online course

    ReplyDelete
  46. Thanks for another informative site. Where else could I get that kind of info written in such a perfect way? I have a project that I am just now working on, and I have been on the look out for such info. ยูฟ่าสล็อต

    ReplyDelete
  47. This is also a very good post which I really enjoyed reading. It is not every day that I have the possibility to see something like this..
    data science courses in aurangabad

    ReplyDelete
  48. I think this is an informative post and it is very useful and knowledgeable. Therefore, I would like to thank you for the efforts you have made in writing this article.
    data scientist training in hyderabad

    ReplyDelete
  49. Great Article. Thank you for sharing! Really an awesome post for everyone.
    DevOps Training in Hyderabad
    DevOps Course in Hyderabad

    ReplyDelete
  50. I loved this blog content. I also tested this article information helped to us Keep on modernizing these kinds of informational articles. Thank you for this blog. This for very interesting and useful.
    Python Training in Hyderabad
    Python Course in Hyderabad

    ReplyDelete
  51. Thanks for posting the best information and the blog is very important.digital marketing institute in hyderabad

    ReplyDelete
  52. Thanks for posting the best information and the blog is very important.artificial intelligence course in hyderabad

    ReplyDelete
  53. I was actually browsing the internet for certain information, accidentally came across your blog found it to be very impressive. I am elated to go with the information you have provided on this blog, eventually, it helps the readers whoever goes through this blog. Hoping you continue the spirit to inspire the readers and amaze them with your fabulous content.

    Data Science Course in Faridabad

    ReplyDelete
  54. Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing. data analytics course in delhi

    ReplyDelete
  55. Great to become visiting your weblog once more, it has been a very long time for me. Pleasantly this article i've been sat tight fosuch a long time. I will require this post to add up to my task in the school, and it has identical subject along with your review. Much appreciated, great offer. data science course in nagpur

    ReplyDelete
  56. Ucuz, kaliteli ve organik sosyal medya hizmetleri satın almak için Ravje Medyayı tercih edebilir ve sosyal medya hesaplarını hızla büyütebilirsin. Ravje Medya ile sosyal medya hesaplarını organik ve gerçek kişiler ile geliştirebilir, kişisel ya da ticari hesapların için Ravje Medyayı tercih edebilirsin. Ravje Medya internet sitesine giriş yapmak için hemen tıkla: https://www.ravje.com

    İnstagram takipçi satın almak için Ravje Medya hizmetlerini tercih edebilir, güvenilir ve gerçek takipçilere Ravje Medya ile ulaşabilirsin. İnstagram takipçi satın almak artık Ravje Medya ile oldukça güvenilir. Hemen instagram takipçi satın almak için Ravje Medyanın ilgili sayfasını ziyaret et: instagram takipçi satın al

    Tiktok takipçi satın al istiyorsan tercihini Ravje Medya yap! Ravje Medya uzman kadrosu ve profesyonel ekibi ile sizlere Tiktok takipçi satın alma hizmetide sunmaktadır. Tiktok takipçi satın almak için hemen tıkla: tiktok takipçi satın al

    ReplyDelete
  57. Great tips and very easy to understand. This will definitely be very useful for me when I get a chance to start my blog.
    data science course in malaysia

    ReplyDelete
  58. It is our mission to guarantee and ensure that all goods under our care are never tampered with. top security companies in London
    We offer specialized asset and high-value goods protection services incomparable to that of any other security company in London.

    ReplyDelete
  59. This was an extremely wonderful post. Thanks for providing this info. Kanye West Donda Vest

    ReplyDelete
  60. Thanks for posting the best information and the blog is very good.data science course in Lucknow

    ReplyDelete
  61. I really enjoyed reading your articles. It looks like you’ve spent a lot of time and effort on your blog. Dreamer Biker Jacket

    ReplyDelete
  62. Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming.
    cyber security course in malaysia

    ReplyDelete
  63. Our the purpose is to share the reviews about the latest Jackets,Coats and Vests also share the related Movies,Gaming, Casual,Faux Leather and Leather materials available Golf Wang Varsity Jacket

    ReplyDelete
  64. I love to recommend you Where can crawl Exciting Products latest Jackets, Coats and Vests Click Here Peter Parker Jacket

    ReplyDelete
  65. Extraordinary blog went amazed with the content that they have developed in a very descriptive manner. This type of content surely ensures the participants to explore more themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

    Data Science Training

    ReplyDelete
  66. I read blogs on a similar topic, but I never visited your blog. I added it to favorites and I’ll be your constant reader. Biker Boyz Jacket

    ReplyDelete
  67. This was an great post . Thanks for sharing your blog post with us....we offer projects in chennai in different domains such like final year projects 2021

    ReplyDelete
  68. As the bright sunshine, hope your day be blessed with positivity and cheerful thoughts. Wishing you a warm Thursday good morning. Thursday Greetings Quotes

    ReplyDelete
  69. Very informative article! Thank you for sharing! I like it so much. If ever you need tree service at Willow Grove Pennsylvania don't hesitate to check it

    ReplyDelete
  70. Thank you for taking the time and sharing this information with us. Ziyyara offers customized CBSE Board Home Tuition online, with experienced and certified tutors, providing personalized attention to each student.
    For more info contact +91-9654271931 or visit Online Tuition for CBSE Board

    ReplyDelete