Munin is an open source monitoring application. Munin has its cons and pros but the following blog post is not going to discuss it.
Munin by itself, like it or not, became a very popular monitoring platform among SysAdmins all around the world. I can easily understand why – despite its cons, it is a brilliant piece of software in my opinion.
I believe that the main features which made Munin so successful are:
1. Ease of operation – installing Munin is easy, even for the novice SysAdmin.
2. Adding custom plugins is just as easy. Choose your favorite language, handle some STDIN and STDOUT and you have a Munin plugin.
Another trait I’d like to mention about Munin is its storage backend – which is RRD. RRD stand for Round Robin Database. Essentially what RRD does is stores data in a circular buffer and also defines an interface for displaying this data in graphs.
Originally RRD was designed to store data for the infamous MRTG monitoring platform. RRD popularity has grown rapidly during the last few years and many other projects have started to use it just as well.
Simplistically, we could say that Munin has a server side and client side. The workflow of Munin is very simple:
- Server wakes up
- Server polls data from all configured clients and stores it in its RRD backend
- Server generates graphs for each host
This is all nice and simple, but one of the main drawbacks of Munin is that the server polls for data, instead of having the client connect to the server and send its data. This usually makes Munin a bit hard to operate behind strict firewalls.
Ideally, from my humble experience, what I would do is set a Munin server for each site and when a problem arises – just have a look at the relevant Munin installation.
So imagine yourself, the SysAdmin you are, you’re sitting in your spacious office, happy but slightly bothered. Happy because you have Munin, but a bit bothered because you have dozens of Munin installations around the world. Yes, they do show you graphs about anything you could have dreamed of, but it’s scattered. You have already invested countless hours in Munin writing custom-tailored plugins for software that runs on your servers, so you don’t want to switch to another monitoring solution. But you do appreciate the power and flexibility of a centralized monitoring platform.
What you would actually want your Munin to do better are probably these 2 things:
1. You would like a centralized interface for monitoring all of your servers
2. You would like a comprehensive system of alerting for all of these
Integrating Munin and Monitis eases the pain. Don’t worry! There’s not much for you to do, your loyal servant has done all of it for you, just read on.
And another “don’t worry!” for the fearful – you can still keep your current Munin installation – actually you have to!
I call the integration of Monitis and Munin – Moominitis or Muminitis. But it’s not the name, it’s what it does.
Continuing on I assume you have a working Munin installation as the procedures and commands following operate on functional Munin RRD data.
First of all, let’s checkout the latest Monitis-Linux-Scripts repository (I’ll work under /usr/share/monitisexchange):
# mkdir -p /usr/share/monitisexchange && cd /usr/share/monitisexchange
# git clone git://github.com/monitisexchange/Monitis-Linux-Scripts.git
# cd Monitis-Linux-Scripts/RRD
# vim monitis_config
While editing monitis_config you should add your proper API key and secret key. This can be easily obtained from the Monitis interface under ‘Tools -> API -> API key’.
After this is working, we can start playing with Monitis and Munin. E.g.:
# ./monitis_rrd.sh munin list_hosts /var/lib/munin
monitis_rrd.sh takes a couple of parameters:
- list_hosts – List the hosts that Munin has collected data for
- list_monitors – List all the monitors that are configured for a certain host
- list_counters_for_monitor – List all counters a monitor can produce
- add_monitor – Adds a monitor using the Monitis REST API
- update_data_for_monitor – Update the latest data for a monitor using the Monitis REST API
The last command showed us the hosts Munin has collected data for. Another important function is list_counters_for_monitor.
In case you’d like to list the counters a monitor produces, use:
# ./monitis_rrd.sh munin list_counters_for_monitor /var/lib/munin localhost memory
This will list all the counters the Munin monitor of ‘memory’ produces for the host ‘localhost’.
But lets go with a concrete example. Say we’d like to add to Monitis a Munin counter that shows the amount of threads running on the system.
I’m assuming here my hostname is called ‘localhost’, but please change it to suit the hostname you are willing to monitor:
# ./monitis_rrd.sh munin add_monitor /var/lib/munin localhost threads threads
If all went good this will add the counter to Monitis, under ‘Monitors -> Manage Monitors -> Custom Monitors’.
Now lets add some data. to test if it is working at all, run the following:
# ./monitis_rrd.sh munin update_data_for_monitor /var/lib/munin localhost threads threads
Via the Monitis interface you should witness under ‘RRD_localhost_munin_threads’ that a result returned, showing the number of threads as polled by Munin:
Hooking it all together
Obviously the trick is to get the data updated as soon as Munin polls for new values. As I previously mentioned, Munin usually wakes up every once in a while (usually every 5 minutes) and polls for new data.
In order to get this data to update also on Moninis, we’ll create the following hook script that would be executed every time Munin does:
exec >& /dev/null
cd /usr/share/monitisexchange/Monitis-Linux-Scripts/RRD && \
./monitis_rrd.sh munin update_data_for_monitor /var/lib/munin localhost threads threads
Note: the script template can be found here.
Save it as monitis_hook.sh and set it with execution permissions (chmod +x monitis_hook.sh).
In Fedora and Debian Munin is invoked in /etc/cron.d/munin, change the following line:
*/5 * * * * munin test -x /usr/bin/munin-cron && /usr/bin/munin-cron
*/5 * * * * munin test -x /usr/bin/munin-cron && /usr/bin/munin-cron && /usr/share/monitisexchange/Monitis-Linux-Scripts/RRD/monitis_hook.sh
That’s it. Every time now Munin will be invoked, it’ll also update in Monitis the counters you’ve set to update.
Look! It’s generating a graph!!!
Yes I know – this specific graph is not that interesting, but do trust me it is authentic Munin RRD data that was sampled from my humble computer.
Good luck with your Moominitis installation!