As a system administrator, you know that one of the key components of the job is understanding, in minute detail, the performance characteristics and capacity of your systems. While there exists an enormous variety of tools available to measure the performance of various aspects of your systems, wouldn’t you rather have one tool that could already measure most or all of them? For many system administrators who have looked for a comprehensive performance measurement tool, the solution is collectd.
From the collectd web site:
collectd gathers statistics about the system it is running on and stores this information. Those statistics can then be used to find current performance bottlenecks (i.e. performance analysis) and predict future system load (i.e. capacity planning). Or if you just want pretty graphs of your private server and are fed up with some homegrown solution you’re at the right place, too ;).
One of the major advantages of collectd is that it comes with over 90 input plugins, and a number of output plugins as well. However, collectd does not have any means of directly displaying the data that it collects. This is often considered one of its strengths, as it does one thing (collect system performance data), and does it very well, leaving other tasks to other tools.
Rendering of the data must be performed via output plugins. By default, collectd writes data to RRD files, which can be consumed by other tools. Other output plugins include writers for Carbon server, network sockets, CSV files, etc.
In this article, we’ll take a look at using a new output plugin for collectd that sends performance metrics to Monitis, using Python and the Monitis REST API.
Using the monitis_writer plugin
Using the monitis_writer plugin requires a few configuration steps.
- Install the Monitis Python SDK
- Get the collectd_monitis source
- Create Monitis custom monitors
- Configure the monitis writer plugin in collectd
Install the Monitis Python SDK
The first step is to ensure that you have the Python SDK for Monitis installed. This is covered in detail in an earlier article, Monitis custom monitors for Amazon AWS CloudWatch metrics. The simplest option is to install the module using pip.
$ pip install Monitis-SDK
Get the collectd_monitis source
The source for collectd_monitis is not included in the module available using pip at the time of this writing. To retrieve the source, the easiest method is to clone the source repository from GitHub.
git clone git://github.com/monitisexchange/Python-SDK.git
This will create a copy of the source in your current directory. You will use the monitis_collectd directory within later when setting up the module path in collectd.conf.
Create Monitis custom monitors
The proceed with the example, you will need some custom monitors. We’ll create two custom monitors, test-host.cpu.cpu.wait and test-host.memory.memory.used. For CPU wait, we’ll be looking at the rate of change, and for used memory we will use the raw value.
In a python shell:
$ python >>> from monitis.api import Monitis >>> from monitis.monitors.custom import CustomMonitor >>> from monitis.monitors.params import ResultParams, DataType >>> rp1 = ResultParams('wait','CPU wait','jiffies/sec',DataType('float')) >>> cm1 = CustomMonitor.add_monitor(rp1,name='test-host.cpu.cpu.wait',tag='collectd') >>> rp2 = ResultParams('used','Memory Used','bytes',DataType('integer')) >>> cm2 = CustomMonitor.add_monitor(rp2,name='test-host.memory.memory.used',tag='collectd') >>> cm1 <CustomMonitor(id=4604, name=test-host.cpu.cpu.wait)> >>> cm2 <CustomMonitor(id=4605, name=test-host.memory.memory.used)>
Note the monitor IDs. These will be useful for identifying the monitors in the collectd configuration.
Configure the monitis writer plugin in collectd
This step consists of two parts: first, enable python plugins in collectd, and then configure the monitis_writer plugin.
Enable python plugins
The Monitis Writer plugin for collectd is based on the Python SDK for the Monitis REST API. To use it, the collectd Python plugin must first be enabled. Note that collectd 4.9 or later is required, as that was the first version to include Python plugin support.
In collectd.conf, add or uncomment the following:
<LoadPlugin python> Globals true </LoadPlugin>
Also, in collectd.conf, enable the monitis_writer plugin, and add metrics that should be sent to Monitis for monitoring and graphing. The following example will be explained in detail as we continue.
<Plugin python> ModulePath "/opt/vagrant_data/Python-SDK/monitis/tools/collectd_monitis" LogTraces true Interactive true Import "monitis_writer" <Module monitis_writer> Prefix "test-host" ApiKey "your Monitis API Key" SecretKey "Your Monitis Secret Key" Sandbox false Debug false <Monitor> Id 4606 Name "test-host.cpu.cpu.wait" ResultParam "wait" Delta true </Monitor> <Monitor> Id 4607 Name "test-host.memory.memory.used" ResultParam "used" </Monitor> </Module> </Plugin>
monitis_writer configuration explained
General instructions for using the collectd Python plugin can be found in the collectd-python man page, or at http://collectd.org/documentation/manpages/collectd-python.5.shtml. Note that the example includes the directory containing monitis_writer in the ModulePath, and Imports the monitis_writer module. Your installation path will probably differ, so adjust it accordingly.
Prefix will be prepended to the collectd metric name, allowing for host names or other identifiers to be included in the monitor name within Monitis. Other than that prefix, the Monitis custom monitor should match the name of the collectd metric. No provision is made in this version of the plugin to map between unmatched names.
ApiKey and SecretKey
These are your Monitis API key and secret key. They can be obtained in the Monitis web user interface under “Tools” -> “API” -> “API Key”. If you don’t already have an account, you can sign up for a free trial at http://www.monitis.com/free_signup.jsp.
If true, use the Monitis Developers’ Sandbox at sandbox.monitis.com. The default is false.
Debug can be used when Interactive is set to true and collectd is run in the foreground, to print the URLs and results of all of the Monitis API calls.
One or more Monitor blocks should exist, to identify the metrics we can to capture, and the custom monitor to send the data to. The block consists of several additional parameters:
ID – The monitor ID, which is returned when the new custom monitor is created via the API
Name – The name assigned to the monitor when it was created
ResultParam – Monitors can capture multiple results, each identified by different ResultParams. This parameter in the block should consist of the ResultParam name that was assigned when the monitor was created. See the example later of creating a monitor for more information.
Delta – For some metrics, such as those which are continuously increasing, it makes more sense to record the change in the value over time, rather than the raw value of the metric itself. When Delta is true, the monitis_writer plugin will report the change in the value as measured by collectd, from one measurement to the next. Since the amount of time between measurements can vary, it is reported in terms of change per second. For example, if the free space of a volume were measured as 100 MB in the first measurement, and then 80 MB 10 seconds later,then the reported change would be -2 MB/s. Delta defaults to false.
Now that it’s all configured, run collectd in foreground mode to watch the debugging output and verify that everything is working as expected.
$ sudo collectd -f
Assuming that works, you can now remove the Debug and Interactive flags (or set them to false) in the monitis_writer block of the collectd.conf file, and run it normally.
$ sudo /etc/init.d/collectd start
Viewing the results
That’s it. With 90+ collectd input plugins available, and a flexible framework for getting metric data into Monitis, almost anything you would care to measure on your systems can be graphed and monitored in Monitis.
To learn more, or to try it out for yourself, be sure to check out the following resources used for this article.