As we have mentioned on our blog few weeks ago. Apache is still the most popular webserver in the world. Monitis is already helpful in monitoring Apache performance (e.g. you can check how to get metrics provided by Apache status module or speed of served static content), but in this article we show how to use simple python script with M3 to monitor number of HTTP request and status codes of response provided by Apache. We will show how to use a new plugin to monitor the most popular types of HTTP requests (GET and POST) or groups of response status codes (2XX, 3XX, 4XX, 5XX).
Apache logs are source of big number of interesting information helping to monitor an application performance and state.To get them we will use modified version of apache-log-parser script avaliable on github (this is orginal version and here you can find one extended by Monitis).
Installation is easy. First you have to download/update Monitis Linux Scripts from GitHub. If you haven’t done it yet. You need to clone repo:
git clone git://github.com/monitisexchange/Monitis-Linux-Scripts.git
If you already downloaded repository you might want to update it:
git update (of course in directory you have cloned repo)
Next copy apache-log-parser.py to /usr/local/monitis. For example from main directory of Monitis-Linux-Scripts git repo run following command:
(ls -l /usr/local/monitis || mkdir /usr/local/monitis) && cp m3v3/plugins/
Now you can test the script:
Without any command line option you should see output similar to the following line (you might see numbers bigger than 0):
22/May/2012, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
If the above script failed it might be because of its limitation. The script works base on following two assumptions. It expects that Apache logs:
- are located in /var/log/apache2/access.log. Can be easily changed in lines 12 and 13.
- are in default format. This is a bit harder to change (you have to understand regex), but it can be done in line 87.
Apache log parser gets apache log file and counts number of selected parameters (e.g. request type) in given period of time (e.g. last hour). At the moment parser supports three: command line options:
- -r (time resolution) – predefine: seconds, minutes, hours or days (default days).
- -m (parameter to measure) -predefine: requests, responses (HTTP status codes), and whole groups of responses, (default responses).
- -t (time format) – predefine: UNIX timestamp and format based on default apache log time format. Controls the way time is presented in results.
For example to get information about request types in last hour with time presented in apache format:
/usr/local/monitis/apache-log-parser.py -r “hour” -m “requests” -t “apache”
Script has also short help (available with option -h or –help).
apache-log-parser.py is very useful when run as command line script, but we would like to use it to prepare some graphs. To connect to Monitis we are going to use version 3 of MonitisMonitorManager (M3V3). So if you haven’t used it yet, please follow installation notes to get it onto your system.
To use M3V3 we need xml config. Our configs are not different to typical M3 config.
a) For status codes, you need following line to define test command:
<exectemplate>/usr/local/monitis/apache-log-parser.py -t “unix” -r “min” -m “response_group”</exectemplate>
The output will be: number, number, number, number, number format (e.g. 1339760630,0,0,0,0) where successive numbers mean: UNIX timestamp, and next 2xx, 3xx, 4xx, 5xx status codes respectively. Therefore, regex for our metrics are defined in way similar to this example (for 3xx codes):
What means a number (any length) followed by comma, possible space, another number, another comma, space again and requested number.
b) To get numbers of request type we use nearly identical command:
<exectemplate>/usr/local/monitis/apache-log-parser.py -t “unix” -r “min” -m “requests”</exectemplate>
have similar output, but with 10 numbers (1339761646, 0, 0, 0, 0, 0, 0, 0, 0, 0) and similar regex (example for POSTs):
You might notice that we measure only 2 metrics and have 9 values in output, this is because script counts all HTTP requests even that less popular, which you rather not want to measure by default.
Finally please remember that to make it works you need only to provide yours Monitis API key and secret.
The script is easy to extend and more detailed description will be provided in another article.