Amazon is a leading Infrastructure as a Service (IaaS) provider with the Amazon Web Services (AWS) platform. This type of platform — in which servers, network services, and more are provided on-demand — is also known as cloud computing. AWS is used by companies both large and small to build scalable infrastructure as needed, without significant up-front costs. There are many types of services available within AWS, including servers, database, load balancing, storage, etc.
System health monitoring is a critical component of any infrastructure, and Amazon provides monitoring through the CloudWatch service. From http://aws.amazon.com/cloudwatch/:
Amazon CloudWatch provides monitoring for AWS cloud resources and the applications customers run on AWS. Developers and system administrators can use it to collect and track metrics, gain insight, and react immediately to keep their applications and businesses running smoothly. Amazon CloudWatch monitors AWS resources such as Amazon EC2 and Amazon RDS DB instances, and can also monitor custom metrics generated by a customer’s applications and services. With Amazon CloudWatch, you gain system-wide visibility into resource utilization, application performance, and operational health.
However, CloudWatch can only monitor services that are hosted within the Amazon cloud. This may be fine for businesses that host the entirety of their infrastructure with Amazon, but it is quite common for AWS hosting to comprise only a portion of a company’s infrastructure, with other services provided by dedicated application providers, hosted in local data centers, or even running in other clouds. For those businesses, CloudWatch is too limited, and another monitoring solution is required. This is a problem for which Monitis provides a simple and robust solution. With its open API, Monitis can collect metrics from a wide variety of sources and present them in a unified and consistent manner, offering many options for alerting and reporting.
In this article, we take a look at how CloudWatch monitoring can be integrated with a Monitis solution, and present a tool to make this integration easy.
The integration between CloudWatch and Monitis is provided by an application named Monitis-Cloudwatch. It is a python application based on the boto framework for interfacing with the AWS API, and the Monitis REST API. Monitis-Cloudwatch currently supports four AWS services, including Elastic Compute Cloud (EC2), Relational Database Service (RDS), Elastic Load Balancer (ELB), and EC2 Block Storage (EBS). Additional AWS services may be supported in the future.
The first step in using Monitis-Cloudwatch is to download the application. It is available in the Monitis Exchange GitHub repository, and can also be installed using pip.
These install instructions assume a working Python environment with pip, a Python package manager. Instructions for getting Python and pip set up are available at http://python.org/getit/ and http://www.pip-installer.org/en/latest/installing.html, respectively.
pip install Monitis-SDK
This will install the SDK plus any missing dependencies, as well as installing the monitis_cloudwatch.py application used in the later examples.
In order to use Monitis-CloudWatch, you will need API keys both for Amazon AWS and for Monitis.
If you don’t already have a Monitis account, you can sign up for a free trial at https://www.monitis.com/free_signup.jsp. Once you’re signed up, you will find your API Key and Secret Key in the Monitis web application under Tools -> API -> API Key. These should be added as environment variables. For Linux, OS X, and other Unix-like systems running the bash shell, you can put the following in the .bashrc file in your home directory.
export MONITIS_APIKEY=REPLACE_THIS_WITH_YOUR_API_KEY export MONITIS_SECRETKEY=REPLACE_THIS_WITH_YOUR_SECRET_KEY
The boto library used by Monitis-CloudWatch to access the AWS API also uses environment variables. You can find these in your AWS Account page at http://aws.amazon.com/ under Security Credentials within the Access Keys tab. You will need both your Access Key ID and Secret Access Key. To make these available to boto, add them as environment variables, as with the Monitis keys.
export AWS_ACCESS_KEY_ID=REPLACE_THIS_WITH_YOUR_ACCESS_KEY_ID export AWS_SECRET_ACCESS_KEY=REPLACE_THIS_WITH_YOUR_SECRET_ACCESS_KEY
Once it’s Installed
Once you have Monitis-CloudWatch installed with valid keys, the next step is to get a catalog of your current metrics. Any metric being collected by CloudWatch can also be managed by Monitis-CloudWatch; you will want to select one or more metrics to use.
To view this catalog, Monitis-CloudWatch takes a single argument, –catalog.
$ monitis_cloudwatch.py --catalog AWS/ELB lb-http-alpha HTTPCode_ELB_4XX AWS/ELB lb-http-alpha HealthyHostCount AWS/ELB lb-http-alpha UnHealthyHostCount AWS/EC2 i-fe68629c CPUUtilization
This listing is for illustrative purposes, and for brevity many results have been omitted.
For this example, we’ll select an EC2 instance, i-fe68629c, and the CPUUtilization metric. Once we’ve selected a metric from the catalog, it needs to have an associated Monitis monitor created. This is done once per metric.
$ monitis_cloudwatch.py --create --ec2 --instance i-bb1c7dde --metric CPUUtilization
After creating the monitor, list the existing monitors to verify that yours was created.
$ monitis_cloudwatch.py --list 3379 AWS/EC2 i-bb1c7dde.CPUUtilization
Next, update the monitor with metric data.
$ monitis_cloudwatch.py --ec2 --instance i-bb1c7dde --metric CPUUtilization \ --update --from -4days --period 900
This command uses a couple of arguments that may need explanation. First “–from -4days” indicates that AWS CloudWatch should be queried for metric data starting four days ago. There is a corresponding argument, –until, for indicating the end of the time to be queried. The default for –until is the current time. Note that if –from is not specified, it defaults to one hour prior to the end time. The –period argument specifies the granularity of metric data queried from CloudWatch. Typically, CloudWatch captures and stores metric data in one or five minute intervals. By specifying a longer period, the data returned from the query will be aggregated into larger chunks. The period is specified in seconds, so in the example above 900 means 15 minutes.
There are a few other options to monitis_cloudwatch.py to be aware of.
Existing monitors can be deleted using –delete. The monitor to delete must be identified by either –id or –name.
$ monitis_cloudwatch.py --list 3379 AWS/EC2 i-bb1c7dde.CPUUtilization $ monitis_cloudwatch.py --list --delete --name i-bb1c7dde.CPUUtilization $ monitis_cloudwatch.py --list $
Monitis provides a development sandbox for developers working on custom monitors. When working with the sandbox, include the –sandbox argument, and supply the following environment variables.
export MONITIS_SANDBOX_APIKEY=REPLACE_THIS_WITH_YOUR_API_KEY export MONITIS_SANDBOX_SECRETKEY=REPLACE_THIS_WITH_YOUR_SECRET_KEY
There are four arguments to select from the supported AWS namespaces. These are used when creating monitors to make appropriate CloudWatch queries. They are –ec2, –rds, –elb, and –ebs.
CloudWatch reports five different kinds of metrics. These are Minimum, Maximum, Average, Sum, and SampleCount, using the flags –min, –avg, –max, –sum, and –count, respectively.
Sometimes, it is useful to group related metrics. Monitis-CloudWatch provides the –prefix argument to add a prefix to the name of created monitors. This prefix will be used in constructing the name both for –create and –update invocations. For example:
$ monitis_cloudwatch.py --create --elb --prefix loadbalancers.production \ --instance lb-http --metric RequestCount --sum $ monitis_cloudwatch.py --update --elb --prefix loadbalancers.production \ --instance lb-http --metric RequestCount --sum --from '-1 week midnight' \ --until 'today midnight' --period 3600 Using AWS data from 2012-02-05 00:00:00 until 2012-02-12 00:00:00 Posted 168 results to Monitis
Note that the prefix shows up in the graph displayed at www.monitis.com.
Monitis-CloudWatch in practice
Putting this into practice, the application can be used to periodically add recent metrics from Amazon AWS CloudWatch into Monitis custom monitors. Continuing with example above, create a cron job that queries CloudWatch for the previous hour’s total RequestCount, and add it to the custom monitor.
First, create a small shell script to set up the necessary environment variables and invoke monitis_cloudwatch.py.
export AWS_ACCESS_KEY_ID=******************** export AWS_SECRET_ACCESS_KEY=**************************************** export MONITIS_SECRETKEY=************************** export MONITIS_APIKEY=************************** /usr/local/bin/monitis_cloudwatch.py --update --elb --prefix loadbalancers.production \ --instance lb-http --metric RequestCount --sum --from 'last hour' --period 3600
With that in place, add it to a cron job that runs at the top of each hour.
0 * * * * sh $HOME/bin/http-lb-hourly.sh
Multiple invocations of monitis_cloudwatch.py can be added to the shell script, to provide for updates to multiple custom monitors. In addition, the frequency of the cron job could be increased to provide more frequent updates to the custom monitors, resulting in more timely graphs.
Try it free and signup Monitis now!.