Monitor your Java application logs in 4 easy steps

As systems administrators, application logs are often the key to our success, but also our biggest hassle. They provide clues to what’s going on when things go awry, and in those situations more detail is generally better. But when you don’t actually know something is wrong, and just want to get a sense for whether things are normal, more detail can create so much noise that it’s all but impossible to glean any useful information.

In those situations, you’d rather just have statistical information about what’s in your logs. In this article, I present a simple and easy solution to turn your logs into useful graphs, in real time. If you ever need to measure the volume of your logs, or perhaps graph the frequency of certain log events, then read on.




The tools

The solution I present uses four key tools:

  • Log4J (though plain log files would fit as well)
  • Logstash
  • StatsD
  • Monitis

With so many moving parts, you might be tempted to think this could be an overcomplicated solution. But in fact — as in the long tradition of Unix command line tools — it is a composition of simple tools each doing one job very well. As with files piped from one Unix command to another, these four components act as a pipeline for log events, with each piece adding value to the stream along the way.


All of the log events in this article start inside of Log4J. If you run Java applications, then this provides an easy way to hook into your logs, to peel off an event stream that you want to see graphed in Monitis. But, Log4J could easily be replaced in this solution with plain log files, syslog, or any number of other logging frameworks.

The key modification that we make to Log4J is to add a SocketAppender that sends a copy of selected Loggers to our logstash server.


The role of logstash in the pipeline is twofold. First, it listens for connections from Java application servers, accepting streams of logs when they connect. Second, it filters, modifies, and routes those streams to the appropriate outputs. In this case, we’ll be handling all of the incoming streams by notifying StatsD each time a log event is received, without actually sending the content of each event.


Logstash will be receiving log events very frequently, but Monitis only wants to receive updates at most once per minute. To resolve this mismatch, StatsD acts as our log stream bean counter, allowing logstash to send increment messages each time an event is received. StatsD records these in counters for each type of log message, and then sends the counts on to Monitis every 60 seconds.


Finally, we get to the end of the pipeline, and Monitis receives the count messages. These are added to the appropriate custom monitors, which are automatically created if they don’t already exist. Once the data is in Monitis, it can be graphed in the Web UI, or used to send alerts when a rate of log events is outside of a user-specified threshold.


The gory details

Now that you’ve seen the overview, let’s take a look at the configuration details that make it happen. Don’t worry, since each component in the pipeline is doing a simple job, there’s really not much to it.

Install and configure the software

Let’s look at installation details for the tools in each step in the pipeline. I’m assuming that you already have Java applications using Log4J. If not, modifying the pipeline to read from log files, receive from syslog, or other options is pretty straightforward, but outside the scope of this article. For that, refer to the logstash documentation on how to set up other kinds of logstash inputs.


In Log4J, we need to set up a SocketAppender. The good news is that it’s actually easier to configure than the common appenders you’re probably already familiar with. And, it’s already built in to Log4J.

As shown below, add a SocketAppender named LOGSTASH to your application’s log4j.xml configuration file. Make sure that you change the RemoteHost parameter to match your logstash server.

<appender name="LOGSTASH" class="">
    <param name="RemoteHost" value="your_logstash_host_address" />
    <param name="ReconnectionDelay" value="60000" />
    <param name="Threshold" value="DEBUG" />

Then, add that appender to your root logger, or to any other loggers in your configuration where you’d like to collect trends in Monitis.

    <level value="INFO"/>
    <appender-ref ref="SERVER"/>
    <appender-ref ref="LOGSTASH"/>

Log4J won’t try to send logs to a server unless it can establish a connection. This won’t happen until logstash is up and running, so it’s okay to activate this configuration now.


Next, set up logstash. There are many, many inputs, outputs, and filters available, but we’ll focus on just four of them.

We’ll start with the log4j input. There are only two options you need to set. First, the type, which is an arbitrary string used to identify these messages later, when configuring the filters and outputs. I chose “log4j-type” as my identifier. And second, the port, which is probably best left as 4560, since that is the default expected by the Log4J SocketAppender.

input {
  log4j {
    type => "log4j-type"
    port => 4560

Next, we need to filter the events as they come in. Java application logs often contain stack traces, and it is better to have the entire stack trace associated with the exception that generated it. The multiline filter takes all messages that start with whitespace (based on the regex in the pattern), and associates them with the previous line.

Before finishing the filtering, we also need to create an identifier that the output can pass on to StatsD, for naming each of the counters. We want this to include the IP address of the host the logs came from, so that counters from multiple hosts can be distinguished. Use the mutate filter to create a new field in the event, called log4j_ip, by copying in the value from _@source_host_. _@source_host_ looks like “a.b.c.d:y”, where “a.b.c.d” is the IP address, and y is the client port number. We just want the IP, so use the mutate filter again to strip off the colon and everything after.

filter {
  multiline {
    type => "log4j-type"
    pattern => "^\\s"
    what => "previous"
  mutate {
    add_field => [ "log4j_ip", "%{@source_host}" ]
  mutate {
    gsub => ["log4j_ip", ":.*$", ""]

And last, for the output, use statsd to send increment messages to the StatsD server, including the Log4J priority and the log4j_ip field we created earlier. By default, logstash assumes that StatsD is on the same host. If that’s not the case in your setup, you can also specify the host parameter.

output {
  file {
    path => "debug.log"
  statsd {
    # Count one hit every event by priority
    increment => "jboss.log.%{priority}"
    sender => "%{log4j_ip}"

StatsD needs to be configured to send updates to Monitis, using statsd-monitis-backend. Instructions for installing StatsD and the Monitis backend are in our previous article, Simple metric aggregation and automated custom monitors with Monitis and StatsD. Add your Monitis API key and secret key.

  backends: ["statsd-monitis-backend"],
  flushInterval: 60000,
  dumpMessages: false,
  monitis: {
    apikey: 'your_monitis_apikey',
    secretkey: 'your_monitis_secret_key',
    debug: false


With our pipeline configured, and all of the components running, you should see log events flowing from one stage to the next, and ultimately into Monitis. After you let it run for a while, there’s nothing left to do but pull up some graphs, and enjoy your work.

Here are a some images from Monitis showing the event volume for several log priorities and from multiple systems. This first image shows the pipeline handling some spikes in volume over 20,000 log events per minute.

The second shows different log priorities on the same host as above.

And finally, the third chart shows log volumes from a different host.


Of course, this isn’t just limited to log priorities. By changing the Log4J loggers pointing to the LOGSTASH appender, adding logstash filters, and/or changing the fields referenced in statsd.increment in the logstash config, it is possible to measure whatever you want. The power of building up this pipeline is that — with a little ingenuity — you can measure almost anything.