How to Filter Logs with Syslog-ng

How to Filter Logs with Syslog-ngAfter the first article on syslog-ng, you should have a pretty good feel of how syslog-ng works. As you recall, the sources define what is logged, destinations determine where the logs go, and the log statements are what tells syslog-ng to create the log. However, much of the time we don’t want all of our system logs going to the same file. It can be quite useful to break up those logs in to multiple files, and to sort them by content. This is where filters come in.


How to Filter Logs with Syslog-ng


Filters act like log routers and select specific types of messages from based on content. There are multiple filter functions, which can be found in the admin guide as well. The syntax for filters is:

  filter { expression; };

Let’s say we want to filter out all authorization messages (such as Jul 29 09:45:54 Louis su[4162]: pam_unix(su:session): session opened for user root by (uid=1000) )into their own file, in which case we could include a filter similar to this:

filter f_auth { facility(auth); };

The facility expression can also use the Boolean operators and, not, and or. The following filters messages not coming from authpriv or mail:

  filter f_syslog { not facility(authpriv, mail); };

Another common and useful function is the level() function. This selects messages based on their priority level, and are often used like so:

filter f_notice { level(notice); };
filter f_warn { level(warn); };
filter f_crit { level(crit); };
filter f_err { level(err); };

Functions are often frequently combined for more complex filters, such as the following, which filters out messages with a priority level from informational to warning, yet do not come from the auth, authpriv, mail, or news facilities:

filter f_messages { level(info..warn)
 and not facility(auth, authpriv, mail, news); };

Putting it together

Now, let’s see how to use these filters with our log statements. As you recall, the log statements should have the following format:

log {
 source(s1); source(s2); ...
 destination(d1); destination(d2); ...
 flags(flag1[, flag2...]);

So, let’s say we want to create a log statement for mail. Providing the corresponding source, filter, and destination statements already exist, it might look something like this:

log { source(src); filter(f_mail); destination(mail); };

Sample syslog-ng.conf

If you recall from the previous article, the sample syslog-ng.conf file was pretty simple and easy to read. If we want to make good use of filters however, it is natural that our configuration files will become a little more complex, as we will want to log to many different files. Consider for example, this sample syslog-ng.conf from the Gentoo Security Handbook (the RHEL EPEL repository provides a similar although somewhat more simplified syslog-ng.conf):

options {
 # The default action of syslog-ng is to log a STATS line
 # to the file every 10 minutes. That's pretty ugly after a while.
 # Change it to every 12 hours so you get a nice daily update of
 # how many messages syslog-ng missed (0).
source src {
 unix-stream("/dev/log" max-connections(256));
source kernsrc { file("/proc/kmsg"); };
# define destinations
 destination authlog { file("/var/log/auth.log"); };
 destination syslog { file("/var/log/syslog"); };
 destination cron { file("/var/log/cron.log"); };
 destination daemon { file("/var/log/daemon.log"); };
 destination kern { file("/var/log/kern.log"); };
 destination lpr { file("/var/log/lpr.log"); };
 destination user { file("/var/log/user.log"); };
 destination mail { file("/var/log/mail.log"); };
 destination mailinfo { file("/var/log/"); };
 destination mailwarn { file("/var/log/mail.warn"); };
 destination mailerr { file("/var/log/mail.err"); };
 destination newscrit { file("/var/log/news/news.crit"); };
 destination newserr { file("/var/log/news/news.err"); };
 destination newsnotice { file("/var/log/news/news.notice"); };
 destination debug { file("/var/log/debug"); };
 destination messages { file("/var/log/messages"); };
 destination console { usertty("root"); };
# By default messages are logged to tty12...
 destination console_all { file("/dev/tty12"); };
# ...if you intend to use /dev/console for programs like xconsole
 # you can comment out the destination line above that references /dev/tty12
 # and uncomment the line below.
 #destination console_all { file("/dev/console"); };
# create filters
 filter f_authpriv { facility(auth, authpriv); };
 filter f_syslog { not facility(authpriv, mail); };
 filter f_cron { facility(cron); };
 filter f_daemon { facility(daemon); };
 filter f_kern { facility(kern); };
 filter f_lpr { facility(lpr); };
 filter f_mail { facility(mail); };
 filter f_user { facility(user); };
 filter f_debug { not facility(auth, authpriv, news, mail); };
 filter f_messages { level(info..warn)
 and not facility(auth, authpriv, mail, news); };
 filter f_emergency { level(emerg); };
 filter f_info { level(info); };
 filter f_notice { level(notice); };
 filter f_warn { level(warn); };
 filter f_crit { level(crit); };
 filter f_err { level(err); };
 filter f_failed { message("failed"); };
 filter f_denied { message("denied"); };

# connect filter and destination
log { source(src); filter(f_authpriv); destination(authlog); };
log { source(src); filter(f_syslog); destination(syslog); };
log { source(src); filter(f_cron); destination(cron); };
log { source(src); filter(f_daemon); destination(daemon); };
log { source(kernsrc); filter(f_kern); destination(kern); };
log { source(src); filter(f_lpr); destination(lpr); };
log { source(src); filter(f_mail); destination(mail); };
log { source(src); filter(f_user); destination(user); };
log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); };
log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); };
log { source(src); filter(f_mail); filter(f_err); destination(mailerr); };
log { source(src); filter(f_debug); destination(debug); };
log { source(src); filter(f_messages); destination(messages); };
log { source(src); filter(f_emergency); destination(console); };
# default log
log { source(src); destination(console_all); };


First, notice that the “options” and “source” sections of this configuration file aren’t much different for the more simplified file used in the previous article. Most of the good stuff happens in the “destination”, “filter”, and “log” sections. Let’s take, for instance, the authentication log. First, a destination is defined for “authlog” to record all logs in /var/log/auth.log. Next, a filter is created with the identifier “f_authpriv” for the “auth” and “authpriv” facilities, and lastly the filter and the destination are connected with the log statement “log { source(src); filter(f_authpriv); destination(authlog); };”. An examination of the above example will show the same sort of breakdown for many other information types, so that instead of all system logs being dumped into one file, they can be filtered and organized into individual and more logical logs.

So, now you can customize syslog-ng to perform basic logging, as well as configure a much more complex break down by employing filters to sort your system logs by content. However, syslog-ng has more still to offer us. After you have become comfortable with filters, take a look at How to Log to PostgreSQL with Syslog-ng (soon).