six demon bag

Wind, fire, all that kind of thing!

2016-09-01

Forwarding JVM Garbage Collector Logs

Recently I was tasked with forwarding logs to a central log server - in this case JVM garbage collector logs from Solr instances. Normally not a big deal. Configure log rotation (to avoid filling the disk), then have rsyslog read the active log file via an imfile input.

Sounds simple, right? Until you realize that JVM log rotation marks the active log with the extension .current and rotates that extension instead of actually rotating the logs.


ARE YOU F***ING KIDDING ME?!

In what alternate reality is that supposed to be a Good Idea™? Varying file names require either the use of wildcards, or multiple inputs. Rsyslog doesn't support the former prior to version 8.5.0, and the latter becomes a pain in the rear if you ever want to change the number of rotated files, because you need to modify your system configuration in two different (and otherwise unrelated) places.

Trying to dodge the issue by setting the number of rotated files to 1 doesn't work either, because then the single log file behaves like a ring buffer, removing old log entries from the beginning of the file as new entries are appended at the end. From that point forward (i.e. after the log reached its maximum size) rsyslog doesn't recognize log changes anymore, and therefore stops forwarding log messages.

What I ended up with is a 2-file rotation:

java \
  -Xloggc:logs/gc.log \
  -verbose:gc \
  -XX:+PrintGCDetails \
  -XX:+PrintGCDateStamps \
  -XX:+PrintGCTimeStamps \
  -XX:+UseGCLogFileRotation \
  -XX:NumberOfGCLogFiles=2 \
  -XX:GCLogFileSize=100M \
  ...

and two inputs (one for each log file when it becomes the active file):

# /etc/rsyslog.d/solr-gc-forward.conf
module(load="imfile" PollingInterval="10")

input(
  type="imfile"
  File="/opt/solr/logs/solr_gc.log.0.current"
  StateFile="solr-gc-0" 
  Tag="solr-gc"
  Severity="info"
)
input(
  type="imfile"
  File="/opt/solr/logs/solr_gc.log.1.current"
  StateFile="solr-gc-1" 
  Tag="solr-gc"
  Severity="info"
)

if $programname == 'solr-gc' then {
  action(
    type="omfwd"
    Target="loghost.example.org"
    Port="514" 
    Protocol="tcp" 
  )
  stop
}

Posted 23:23 [permalink]