Years ago I've set up a central logserver for our infrastructure at Compare Group. Back then I decided to use rsyslog for this. I've been using rsyslog for a long time. I've never liked it much. The weird syntax, the bad documentation with the many assumptions and opinions in it have not made it easy to debug weird behaviour. I think I'm not alone in this. That is why I was very happy when logstash came into existence. It has its own flaws, but at least it's easier to debug and use. I don't like spending a lot of time rewriting our (huge) rsyslog configuration to logstash in one go and I do want all logs to arrive on the same box. Since not all apps support logging to a different port than 514, it'll be either rsyslog or logstash that's going to listen on that port and handing off messages to the other as we slowly migrate from the one to the other.
Because I've decided that logstash is our new default, I've configured it to listen to port 514. Most of the logic to parse and write out messages is still in rsyslog, so many messages that arrive on our central logserver need to be forwarded from logstash to rsyslog. It seems there aren't many people doing this. Even searching (use www.duckduckgo.com) for this topic only finds references for people sending from rsyslog to logstash, not the other way around. So I figured this couldn't be that hard.
Indeed it's not 'hard' as in difficult. It's just a lot of work puzzeling the pieces together.
You'll need a logstash 2.2 or newer with the syslog output module. Maybe it'll work with older versions, but 2.2 is the version I built and tested this setup with. It's not included by default so install the plugin.
Below you'll find the resulting configurations for logstash that I've used. I'll explain the setup:
The input listens on udp and tcp. We receive most messages from servers via tcp but not all applications and devices support tcp so we need udp also.
the syslog output module by default sets the programname, facility and priority to a default that it gets from logstash. so all messages are then sent as if coming from logstash from the logstash server with the same facility and priority.
So the received message needs to be parsed by logstash and based on what the message contents is, we can overwrite the output plugin's defaults and set them back to the values that logstash received from the remove device.
We also want to specifically keep the original date-time the message was generated at the source, not the date-time of logstash. Due to buffering (especially when there are problems or is maintenance with the central log server) there may be large time differences between the source time and the logstash time.
Similarly not all devices send an RFC compliant date/time, so try to match those into the logstash format.
Because not all messages have a process-id, one must be set or the variable name will be printed in the output when it is not set. We don't want that, so the default is "-".
Once all that parsing is done and the messages can be put through, we can start to prepare for the migration. In this case we'll be matching the program name and putting that in specific folders based on the sending hostname and files according to the application. Many applications can log to the same file, but based on access rights some log to other files. Not everyone needs to see everything.
# This configuration togs incoming messages for processing by filters in other
# logstash config files. That way we have 1 place where the 'type' is declared
# and the logic of the processing can happens somewhere else. That should keep
# our implementation of logstash cleaner and more manageable.
input {
udp {
type => 'generic-syslog'
port => 514
buffer_size => 65536
workers => 10
}
tcp {
type => 'generic-syslog'
port => 514
}
}
filter {
# Detect what type of message it is, then tag it accordingly.
# Manual grok because of not using syslog input
grok {
# This match is based on the logstash pattern SYSLOGLINE but modified to
# better match our needs.
match => { "message" => "<%{POSINT:syslog_pri}>(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|)(\s)%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
# Decode the facility and priority of the message.
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
# if the PID was not found/set, make it a - for the syslog output to work properly towards rsyslog
if !([pid] =~ /.+/) {
mutate { add_field => [ "pid", "-" ] }
}
# --------------------------- configureable part starts here -----------------------
# don't change anything above this line
# use http://stackoverflow.com/questions/29673424/logstash-replace-field-values-matching-pattern for this, it seems better
# firewall
#$template firewall, "/export/logs/%HOSTNAME%/security/%$YEAR%-%$MONTH%-%$DAY%/firewall.log"
#:syslogtag, contains, "firewall:" ?firewall
if [program] == "firewall" {
mutate { replace => ["type", "firewall"] }
}
# cron
#$template crond, "/export/logs/%HOSTNAME%/management/%$YEAR%-%$MONTH%-%$DAY%/cron.log"
#cron.* ?crond
#& ~
if [program] == "cron" or [program] == "crond" {
mutate { replace => ["type", "cron"] }
}
# sudo
# sshd
#
}
output {
# if [type] == 'generic-syslog' or [type] == 'firewall' or [type]=='cron' {
# stdout { codec => rubydebug }
# }
if [type] == 'cron' {
file {
path => "/export/logs/%{short_host}/management/%{+YYYY-MM-dd}/cron.by.logstash.log"
codec => line { format => "%{message}" }
flush_interval => "0"
}
}
if [type] == 'firewall' {
file {
path => "/export/logs/%{short_host}/security/%{+YYYY-MM-dd}/firewall.by.logstash.log"
codec => line { format => "%{message}" }
flush_interval => "0"
}
}
# These are sent by logback/log4j through rsyslog by the application.
if [type] == 'generic-syslog' {
# file {
# path => "/export/logs/debugging/send-to-syslog.log"
# codec => line { format => "%{message}" }
# flush_interval => "0"
# }
syslog {
appname => "%{program}"
procid => "%{pid}"
message => "%{syslog_message}"
facility => "%{syslog_facility}"
severity => "%{syslog_severity}"
sourcehost => "%{logsource}"
host => "localhost"
port => 5140
workers => 10
}
}
if [type] == 'generic-syslog' and [tag] == "_grokparsefailure" {
file {
path => "/export/logs/debugging/.grokfailure.log"
codec => line { format => "%{syslog_message} AND %{message}" }
flush_interval => "0"
}
}
}
Because I've decided that logstash is our new default, I've configured it to listen to port 514. Most of the logic to parse and write out messages is still in rsyslog, so many messages that arrive on our central logserver need to be forwarded from logstash to rsyslog. It seems there aren't many people doing this. Even searching (use www.duckduckgo.com) for this topic only finds references for people sending from rsyslog to logstash, not the other way around. So I figured this couldn't be that hard.
Indeed it's not 'hard' as in difficult. It's just a lot of work puzzeling the pieces together.
You'll need a logstash 2.2 or newer with the syslog output module. Maybe it'll work with older versions, but 2.2 is the version I built and tested this setup with. It's not included by default so install the plugin.
Below you'll find the resulting configurations for logstash that I've used. I'll explain the setup:
The input listens on udp and tcp. We receive most messages from servers via tcp but not all applications and devices support tcp so we need udp also.
the syslog output module by default sets the programname, facility and priority to a default that it gets from logstash. so all messages are then sent as if coming from logstash from the logstash server with the same facility and priority.
So the received message needs to be parsed by logstash and based on what the message contents is, we can overwrite the output plugin's defaults and set them back to the values that logstash received from the remove device.
We also want to specifically keep the original date-time the message was generated at the source, not the date-time of logstash. Due to buffering (especially when there are problems or is maintenance with the central log server) there may be large time differences between the source time and the logstash time.
Similarly not all devices send an RFC compliant date/time, so try to match those into the logstash format.
Because not all messages have a process-id, one must be set or the variable name will be printed in the output when it is not set. We don't want that, so the default is "-".
Once all that parsing is done and the messages can be put through, we can start to prepare for the migration. In this case we'll be matching the program name and putting that in specific folders based on the sending hostname and files according to the application. Many applications can log to the same file, but based on access rights some log to other files. Not everyone needs to see everything.
# This configuration togs incoming messages for processing by filters in other
# logstash config files. That way we have 1 place where the 'type' is declared
# and the logic of the processing can happens somewhere else. That should keep
# our implementation of logstash cleaner and more manageable.
input {
udp {
type => 'generic-syslog'
port => 514
buffer_size => 65536
workers => 10
}
tcp {
type => 'generic-syslog'
port => 514
}
}
filter {
# Detect what type of message it is, then tag it accordingly.
# Manual grok because of not using syslog input
grok {
# This match is based on the logstash pattern SYSLOGLINE but modified to
# better match our needs.
match => { "message" => "<%{POSINT:syslog_pri}>(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|)(\s)%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
# Decode the facility and priority of the message.
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
# if the PID was not found/set, make it a - for the syslog output to work properly towards rsyslog
if !([pid] =~ /.+/) {
mutate { add_field => [ "pid", "-" ] }
}
# --------------------------- configureable part starts here -----------------------
# don't change anything above this line
# use http://stackoverflow.com/questions/29673424/logstash-replace-field-values-matching-pattern for this, it seems better
# firewall
#$template firewall, "/export/logs/%HOSTNAME%/security/%$YEAR%-%$MONTH%-%$DAY%/firewall.log"
#:syslogtag, contains, "firewall:" ?firewall
if [program] == "firewall" {
mutate { replace => ["type", "firewall"] }
}
# cron
#$template crond, "/export/logs/%HOSTNAME%/management/%$YEAR%-%$MONTH%-%$DAY%/cron.log"
#cron.* ?crond
#& ~
if [program] == "cron" or [program] == "crond" {
mutate { replace => ["type", "cron"] }
}
# sudo
# sshd
#
}
output {
# if [type] == 'generic-syslog' or [type] == 'firewall' or [type]=='cron' {
# stdout { codec => rubydebug }
# }
if [type] == 'cron' {
file {
path => "/export/logs/%{short_host}/management/%{+YYYY-MM-dd}/cron.by.logstash.log"
codec => line { format => "%{message}" }
flush_interval => "0"
}
}
if [type] == 'firewall' {
file {
path => "/export/logs/%{short_host}/security/%{+YYYY-MM-dd}/firewall.by.logstash.log"
codec => line { format => "%{message}" }
flush_interval => "0"
}
}
# These are sent by logback/log4j through rsyslog by the application.
if [type] == 'generic-syslog' {
# file {
# path => "/export/logs/debugging/send-to-syslog.log"
# codec => line { format => "%{message}" }
# flush_interval => "0"
# }
syslog {
appname => "%{program}"
procid => "%{pid}"
message => "%{syslog_message}"
facility => "%{syslog_facility}"
severity => "%{syslog_severity}"
sourcehost => "%{logsource}"
host => "localhost"
port => 5140
workers => 10
}
}
if [type] == 'generic-syslog' and [tag] == "_grokparsefailure" {
file {
path => "/export/logs/debugging/.grokfailure.log"
codec => line { format => "%{syslog_message} AND %{message}" }
flush_interval => "0"
}
}
}
