Forward syslog messages to flume with rsyslog

2015-08-27 14:47

As usual, brain dump, just instructions, not much content.

download flume from here: https://flume.apache.org/download.html

I'm using this one: http://www.apache.org/dyn/closer.cgi/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz

unpack and put it somewhere.

create a file with the following content, I will name it flume-syslog.conf and place it in ~/tmp/, you should too if you are lazy and don't want to change the commands:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# I'll be using TCP based Syslog source
a1.sources.r1.type = syslogtcp
# the port that Flume Syslog source will listen on
a1.sources.r1.port = 7077
# the hostname that Flume Syslog source will be running on
a1.sources.r1.host = localhost

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

Install rsyslog if you don't have it and start it, I'm using fedora 22, change for your distro:

sudo dnf install rsyslog
sudo service rsyslog start

Configure rsyslog with your rule, you can do it directly on /etc/rsyslog.conf or better, check that the following line is uncommented:

$IncludeConfig /etc/rsyslog.d/*.conf

And put your config under /etc/rsyslog.d/50-default.conf (create it if it doesn't exist)

We are going to forward only messages with a given tag, since we are interested on a subset of the logs, in this case we only want log lines with the tag "test", add this to the rsyslog config file:

:syslogtag, isequal, "test:" @@127.0.0.1:7077

Save and restart rsyslog:

sudo service rsyslog start

Start flume with your configuration:

./bin/flume-ng agent --conf conf --conf-file ~/tmp/flume-syslog.conf --name a1 -Dflume.root.logger=INFO,console  -Dorg.apache.flume.lifecycle.LifecycleSuperviso=INFO,console

Now generate a log line with our tag:

logger -t test 'Testing Flume with Syslog!

you should see a line like this:

2015-08-27 18:06:25,096 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{host=ganesha, Severity=5, Facility=1, priority=13, timestamp=1440695180000} body: 74 65 73 74 3A 20 54 65 73 74 69 6E 67 20 46 6C test: Testing Fl }

If you don't see the line check /var/log/messages to see if your message is there:

sudo vim /var/log/messages

Bonus track! sending apache logs to syslog and from there to flume.

for this install apache 2, on fedora:

sudo dnf install httpd
sudo service httpd start
sudo bash -c "echo 'welcome!' > /var/www/html/index.html"

curl localhost

The output should be:

welcome!

Now configure apache to forward logs to syslog, open /etc/httpd/conf.d/welcome.conf and add at the bottom:

CustomLog "|/usr/bin/logger -t test" combined

Restar apache:

sudo service httpd restart

Now open the page or use curl to get a page:

/etc/httpd/conf.d/welcome.conf

You should see a new log on flume.

Where to go from here?

Put flume on another machine, change the ip address 127.0.0.1 to that address
change the tag (test) on rsyslog and on welcome.conf to something else
Buy me a beer

Mariano Guerra's Log