Skip to main content

Mariano Guerra's Log

Hi, I'm Mariano Guerra, below is my blog, if you want to learn more about me and what I do check a summary here: marianoguerra.github.io or find me on twitter @warianoguerra or Mastodon @marianoguerra@hachyderm.io

Forward syslog messages to flume with rsyslog

As usual, brain dump, just instructions, not much content.

download flume from here: https://flume.apache.org/download.html

I'm using this one: http://www.apache.org/dyn/closer.cgi/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz

unpack and put it somewhere.

create a file with the following content, I will name it flume-syslog.conf and place it in ~/tmp/, you should too if you are lazy and don't want to change the commands:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# I'll be using TCP based Syslog source
a1.sources.r1.type = syslogtcp
# the port that Flume Syslog source will listen on
a1.sources.r1.port = 7077
# the hostname that Flume Syslog source will be running on
a1.sources.r1.host = localhost

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

Install rsyslog if you don't have it and start it, I'm using fedora 22, change for your distro:

sudo dnf install rsyslog
sudo service rsyslog start

Configure rsyslog with your rule, you can do it directly on /etc/rsyslog.conf or better, check that the following line is uncommented:

$IncludeConfig /etc/rsyslog.d/*.conf

And put your config under /etc/rsyslog.d/50-default.conf (create it if it doesn't exist)

We are going to forward only messages with a given tag, since we are interested on a subset of the logs, in this case we only want log lines with the tag "test", add this to the rsyslog config file:

:syslogtag, isequal, "test:" @@127.0.0.1:7077

Save and restart rsyslog:

sudo service rsyslog start

Start flume with your configuration:

./bin/flume-ng agent --conf conf --conf-file ~/tmp/flume-syslog.conf --name a1 -Dflume.root.logger=INFO,console  -Dorg.apache.flume.lifecycle.LifecycleSuperviso=INFO,console

Now generate a log line with our tag:

logger -t test 'Testing Flume with Syslog!

you should see a line like this:

2015-08-27 18:06:25,096 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{host=ganesha, Severity=5, Facility=1, priority=13, timestamp=1440695180000} body: 74 65 73 74 3A 20 54 65 73 74 69 6E 67 20 46 6C test: Testing Fl }

If you don't see the line check /var/log/messages to see if your message is there:

sudo vim /var/log/messages

Bonus track! sending apache logs to syslog and from there to flume.

for this install apache 2, on fedora:

sudo dnf install httpd
sudo service httpd start
sudo bash -c "echo 'welcome!' > /var/www/html/index.html"

curl localhost

The output should be:

welcome!

Now configure apache to forward logs to syslog, open /etc/httpd/conf.d/welcome.conf and add at the bottom:

CustomLog "|/usr/bin/logger -t test" combined

Restar apache:

sudo service httpd restart

Now open the page or use curl to get a page:

/etc/httpd/conf.d/welcome.conf

You should see a new log on flume.

Where to go from here?

  • Put flume on another machine, change the ip address 127.0.0.1 to that address

  • change the tag (test) on rsyslog and on welcome.conf to something else

  • Buy me a beer