Odd syslog-ng problem


I got a weird problem with my syslog-ng setup, im logging from alot of
cisco machines and that works great.
The problem is that when i "pass" this further to a shell program, some
lines disapere.

My destination looks like this
destination hosts {
   owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes));
   program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl));
The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats
putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first
line, if it gets flooded (like 5 rows per second).

Do anyone of you have any idea of what might be the problem?


It's hard to say without seeing the actual script. Is your script running
as a daemon or are you counting on syslog-ng to start your program per
message. If the latter, that's probably not the best strategy.

Off topic.
The issue is with the daemon, not your devices.


Another off topic (question) - what kind of fronted UI you use with syslog-ng? I see log analyser based on PHP is common. In my tests it worked fine but it’s major issue I saw was that I couldn’t sort all logs based on individual hosts/devices.

What kind of open source web UI everyone is using, just wondering?


I use kibana / elasticsearch


Peter, it's a bit difficult to tell what's going on without seeing the
rest of the syslog-ng configuration and your script's source code.

However, a couple possibilities come to mind:
- Your script is only reading one line at a time. syslog-ng starts a
program() output persistently and expects that it can send multiple
messages into its pipe to your script's stdin.
- Messages are being buffered inside of syslog-ng. Check out the
flush_lines() and flush_timeout() flags to syslog-ng's program()
output. Find the right page for your version, but here's v3.3.:
- Messages are being buffered in your shell or script. Maybe try some
non-blocking IO with a smallish buffer to see data as it comes in
before a whole line or block fills and flushes in.

To Anurag's question about open source log management with a WebUI, I
agree with Blake: logstash ingesting syslog and inputting it into
elasticsearch makes for a great backend for Kibana.
The logstash grok filter is great for pulling apart and indexing weird
vendor-specific logging formats: