I got a weird problem with my syslog-ng setup, im logging from alot of
cisco machines and that works great.
The problem is that when i "pass" this further to a shell program, some
lines disapere.
My destination looks like this
destination hosts {
file("/var/log/ciscorouters/$HOST.log"
owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes));
program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl));
};
The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats
putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first
line, if it gets flooded (like 5 rows per second).
Do anyone of you have any idea of what might be the problem?
It's hard to say without seeing the actual script. Is your script running
as a daemon or are you counting on syslog-ng to start your program per
message. If the latter, that's probably not the best strategy.
Another off topic (question) - what kind of fronted UI you use with syslog-ng? I see log analyser based on PHP is common. In my tests it worked fine but it’s major issue I saw was that I couldn’t sort all logs based on individual hosts/devices.
What kind of open source web UI everyone is using, just wondering?
Peter, it's a bit difficult to tell what's going on without seeing the
rest of the syslog-ng configuration and your script's source code.
However, a couple possibilities come to mind:
- Your script is only reading one line at a time. syslog-ng starts a
program() output persistently and expects that it can send multiple
messages into its pipe to your script's stdin.
- Messages are being buffered inside of syslog-ng. Check out the
flush_lines() and flush_timeout() flags to syslog-ng's program()
output. Find the right page for your version, but here's v3.3.: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html/reference_destination_program.html
- Messages are being buffered in your shell or script. Maybe try some
non-blocking IO with a smallish buffer to see data as it comes in
before a whole line or block fills and flushes in.
To Anurag's question about open source log management with a WebUI, I
agree with Blake: logstash ingesting syslog and inputting it into
elasticsearch makes for a great backend for Kibana.
The logstash grok filter is great for pulling apart and indexing weird
vendor-specific logging formats: http://logstash.net/docs/1.4.1/filters/grok