Greetings,
Anyone has good recommendations for an open-sourced log parsing and
analyzing application? It will be used to work with syslog-ng and other
general syslog and application logs.
I have been looking at swatch and logwatch, but would like to find out if
there are other good choices, thanks
FD
Splunk
ZanOSS
PHP-Syslog-NG aka logzilla
LogLogic
SEC (Simplet Event Correlator) is a very effective tool for this, IMHO. I
am by no means an expert with it, but I know several people who are, and
while it is not as well known as splunk or some other tools, I have been
very impressed by the results I've seen using it.
As with any event correlation tool, there is a significant level of invested
effort required to make use of this.
http://simple-evcorr.sourceforge.net/
Below is a presentation about SEC.
I personally like SEC (Simple Event Correlator), check out
http://simple-evcorr.sourceforge.net/
Jeff Rooney
jtrooney@nexdlevel.com
ah, never heard of SEC before and it really looks interesting,
Thanks everyone for the great input!
FD
Take a look at SLCT, also by Risto Vaarandi:
http://ristov.users.sourceforge.net/slct/
SLCT can parse huge amounts of logs very fast. We use it to
crunch firewall logs and also to find ports that are flapping
excessively.
Dale
+1, SLCT definitely finds the needles in haystacks of huge syslog files
Gord
SEC does seem to be the "gold standard" in advanced log correlation beyond
that available in "grep | mail" type systems such as logwatch. However it
is incredibly arcane, and despite reading a lot of documentation for it I've
never really been able to wrap my head around it.
A colleague has started to write a SEC-like tool with (I hope) a more
approachable mental model; take a look at http://github.com/rodjek/grok. I
must (embarrasedly) admit I haven't looked at it yet, but he claims that he
reimplemented sshd_sentry (the fail2ban equivalent we use) in two lines of
rules, which seems like a nice (basic) demonstration.
- Matt