Enterprise syslog management and alert generation.

Some people call this 'Netcool' or products of a similiar stripe. I'm ramping up a project to rebuild some previous work done on this front with an open source distribution in mind (those of you on the syslog-ng list have seen mention of it), so I'm fishing for requirements I may not have already covered.

I currently have:
Perl regexp engine for applied rules.
Tokenization and extraction of data from inbound syslog data.
Assigning (single|multiple) customized event handlers to rule matches
Ability to run multiple analyzers concurrently
Optional linear rule application versus weighted optimization
SQL storage of rules for centralized management and redistribution.
Fully customized alert generation.

My current production implementation has handled over 20 gigs a day, at peak, on a single analyzer (dual amd 2800+), using syslog-ng as a transport mechanism (forked socket transport with local disk logging for backup).

Every network is different, as are particular requirements. Who's got wish lists? I personally wouldn't mind an on-list discussion about this, as it applies to standard operations toolsets, but if that's not kosher, feel free to contact me off-list.

- billn

In such products, only 20% value is in engine; 80% are in rules, because I
can not wrire rules myself - I have not event until it happen, and I can not
filetr out noice until it happen.

We use a few syslog analyzers (using syslog-ng as a transport), some with
simple logcheck, other with database for rules and hosts; and every time
problem is the same - writing rules is 90% of the problem.

But... do you have rules, such as fort example _send alert if any system
began to generate 10 times logs / hour more vs. average? Or saying _single
PCI ERROR on Solaris - ignore, 10 in a straight line - send warning...

The X over time is a new one, it's been mentioned a couple times today, and I can certainly account for it. I've added it to my rapidly growing list.

- billn