Syslog monitor addon beta

[Pumpi]

ok. I

I have added haloe.sql file but it seems that cigamit has modified other parts from the orginal haloe 1.2b files.

I should ask him

[knobdy]

Just out of curiosity, I'd like to know how many devices you all are monitoring and at what level?

We run, for instance, most of our firewalls at debug. Trying to put all of that into the database and search on it simply KILLED our previous logging server - using a similar system. Is this plugin that much more efficient?

I know one of the issues with the old system is that we were compressing the files hourly - meaning that to search them they would have to be expanded into a temporary table, etc..

[cigamit]

Problem adding syslog.sql to haloe database (using plugin):

I use command mysql haloe < syslog.sql after I have created the dabase haloe but I get following error:

ERROR 1064 at line 5: You have an error in your SQL Syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near 'collate_latin1_general_ci default NULL, priority varchar(10)

I'm running MySQL server version 4.0.18 on aSUSE Enterprise Server 9.

Fixed, I had already taken it out on my side (My second test box runs Mysql 4), but didn't happen to remove it from the one I posted online.

Thanks!

[cigamit]

Just out of curiosity, I'd like to know how many devices you all are monitoring and at what level?

We run, for instance, most of our firewalls at debug. Trying to put all of that into the database and search on it simply KILLED our previous logging server - using a similar system. Is this plugin that much more efficient?

I know one of the issues with the old system is that we were compressing the files hourly - meaning that to search them they would have to be expanded into a temporary table, etc..

When I first started testing this, I had 38 windows servers, all dumping all Event Logs (several of them domain controllers dumping 10+ authenication events every second). This worked fine, but was very ineffecient when it came to trying to process new events in order to alert and remove ones we didn't want. So I switched it to a 2 table scheme, where all new events are added to the incoming table, and from there they would be parsed, alerted on, or removed. Everything left over was moved to the normal "viewing / storage" database. At the time, my server could easily handle 100,000 events in one processing (processes every 5 minutes after the poller) and querying the database wasn't very slow at all. I did find Msyslog to begin choking at around 150,000/per 5 minutes, as it would just start dropping random ones when they were coming in that fast, I didn't try the option to delay inserts, which probably would have helped slightly, but only if the alerts ever slowed down.

Since then, I have ofcourse added lots of removal rules to define things I don't ever need to see. Changed the Windows Servers to only report errors or problems, and modified lots of code, so who knows what it can handle now (maybe for the worst). My current viewing database only has 300,000 records and 42 hosts. Running 20 removal rules, and normally only process 200 new records every polling, with 10-15 actually making past the filters (sometimes up to 2000). I even have nice pretty little graphs to display the amount processed, etc..

Dumping lots of info to an MySQL database is ofcourse going to make anything go slow. Our current Netflow implmentation logs to a MySQL database, is currently 3.4 GB and holds 28 Million records. Querying that beast takes a good 10-50 seconds depending on the query. Compressing the database if you are going to be doing lots of queries just compounds the issue even further. It also depends on what type of hardware you are running, we use custom built rackmount Chasis running 1.2 Duron and 1.5G Ram. The box also polls 42 hosts, runs NTop, scans via Mactrack every 30 minutes, and for the heck of it, scans via the Discovery plugin. Load average is around .5 (higher if doing lots of Netflow queries) and usually only using around half the memory.

[cigamit]

Did a minor change to syslog which could be easy implemented.
I wanted to display the number of entry's per ip in the syslog database in order to see problems on certain switches or routers faster.

Changed some code in haloe.php

|snip|

Hope someone can use this

Michael

Thanks, I like it and will be adding it in sortly. Possibly as an option as I could see this slowing down the querying alot if you have lots of hosts and/or events.

[knobdy]

Yeah, that's kind of what I figured. I have a guy on the firewalls-wizards list that will, hopefully, be providing me with some ideas on how to prune down the debug messages I'll be processing, but even then I won't be able to put them into mysql - just way too slow. Thanks.

[Jeppe]

I have about 90 servers logging their syslogs into an innodb table.
On average that's 80 events per second. Just testing so far how
the system is able to handle this. Querying does get to be a little
slow when the incoming table has grown to a few millions lines.

Cigamit, planning on releasing your version? Sounds good.
Would let me not do it myself.

-J

[Pumpi]

cigamit,

still having a problem to add syslog.sql

ERROR 1604 at line 5: ..... near 'CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, ....

Can I remove this item ?

Pumpi

[cigamit]

cigamit,

still having a problem to add syslog.sql

ERROR 1604 at line 5: ..... near 'CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, ....

Can I remove this item ?

Pumpi

What version of MySQL are you running?

[Pumpi]

I'm running MySQL server version 4.0.18 on aSUSE Enterprise Server 9.

I'm waiting that the internal update server is working again to update a lot of stuff, but this the the orginal version on the CD set.

Pumpi

[flavour]

Seems like CURRENT_TIMESTAMP is only implemented in MySQL 4.1+ :/

Not sure if there's an easy workaround....

F

[cigamit]

Seems like CURRENT_TIMESTAMP is only implemented in MySQL 4.1+ :/

Not sure if there's an easy workaround....

F

The code that currently uses it isn't very critical, so I can hopefully just remove certain parts of it.

[kabttu]

I'm sure I'm doing something wrong here but I figured I'd ask and see. I've got syslog-ng piping the data to mysql and the plugin is working and displaying the syslogs, my problem is that I'll get this message:

syslog-ng[26507]: syslog-ng version 1.6.9 going down

and the only way to get things working again is to restart syslog-ng. Any ideas?

Thanks!

[ccsdnoc]

GREAT tool! I was able to get it to work, I did have to make the color change to match the lowercase of my logs "info, warn, crit", etc. I am still unable to get syslog-ng to run on startup as well as the pipe script "syslogpip.sh". If anyone could help out, that'd be great!!

P.S. I am using Fedora Core 4

[egarnel]

GREAT tool! I was able to get it to work, I did have to make the color change to match the lowercase of my logs "info, warn, crit", etc. I am still unable to get syslog-ng to run on startup as well as the pipe script "syslogpip.sh". If anyone could help out, that'd be great!!

P.S. I am using Fedora Core 4

can't you just put it in /etc/rc.d/rc.local ?

be sure to use full paths for the commands