Syslog Plugin Very Slow

[chrisgapske]

WHen I am using the syslog plugin it can take up to two min to come back to me with the data results.
Has anybody done any work out there to speed this process up.

Any Ideas.

[warnesj]

Yes and no.

If I understand your question, and how the syslog plugin works, the reason it takes a couple of minutes (or up to 5 minutes) for the syslog messages to show up is because of the polling period of Cacti. The syslog messages are actually stored in a temporary table of your syslog database as they are recieved (the syslog_incoming table). Then when the main Cacti poller runs the syslog poller, those messages are examined and any filters you've defined are applied. After that the remaining messages are checked against any alerts you've defined. Once all the filters and alerts have been run, the filtered messages are then stored in the another table which is viewable in Cacti (the syslog table).

So is there a way to make those come in faster? Yes, run your poller at more frequent intervals. Though I don't know what this will do to the rest of your stuff.

[chrisgapske]

Yes and no.

If I understand your question, and how the syslog plugin works, the reason it takes a couple of minutes (or up to 5 minutes) for the syslog messages to show up is because of the polling period of Cacti. The syslog messages are actually stored in a temporary table of your syslog database as they are recieved (the syslog_incoming table). Then when the main Cacti poller runs the syslog poller, those messages are examined and any filters you've defined are applied. After that the remaining messages are checked against any alerts you've defined. Once all the filters and alerts have been run, the filtered messages are then stored in the another table which is viewable in Cacti (the syslog table).

So is there a way to make those come in faster? Yes, run your poller at more frequent intervals. Though I don't know what this will do to the rest of your stuff.

OK Almost but not not the problem I am having.

I am talking about using syslog plugin when I use the plugin it is very slow for me. Not the input of data from syslog sources but using the plugin from selecting to clicking opjects takes a very long time. I even cut my data storage to 2 days to somewhat shrink the database. It is just super slow.

[eternal]

Try to optimize the syslog mysql table.

OPTIMIZE TABLE table_name[,table_name]

[chrisgapske]

Thank you that helped. I am still a nebie DBA.

[sterpstra]

Hey guys...

Love this plugin!! It fits well into the system but I'm having some performance issues and wanted to get some opinions... I have several systems dumping their syslogs to this server. I originally had the archive set at 30 days and after the first 3 days, I was at about 20 million rows... I have since changed to 3 days but still average 8-10 million rows. My problem is two-fold:

1. It takes between 8-12 minutes to even bring up the plugin
2. It takes more than 15 minutes to search by Host/Text.

This is currently on a box with 2GB memory and an Intel Duo-core (I think 2.0Ghz). I understand there are obviously some general hardware issues when searching and I've just ordered a new Dell server with 4GB mem, Xeon Quad-Core and 10K drives. I'm curious how much this will affect the queries in SYSLOG. I would like to increase both the amount of hosts sending to this server as well as the days (would like 7 at least) before deletion.

Anyone had similar issues?

[TheWitness]

Depending on when you installed, you may have missed some very critical indexes. Do the following:

Code: Select all

mysql syslog
> show create table syslog;

From there, you should see the following:

Code: Select all

CREATE TABLE  `syslog` (
  `facility` varchar(10) default NULL,
  `priority` varchar(10) default NULL,
  `date` date default NULL,
  `time` time default NULL,
  `host` varchar(128) default NULL,
  `message` text,
  `seq` bigint(20) unsigned NOT NULL auto_increment,
  PRIMARY KEY  (`seq`),
  KEY `date` (`date`),
  KEY `time` (`time`),
  KEY `host` (`host`),
  KEY `priority` (`priority`),
  KEY `facility` (`facility`)
) ENGINE=MyISAM AUTO_INCREMENT=6498155 DEFAULT CHARSET=latin1;

You will note the "KEY" statements. If you are missing some, issue the following per index:

Code: Select all

ALTER TABLE syslog ADD INDEX `field` (`field`);

You can combine multiple ADD's into one statement. I believe that you must separate with comma though.

TheWitness

[sterpstra]

mysql> show create table syslog;
syslog | CREATE TABLE `syslog` (
`facility` varchar(10) default NULL,
`priority` varchar(10) default NULL,
`date` date default NULL,
`time` time default NULL,
`host` varchar(128) default NULL,
`message` text,
`seq` int(10) unsigned NOT NULL auto_increment,
PRIMARY KEY (`seq`),
KEY `date` (`date`),
KEY `time` (`time`),
KEY `host` (`host`),
KEY `priority` (`priority`),
KEY `facility` (`facility`)
) ENGINE=MyISAM AUTO_INCREMENT=28197952 DEFAULT CHARSET=latin1 |

1 row in set (0.02 sec)

Looks like they are all there

[TheWitness]

Looks like you need to tweak your MySQL Settings for syslog. Tell me about your system:

OS
CPU's
Core's
Physical Memory
Disk Number on MySQL mount
du -kh /var/www/html/cacti/rra
cat > /etc/my.cnf

Please provide that information and I will comment further. However, you need to optimize your system for MySQL. Syslog can become quite big, to the point of requiring your Cacti installation to have 1) fast disk (raid0 stripped, and 2) lot's of memory) or moving Syslog itself to it's own database. If you have lot's of RRA's "du -kh" is more than 60% of your physical memory, you need to split the web site and database into separate servers.

I have been working with one of the Cacti developers on a Syslog update that will resolve "some" of the issues.

TheWitness

[sterpstra]

OS
CPU's
Core's
Physical Memory
Disk Number on MySQL mount
du -kh /var/www/html/cacti/rra
cat > /etc/my.cnf

Thanks for the help... Here ya go:

Ubuntu 8.04/Hardy [Linux 2.6.24-16-server]
1 CPU x Intel Core 2 3.9Ghz
2GB RAM

207M /var/lib/cacti/rra
CAT Results:
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
skip-external-locking
bind-address = 127.0.0.1
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 128K
thread_cache_size = 8
query_cache_limit = 1M
query_cache_size = 16M
expire_logs_days = 10
max_binlog_size = 100M
skip-bdb
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/

[TheWitness]

Increase your key bugger to 768MB. That should help a bit. Then restart MySQL.

TheWitness

[streaker69]

Increase your key bugger to 768MB. That should help a bit. Then restart MySQL.

TheWitness

You do mean 'buffer', right?

[TheWitness]

lol. Yes. Just got back from Japan. I guess I needed sleep more than I thought

TheWitness

[spoonman]

Hello all....I seem to have the same issue and here are my thoughts on the plugin slowness to view...which btw I love and its so helpful especially having it as a plugin in cacti. I have about 30 or so devices i receive syslog data from...it seems to me that the reason syslog tab seems slow is the amount of data its trying to read but in the end it only shows the top 100 or whatever entries on the page....to me could the plugin only pull those top 100 or 200 entries from the database to present to the viewer?? Can the viewer only view the top rows?? instead of it seems to pull the whole table before it presents the view?? just my opinion...by no means do I know how the code is or am i a dba///

Thanks guys...esp Witness...whose helped me numerous times..

Spoon

[cigamit]

Hello all....I seem to have the same issue and here are my thoughts on the plugin slowness to view...which btw I love and its so helpful especially having it as a plugin in cacti. I have about 30 or so devices i receive syslog data from...it seems to me that the reason syslog tab seems slow is the amount of data its trying to read but in the end it only shows the top 100 or whatever entries on the page....to me could the plugin only pull those top 100 or 200 entries from the database to present to the viewer?? Can the viewer only view the top rows?? instead of it seems to pull the whole table before it presents the view?? just my opinion...by no means do I know how the code is or am i a dba///

Thanks guys...esp Witness...whose helped me numerous times..

Spoon

It should only be pulling the exact number of messages it needs. The biggest issue is when there are no indexes on the table, so it is extremely slow to query which results it needs.