Syslog Stopped Working

[BigO]

Cacti 0.8.7g / PIA 2.9 / Syslog 1.22 stopped working. The server had been working fine but is no longer processing incoming syslog events. I can see that systems are sending syslogs to TCP/514 and that they are landing in the syslog_incoming table. I have run the ../cacti/plugins/syslog/syslog.php --debug and everything looks OK. I am now seeing an error in the httpd logs though.

SYSLOG: Syslog Table IS Partitioned
SYSLOG: Unique ID = 45
SYSLOG: Found 0, New Message(s) to process
SYSLOG: Stats , Record(s) to the 'syslog_statistics' table
SYSLOG: Found 8, Removal Rule(s) to process
SYSLOG: Deleted 0, Messages for removal rule 'Riverbed Junk removal'
SYSLOG: Deleted 10, Messages for removal rule 'New Removal Rule'
SYSLOG: Deleted 10, Messages for removal rule 'debug removal'
SYSLOG: Deleted 10, Messages for removal rule 'info junk '
SYSLOG: Deleted 10, Messages for removal rule 'Crond info removal'
SYSLOG: Deleted 11, Message for removal rule 'debug delete'
SYSLOG: Deleted 11, Messages for removal rule 'Delete Cert Failed alert for Cert 7BC0261A000100000007'
SYSLOG: Deleted 40, Messages for removal rule 'New Removal Rule'
SYSLOG: Found 19, Alert Rules to process
SYSLOG: Moved , Message(s) to the 'syslog' table
SYSLOG: Deleted 39, Already Processed Message(s) from incoming
SYSLOG: Deleted 0, Syslog Statistics Record(s)
SYSLOG: Deleted 0, Syslog alarm log Record(s)
SYSLOG: Processing Reports...
SYSLOG: We have 0 Reports in the database
SYSLOG: Finished processing Reports...
01/16/2014 04:26:48 PM - SYSTEM SYSLOG STATS:Time:0.12 Deletes:0 Incoming:0 Removes:40 XFers:0 Alerts:19 Alarms:0 Reports:0

=========== ERROR FROM HTTP ERROR LOG ====================
PHP Notice: Undefined index: 12 in /var/www/html/cacti/plugins/syslog/syslog.php on line 1307

============ ERRORS FROM CACTI LOG =======================
01/17/2014 07:45:42 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `cactidb`.`syslog` (logtime, priority_id, facility_id, host_id, message) SELECT TIMESTAMP(`date`, `time`), priority_id, facility_id, host_id, message FROM (SELECT date, time, priority_id, facility_id, host_id, message FROM syslog_incoming AS si INNER JOIN syslog_facilities AS sf ON sf.facility=si.facility INNER JOIN syslog_priorities AS sp ON sp.priority=si.priority INNER JOIN syslog_hosts AS sh ON sh.host=si.host WHERE status=16) AS merge'
01/17/2014 07:45:42 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `cactidb`.`syslog_statistics` (host_id, facility_id, priority_id, insert_time, records) SELECT host_id, facility_id, priority_id, NOW(), sum(records) AS records FROM (SELECT host_id, facility_id, priority_id, count(*) AS records FROM syslog_incoming AS si INNER JOIN syslog_facilities AS sf ON sf.facility=si.facility INNER JOIN syslog_priorities AS sp ON sp.priority=si.priority INNER JOIN syslog_hosts AS sh ON sh.host=si.host WHERE status=16 GROUP BY host_id, priority_id, facility_id) AS merge GROUP BY host_id, priority_id, facility_id'
01/17/2014 07:45:42 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `cactidb`.`syslog_host_facilities` (host_id, facility_id) SELECT host_id, facility_id FROM ((SELECT DISTINCT host, facility FROM `cactidb`.`syslog_incoming` WHERE status=16) AS s INNER JOIN `cactidb`.`syslog_hosts` AS sh ON s.host=sh.host INNER JOIN `cactidb`.`syslog_facilities` AS sf ON sf.facility=s.facility) ON DUPLICATE KEY UPDATE host_id=VALUES(host_id), last_updated=NOW()'
01/17/2014 07:45:42 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `cactidb`.`syslog_facilities` (facility) SELECT DISTINCT facility FROM `cactidb`.`syslog_incoming` ON DUPLICATE KEY UPDATE facility=VALUES(facility), last_updated=NOW()'

Any help would be greatly appreciated.

Thanks,
Scott

[BigO]

Problem solved.

Appears to have been something in the database. I ran mysqlcheck and that seems to have worked. This is what the mysqlcheck showed.


#> mysqlcheck --auto-repair --databases cacti

cactidb.syslog
warning : 1 client is using or hasn't closed the table properly
warning : 2 clients are using or haven't closed the table properly
warning : 1 client is using or hasn't closed the table properly
warning : 1 client is using or hasn't closed the table properly
warning : 1 client is using or hasn't closed the table properly
warning : 1 client is using or hasn't closed the table properly
warning : 2 clients are using or haven't closed the table properly
warning : 2 clients are using or haven't closed the table properly
warning : 1 client is using or hasn't closed the table properly
warning : 1 client is using or hasn't closed the table properly
warning : 2 clients are using or haven't closed the table properly
warning : 1 client is using or hasn't closed the table properly
status : OK
cactidb.syslog_alarm_log OK
cactidb.syslog_alert OK
cactidb.syslog_facilities
warning : Table is marked as crashed
warning : 8 clients are using or haven't closed the table properly
error : Checksum for key: 3 doesn't match checksum for records
error : Corrupt
cactidb.syslog_host_facilities
warning : 8 clients are using or haven't closed the table properly
status : OK
cactidb.syslog_hosts
warning : 9 clients are using or haven't closed the table properly
status : OK
cactidb.syslog_incoming OK
cactidb.syslog_logs
warning : 9 clients are using or haven't closed the table properly
status : OK
cactidb.syslog_priorities
warning : 9 clients are using or haven't closed the table properly
status : OK
cactidb.syslog_remove OK
cactidb.syslog_removed OK
cactidb.syslog_reports OK
cactidb.syslog_statistics
warning : 9 clients are using or haven't closed the table properly
status : OK