Syslog stopped working

[fredeve]

Hi all,
Suddently my Syslog stopped logging, and i cant seem to find out why. When i ssh into the server it tells me:

Code: Select all

Message from syslogd@2014 at Jan 31 08:57:15 ...
 IP-address KERNEL(AP-name@IP-address): [1289623.911483] Watchdog petted by same CPU 1 for 0 seconds. jiffies[644811753] History[0010101010101010101001101010101010101010100101010101001101010]

Message from syslogd@2014 at Feb  1 21:05:24 ...
 IP-address KERNEL(AP-name@IP-address): [1419713.555414] Watchdog petted by same CPU 1 for 0 seconds. jiffies[709856575] History[0010101010101010101010100101101010101010101010101010100101010]

Message from syslogd@2014 at Feb  6 03:30:35 ...
 IP-address KERNEL(AP-name@IP-address): [1788422.900091] Watchdog petted by same CPU 1 for 0 seconds. jiffies[894211247] History[0101010101010101010101010101010101010101010100110011001101010]

The message is from an access point on our network. I have configured the controller to send syslog messages, but not the AP.

Im running CactiEZ v0.7

Cacti log file:

Code: Select all

02/06/2014 03:28:09 PM - SYSTEM SYSLOG STATS:Time:0.34 Deletes:0 Incoming: Removes:0 XFers: Alerts:0 Alarms:0 Reports:0
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"DELETE FROM `syslog`.`syslog_incoming` WHERE status=110'
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog` (logtime, priority_id, facility_id, host_id, message) SELECT TIMESTAMP(`date`, `time`), priority_id, facility_id, host_id, message FROM (SELECT date, time, priority_id, facility_id, host_id, message FROM syslog_incoming AS si INNER JOIN syslog_facilities AS sf ON sf.facility=si.facility INNER JOIN syslog_priorities AS sp ON sp.priority=si.priority INNER JOIN syslog_hosts AS sh ON sh.host=si.host WHERE status=110) AS merge'
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'145', SQL:"SELECT count(*) FROM `syslog`.`syslog_incoming` WHERE status=110"
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_statistics` (host_id, facility_id, priority_id, insert_time, records) SELECT host_id, facility_id, priority_id, NOW(), sum(records) AS records FROM (SELECT host_id, facility_id, priority_id, count(*) AS records FROM syslog_incoming AS si INNER JOIN syslog_facilities AS sf ON sf.facility=si.facility INNER JOIN syslog_priorities AS sp ON sp.priority=si.priority INNER JOIN syslog_hosts AS sh ON sh.host=si.host WHERE status=110 GROUP BY host_id, priority_id, facility_id) AS merge GROUP BY host_id, priority_id, facility_id'
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_host_facilities` (host_id, facility_id) SELECT host_id, facility_id FROM ((SELECT DISTINCT host, facility FROM `syslog`.`syslog_incoming` WHERE status=110) AS s INNER JOIN `syslog`.`syslog_hosts` AS sh ON s.host=sh.host INNER JOIN `syslog`.`syslog_facilities` AS sf ON sf.facility=s.facility) ON DUPLICATE KEY UPDATE host_id=VALUES(host_id), last_updated=NOW()'
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_hosts` (host) SELECT DISTINCT host FROM `syslog`.`syslog_incoming` WHERE status=110 ON DUPLICATE KEY UPDATE host=VALUES(host), last_updated=NOW()'
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_priorities` (priority) SELECT DISTINCT priority FROM `syslog`.`syslog_incoming` ON DUPLICATE KEY UPDATE priority=VALUES(priority), last_updated=NOW()'
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_facilities` (facility) SELECT DISTINCT facility FROM `syslog`.`syslog_incoming` ON DUPLICATE KEY UPDATE facility=VALUES(facility), last_updated=NOW()'
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"UPDATE `syslog`.`syslog_incoming` SET priority='warn' WHERE priority='warning''
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"UPDATE `syslog`.`syslog_incoming` SET status=110 WHERE status=0'
02/06/2014 03:28:09 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'145', SQL:"SELECT count(*) FROM `syslog`.`syslog_incoming` WHERE status=110"
02/06/2014 03:27:10 PM - SYSTEM SYSLOG STATS:Time:0.35 Deletes:0 Incoming: Removes:0 XFers: Alerts:0 Alarms:0 Reports:0
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"DELETE FROM `syslog`.`syslog_incoming` WHERE status=89'
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog` (logtime, priority_id, facility_id, host_id, message) SELECT TIMESTAMP(`date`, `time`), priority_id, facility_id, host_id, message FROM (SELECT date, time, priority_id, facility_id, host_id, message FROM syslog_incoming AS si INNER JOIN syslog_facilities AS sf ON sf.facility=si.facility INNER JOIN syslog_priorities AS sp ON sp.priority=si.priority INNER JOIN syslog_hosts AS sh ON sh.host=si.host WHERE status=89) AS merge'
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'145', SQL:"SELECT count(*) FROM `syslog`.`syslog_incoming` WHERE status=89"
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_statistics` (host_id, facility_id, priority_id, insert_time, records) SELECT host_id, facility_id, priority_id, NOW(), sum(records) AS records FROM (SELECT host_id, facility_id, priority_id, count(*) AS records FROM syslog_incoming AS si INNER JOIN syslog_facilities AS sf ON sf.facility=si.facility INNER JOIN syslog_priorities AS sp ON sp.priority=si.priority INNER JOIN syslog_hosts AS sh ON sh.host=si.host WHERE status=89 GROUP BY host_id, priority_id, facility_id) AS merge GROUP BY host_id, priority_id, facility_id'
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_host_facilities` (host_id, facility_id) SELECT host_id, facility_id FROM ((SELECT DISTINCT host, facility FROM `syslog`.`syslog_incoming` WHERE status=89) AS s INNER JOIN `syslog`.`syslog_hosts` AS sh ON s.host=sh.host INNER JOIN `syslog`.`syslog_facilities` AS sf ON sf.facility=s.facility) ON DUPLICATE KEY UPDATE host_id=VALUES(host_id), last_updated=NOW()'
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_hosts` (host) SELECT DISTINCT host FROM `syslog`.`syslog_incoming` WHERE status=89 ON DUPLICATE KEY UPDATE host=VALUES(host), last_updated=NOW()'
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_priorities` (priority) SELECT DISTINCT priority FROM `syslog`.`syslog_incoming` ON DUPLICATE KEY UPDATE priority=VALUES(priority), last_updated=NOW()'
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"INSERT INTO `syslog`.`syslog_facilities` (facility) SELECT DISTINCT facility FROM `syslog`.`syslog_incoming` ON DUPLICATE KEY UPDATE facility=VALUES(facility), last_updated=NOW()'
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"UPDATE `syslog`.`syslog_incoming` SET priority='warn' WHERE priority='warning''
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"UPDATE `syslog`.`syslog_incoming` SET status=89 WHERE status=0'
02/06/2014 03:27:10 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'145', SQL:"SELECT count(*) FROM `syslog`.`syslog_incoming` WHERE status=89"

Any help would be greatly appreciated!

[fredeve]

I found the problem. The incoming table in the syslog mysql database crashed.

[dogs1005]

I found the problem. The incoming table in the syslog mysql database crashed.

If you could be so kind as to explain what you actually did to fix this please as I have the same issue.

EDIT
For anyone else having this issue you just need to type the following command:

Code: Select all

mysqlcheck --user=cactiuser -p --auto-repair --databases syslog

The password for cactiuser can be found in the file /root/mysqlpass.txt

[ffaiz]

Thankyou so much,
You recommended solution worked for me

[ffaiz]

For this error:
04/05/2018 09:23:56 AM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'145', SQL:"SELECT count(*) FROM `syslog`.`syslog_incoming` WHERE status=23"

After running your recommended cmd i got the mentioned below Output

[root@nms-cli]# mysqlcheck --user=cactiuser -p --auto-repair --databases syslog
Enter password:
syslog.syslog OK
syslog.syslog_alert OK
syslog.syslog_facilities
warning : 3 clients are using or haven't closed the table properly
status : OK
syslog.syslog_host_facilities
warning : 3 clients are using or haven't closed the table properly
status : OK
syslog.syslog_hosts
warning : 3 clients are using or haven't closed the table properly
status : OK
syslog.syslog_incoming
warning : Table is marked as crashed
warning : 1 client is using or hasn't closed the table properly
error : Record at pos: 47812 is not remove-marked
error : record delete-link-chain corrupted
error : Corrupt
syslog.syslog_logs OK
syslog.syslog_priorities
warning : 3 clients are using or haven't closed the table properly
status : OK
syslog.syslog_remove OK
syslog.syslog_removed OK
syslog.syslog_reports OK
syslog.syslog_statistics OK

Repairing tables
syslog.syslog_incoming
warning : Number of rows changed from 5 to 81
status : OK



And i was also getting mentioned below logs as well and because of this my graphs stopped plotting:
04/05/2018 09:16:47 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1194', SQL:"select poller_output.output, poller_output.time, UNIX_TIMESTAMP(poller_output.time) as unix_time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"

So i ran mentioned the mentioned below syntax - output also mentioned:

[root@nms-cli]#cd /var/www/html/cli/
[root@nms-cli]# php -q repair_database.php
Repairing All Cacti Database Tables
Repairing Table -> 'cdef' Successful
Repairing Table -> 'cdef_items' Successful
Repairing Table -> 'colors' Successful
Repairing Table -> 'data_input' Successful
Repairing Table -> 'data_input_data' Successful
Repairing Table -> 'data_input_fields' Successful
Repairing Table -> 'data_local' Successful
Repairing Table -> 'data_source_stats_daily' Successful
Repairing Table -> 'data_source_stats_hourly' Successful
Repairing Table -> 'data_source_stats_hourly_cache' Successful
Repairing Table -> 'data_source_stats_hourly_last' Successful
Repairing Table -> 'data_source_stats_monthly' Successful
Repairing Table -> 'data_source_stats_weekly' Successful
Repairing Table -> 'data_source_stats_yearly' Successful
Repairing Table -> 'data_template' Successful
Repairing Table -> 'data_template_data' Successful
Repairing Table -> 'data_template_data_rra' Successful
Repairing Table -> 'data_template_rrd' Successful
Repairing Table -> 'graph_local' Successful
Repairing Table -> 'graph_template_input' Successful
Repairing Table -> 'graph_template_input_defs' Successful
Repairing Table -> 'graph_templates' Successful
Repairing Table -> 'graph_templates_gprint' Successful
Repairing Table -> 'graph_templates_graph' Successful
Repairing Table -> 'graph_templates_item' Successful
Repairing Table -> 'graph_tree' Successful
Repairing Table -> 'graph_tree_items' Successful
Repairing Table -> 'host' Successful
Repairing Table -> 'host_graph' Successful
Repairing Table -> 'host_snmp_cache' Successful
Repairing Table -> 'host_snmp_query' Successful
Repairing Table -> 'host_template' Successful
Repairing Table -> 'host_template_graph' Successful
Repairing Table -> 'host_template_snmp_query' Successful
Repairing Table -> 'mac_track_aggregated_ports' Successful
Repairing Table -> 'mac_track_approved_macs' Successful
Repairing Table -> 'mac_track_device_types' Successful
Repairing Table -> 'mac_track_devices' Successful
Repairing Table -> 'mac_track_interface_graphs' Successful
Repairing Table -> 'mac_track_interfaces' Successful
Repairing Table -> 'mac_track_ip_ranges' Successful
Repairing Table -> 'mac_track_ips' Successful
Repairing Table -> 'mac_track_macauth' Successful
Repairing Table -> 'mac_track_macwatch' Successful
Repairing Table -> 'mac_track_oui_database' Successful
Repairing Table -> 'mac_track_ports' Successful
Repairing Table -> 'mac_track_processes' Successful
Repairing Table -> 'mac_track_scan_dates' Successful
Repairing Table -> 'mac_track_scanning_functions' Successful
Repairing Table -> 'mac_track_sites' Successful
Repairing Table -> 'mac_track_snmp' Successful
Repairing Table -> 'mac_track_snmp_items' Successful
Repairing Table -> 'mac_track_temp_ports' Successful
Repairing Table -> 'mac_track_vlans' Successful
Repairing Table -> 'plugin_aggregate_color_template_items' Successful
Repairing Table -> 'plugin_aggregate_color_templates' Successful
Repairing Table -> 'plugin_aggregate_graph_templates' Successful
Repairing Table -> 'plugin_aggregate_graph_templates_item' Successful
Repairing Table -> 'plugin_aggregate_graphs' Successful
Repairing Table -> 'plugin_aggregate_graphs_graph_item' Successful
Repairing Table -> 'plugin_aggregate_graphs_items' Successful
Repairing Table -> 'plugin_autom8_graph_rule_items' Successful
Repairing Table -> 'plugin_autom8_graph_rules' Successful
Repairing Table -> 'plugin_autom8_match_rule_items' Successful
Repairing Table -> 'plugin_autom8_tree_rule_items' Successful
Repairing Table -> 'plugin_autom8_tree_rules' Successful
Repairing Table -> 'plugin_config' Successful
Repairing Table -> 'plugin_db_changes' Successful
Repairing Table -> 'plugin_discover_hosts' Successful
Repairing Table -> 'plugin_discover_template' Successful
Repairing Table -> 'plugin_flowview_devices' Successful
Repairing Table -> 'plugin_flowview_dnscache' Successful
Repairing Table -> 'plugin_flowview_ports' Successful
Repairing Table -> 'plugin_flowview_queries' Successful
Repairing Table -> 'plugin_flowview_schedules' Successful
Repairing Table -> 'plugin_hmib_hrDevices' Successful
Repairing Table -> 'plugin_hmib_hrProcessor' Successful
Repairing Table -> 'plugin_hmib_hrSWInstalled' Successful
Repairing Table -> 'plugin_hmib_hrSWRun' Successful
Repairing Table -> 'plugin_hmib_hrSWRun_ignore' Successful
Repairing Table -> 'plugin_hmib_hrSWRun_last_seen' Successful
Repairing Table -> 'plugin_hmib_hrStorage' Successful
Repairing Table -> 'plugin_hmib_hrSystem' Successful
Repairing Table -> 'plugin_hmib_hrSystemTypes' Successful
Repairing Table -> 'plugin_hmib_processes' Successful
Repairing Table -> 'plugin_hmib_types' Successful
Repairing Table -> 'plugin_hooks' Successful
Repairing Table -> 'plugin_maint_hosts' Successful
Repairing Table -> 'plugin_maint_schedules' Successful
Repairing Table -> 'plugin_nectar' Successful
Repairing Table -> 'plugin_nectar_items' Successful
Repairing Table -> 'plugin_notification_lists' Successful
Repairing Table -> 'plugin_realms' Successful
Repairing Table -> 'plugin_thold_contacts' Successful
Repairing Table -> 'plugin_thold_host_failed' Successful
Repairing Table -> 'plugin_thold_log' Successful
Repairing Table -> 'plugin_thold_template_contact' Successful
Repairing Table -> 'plugin_thold_threshold_contact' Successful
Repairing Table -> 'poller' Successful
Repairing Table -> 'poller_command' Successful
Repairing Table -> 'poller_item' Successful
Repairing Table -> 'poller_output' Successful
Repairing Table -> 'poller_output_boost' Successful
Repairing Table -> 'poller_output_boost_processes' Successful
Repairing Table -> 'poller_output_rt' Successful
Repairing Table -> 'poller_reindex' Successful
Repairing Table -> 'poller_time' Successful
Repairing Table -> 'rra' Successful
Repairing Table -> 'rra_cf' Successful
Repairing Table -> 'settings' Successful
Repairing Table -> 'settings_graphs' Successful
Repairing Table -> 'settings_tree' Successful
Repairing Table -> 'snmp_query' Successful
Repairing Table -> 'snmp_query_graph' Successful
Repairing Table -> 'snmp_query_graph_rrd' Successful
Repairing Table -> 'snmp_query_graph_rrd_sv' Successful
Repairing Table -> 'snmp_query_graph_sv' Successful
Repairing Table -> 'superlinks_auth' Successful
Repairing Table -> 'superlinks_pages' Successful
Repairing Table -> 'thold_data' Successful
Repairing Table -> 'thold_template' Successful
Repairing Table -> 'user_auth' Successful
Repairing Table -> 'user_auth_perms' Successful
Repairing Table -> 'user_auth_realm' Successful
Repairing Table -> 'user_log' Successful
Repairing Table -> 'version' Successful
Repairing Table -> 'weathermap_auth' Successful
Repairing Table -> 'weathermap_data' Successful
Repairing Table -> 'weathermap_groups' Successful
Repairing Table -> 'weathermap_maps' Successful
Repairing Table -> 'weathermap_settings' Successful

NOTE: Checking for Invalid Cacti Templates
NOTE: 2 Invalid CDEF Item Rows Found in Graph Templates

WARNING: Serious Cacti Template Problems found in your Database. Using the '--force' option will remove
the invalid records. However, these changes can be catastrophic to existing data sources. Therefore, you
should contact your support organization prior to proceeding with that repair.