[SOLVED] Poller crash - MySQL error

[terziyski]

For some reason CACTID gives me the message:

CACTID: Poller[0] ERROR: Problem with MySQL: 'Can't open file: 'poller_output.MYI' (errno: 145)'
and I didn't get any updates in my graphs.

After mysql --autorepair cacti, it said that poller_output table has been repaired.After that I rebuilt the poller cache and error messages disapeared, but for a while.
After two poll intervals Cacti gives me again the same error:

CACTID: Poller[0] ERROR: Problem with MySQL: 'Can't open file: 'poller_output.MYI' (errno: 145)'

Any suggestions how can I fix this ?

[TheWitness]

Your database is damaged. Goto the cli directory and run:

Code: Select all

php -q database_repair.php

TheWitness

[terziyski]

Thank you for your quick answer. Unfotunately I am using Cacti 0.8.6j and I do not have CLI folder at all. I have managed to fix the problem with deleting and then recreating poller_output table by sql commands from cacti.sql file.

[Diggit2001]

Sorry to hijack this thread but I started experiencing the "CACTID: Poller[0] ERROR: Problem with MySQL: 'Can't open file: 'poller_output.MYI' (errno: 145)'" error this morning after adding some new graph templates. I ran the repair_database.php file and this did seem to at least make my database usable again. Unfortunately, stuff is not quite normal yet. CPU usage on my cacti server is much higher than normal and using the Cacti site is much slower now. My logs are showing this every poller cycle:

05/07/2008 04:46:54 PM - SYSTEM STATS: Time:112.9826 Method:spine Processes:1 Threads:5 Hosts:72 HostsPerProcess:72 DataSources:3839 RRDsProcessed:1682
05/07/2008 04:45:03 PM - CACTID: Poller[0] Host[1] DS[2112] WARNING: Result from SCRIPT not valid. Partial Result: ...
05/07/2008 04:45:03 PM - CACTID: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
05/07/2008 04:45:03 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into settings values ('syslog_last_incoming','')'

I am not exactly sure what this means but it seems to be my biggest clue at the moment. Can anyone suggest what I should try next? I appreciate the help.
Thanks!
-Chris

[Diggit2001]

OK, I have another clue. I came into work this morning and opened Cacti like I usually do and was told that the database could not be found. When I attempt to open the cacti database through webmin, I am told, "DBI connect failed : Too many connections". I rebooted the cacti box and then was able to log on. Upon logging on, I found that nothing has been working since around 7:00 PM last night. All graphs are empty after that point.

I'm not sure exactly what the repair_database.php did, but it at least allowed me to get back in to the database. Does anyone know what I should look at/what I should do to get back up and running?

Thanks much.
-Chris

[Diggit2001]

Whoa. I just noticed that the hard drive usage on my Cacti box has been steadily increasing since yesterday when all this started. Looks like I now have only about 2 GB free as opposed to the 40 GB free I had 2 days ago. Being a Linux novice, I'm not too sure how to tell what the heck is taking up all the space. Could it be my database that slowly growing?

The future looks bleak. I am sad.

[Diggit2001]

It was my /var/log/mysqld.log file that was taking up all the space. It had grown to 48 GB and I had only 300 MB free on the drive before I figured it out. I deleted the file and rebooted the server. I can now log on to the cacti site again and I re-enabled the poller cron job since I had shut it down earlier to prevent further growage.

Now that I can log back on, I went in and had a loog at the mysql log. All that's in it is this, logged about 3 times a second:

080508 11:33:01 [ERROR] /usr/libexec/mysqld: Can't open file: 'syslog_incoming.MYI' (errno: 145)

Can anyone tell me what this means?

Thanks.
-Chris

[Diggit2001]

I found a post on cactiusers.org describing this same issue, that recommended running this:

mysqlcheck --auto-repair --databases syslog
mysqlcheck --auto-repair --databases cacti

I did that and received this:

[root@hbicacti ~]# mysqlcheck --auto-repair --databases syslog
syslog.syslog
warning : 2 clients are using or haven't closed the table properly
status : OK
syslog.syslog_alert OK
syslog.syslog_incoming
warning : Table is marked as crashed
warning : 2 clients are using or haven't closed the table properly
error : Record at pos: 1062960 is not remove-marked
error : record delete-link-chain corrupted
error : Corrupt
syslog.syslog_remove OK

Repairing tables
syslog.syslog_incoming
warning : Number of rows changed from 5 to 6
status : OK
[root@hbicacti ~]# mysqlcheck --auto-repair --databases cacti
cacti.cdef OK
cacti.cdef_items OK
cacti.colors OK
cacti.data_input OK
cacti.data_input_data OK
cacti.data_input_fields OK
cacti.data_local OK
cacti.data_template OK
cacti.data_template_data OK
cacti.data_template_data_rra OK
cacti.data_template_rrd OK
cacti.graph_local OK
cacti.graph_template_input OK
cacti.graph_template_input_defs OK
cacti.graph_templates OK
cacti.graph_templates_gprint OK
cacti.graph_templates_graph OK
cacti.graph_templates_item OK
cacti.graph_tree OK
cacti.graph_tree_items OK
cacti.host OK
cacti.host_graph OK
cacti.host_snmp_cache OK
cacti.host_snmp_query OK
cacti.host_template OK
cacti.host_template_graph OK
cacti.host_template_snmp_query OK
cacti.mac_track_device_types OK
cacti.mac_track_devices OK
cacti.mac_track_ips OK
cacti.mac_track_ports OK
cacti.mac_track_processes OK
cacti.mac_track_scan_dates OK
cacti.mac_track_scanning_functions OK
cacti.mac_track_sites OK
cacti.mac_track_temp_ports OK
cacti.manage_alerts OK
cacti.manage_device_type OK
cacti.manage_groups OK
cacti.manage_host OK
cacti.manage_method OK
cacti.manage_poller_output OK
cacti.manage_process OK
cacti.manage_services OK
cacti.manage_sites OK
cacti.manage_tcp OK
cacti.manage_templates OK
cacti.plugin_discover_hosts OK
cacti.plugin_discover_os OK
cacti.plugin_flowview_dnscache
note : The storage engine for the table doesn't support check
cacti.plugin_report_definitions OK
cacti.plugin_report_items OK
cacti.plugin_report_last_values OK
cacti.plugin_reports OK
cacti.plugin_thold_contacts OK
cacti.plugin_thold_log OK
cacti.plugin_thold_template_contact OK
cacti.plugin_thold_threshold_contact OK
cacti.plugin_update_info OK
cacti.poller OK
cacti.poller_command OK
cacti.poller_item OK
cacti.poller_output OK
cacti.poller_reindex OK
cacti.poller_time OK
cacti.rra OK
cacti.rra_cf OK
cacti.settings OK
cacti.settings_graphs OK
cacti.settings_tree OK
cacti.snmp_query OK
cacti.snmp_query_graph OK
cacti.snmp_query_graph_rrd OK
cacti.snmp_query_graph_rrd_sv OK
cacti.snmp_query_graph_sv OK
cacti.thold OK
cacti.thold_data OK
cacti.thold_template OK
cacti.tholdset OK
cacti.uptime OK
cacti.uptime_data OK
cacti.user_auth OK
cacti.user_auth_perms OK
cacti.user_auth_realm OK
cacti.user_log OK
cacti.version OK
cacti.weathermap_auth OK
cacti.weathermap_data OK
cacti.weathermap_maps OK

The sql log continues to grow, filling up with the same info I mentioned above and my cacti log is still full of this:

05/08/2008 12:36:47 PM - SYSTEM STATS: Time:104.8778 Method:spine Processes:1 Threads:5 Hosts:72 HostsPerProcess:72 DataSources:3839 RRDsProcessed:1686
05/08/2008 12:35:02 PM - CACTID: Poller[0] Host[1] DS[2112] WARNING: Result from SCRIPT not valid. Partial Result: ...
05/08/2008 12:35:02 PM - CACTID: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
05/08/2008 12:35:02 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into settings values ('syslog_last_incoming','1877252')'

Graphs are now being updated and CPU usage on the cacti box is lower but stuff still ain't right.

Thanks
-Chris

[smartech1]

Hi Chris,.....

I ran the following two commands you mentioned boss. But the first command for 'syslog' says " select database", I am affraid syslog is not a database in my case.

However second command run succesfully and got same result as you got.
But I m sad to see my server CPU &Memory utilization going high and I am unable to login the server at all............

Well then I rebooted it and was fine for some time, but again started to hang.

Could you please let me know anything else you did to over come this.

Thanks & regards,
S.toufeeq

[gandalf]

I answered to your other thread already. Please do not crosspost
Reinhard

[dieselboy]

I found a post on cactiusers.org describing this same issue, that recommended running this:

mysqlcheck --auto-repair --databases syslog
mysqlcheck --auto-repair --databases cacti

-Chris

Thanks Chris,
After searching for hours I finally found this post of yours which fixed my problem. The issue for me was similar cacti logs and no syslogs in the plugin at all - even though these were working fine and no changes had been made.
I'm still getting the "Poller[0] ERROR: A DB Exec Failed!, Error:'1062'" in the cacti log which I cannot explain, though. I give up on that.

Thanks!