Graphs Stopped Working After Disk Full

[GenericUser]

Hi All,

Last week our Cacti server's disk filled up which caused most of the graphs to stop working.
Disabling and re-enabling data sources caused some of the graphs, such as CPU usage, to become operational again; but not most of them, such as traffic statistics.

The rrd files are not being updated for the broken graphs.

I don't see any relevant errors in the log. Does anyone have an idea of how to get these graphs working again without deleting and recreating them?


EDIT:

I noticed that the Cacti web UI is only displaying 201 "Interface - Traffic" data sources while there are over 11,000 "Interface - Traffic" graphs.
The graphs associated with those 201 data sources are working.
All other data sources still exist and can be seen from the CLI, but have not been updated.

[cigamit]

Usually when this happens, it means you also corrupted some MySQL tables. You will need to repair them.

[GenericUser]

Okay thanks cigamit.

I had found a thread about recreating the poller_output table, but we tested that table and it was fine.

[cigamit]

You should just need to run "REPAIR TABLE" on all the tables.

[GenericUser]

Okay thank you.

We'll try it.

[GenericUser]

No luck.

Here's the result of the table repair:

mysqlcheck -u cactiuser -h localhost -p --repair --databases cacti
Enter password:
cacti.cdef OK
cacti.cdef_items OK
cacti.colors OK
cacti.data_input OK
cacti.data_input_data OK
cacti.data_input_fields OK
cacti.data_local OK
cacti.data_template OK
cacti.data_template_data OK
cacti.data_template_data_rra OK
cacti.data_template_rrd OK
cacti.graph_local OK
cacti.graph_template_input OK
cacti.graph_template_input_defs OK
cacti.graph_templates OK
cacti.graph_templates_gprint OK
cacti.graph_templates_graph OK
cacti.graph_templates_item OK
cacti.graph_tree OK
cacti.graph_tree_items OK
cacti.host OK
cacti.host_graph OK
cacti.host_snmp_cache OK
cacti.host_snmp_query OK
cacti.host_template OK
cacti.host_template_graph OK
cacti.host_template_snmp_query OK
cacti.plugin_aggregate_color_template_items OK
cacti.plugin_aggregate_color_templates OK
cacti.plugin_autom8_graph_rule_items OK
cacti.plugin_autom8_graph_rules OK
cacti.plugin_autom8_match_rule_items OK
cacti.plugin_autom8_tree_rule_items OK
cacti.plugin_autom8_tree_rules OK
cacti.plugin_config OK
cacti.plugin_db_changes OK
cacti.plugin_hooks OK
cacti.plugin_mURLin_cache OK
cacti.plugin_mURLin_index OK
cacti.plugin_notification_lists OK
cacti.plugin_realms OK
cacti.plugin_thold_contacts OK
cacti.plugin_thold_host_failed OK
cacti.plugin_thold_log OK
cacti.plugin_thold_template_contact OK
cacti.plugin_thold_threshold_contact OK
cacti.poller OK
cacti.poller_command OK
cacti.poller_item OK
cacti.poller_output OK
cacti.poller_output_rt OK
cacti.poller_reindex OK
cacti.poller_time OK
cacti.quicktree_graphs OK
cacti.rra OK
cacti.rra_cf OK
cacti.settings OK
cacti.settings_graphs OK
cacti.settings_tree OK
cacti.snmp_query OK
cacti.snmp_query_graph OK
cacti.snmp_query_graph_rrd OK
cacti.snmp_query_graph_rrd_sv OK
cacti.snmp_query_graph_sv OK
cacti.thold_data OK
cacti.thold_template OK
cacti.user_auth OK
cacti.user_auth_perms OK
cacti.user_auth_realm OK
cacti.user_log OK
cacti.version OK
cacti.weathermap_auth OK
cacti.weathermap_data OK
cacti.weathermap_groups OK
cacti.weathermap_maps OK
cacti.weathermap_settings OK

I then rebooted the server to get it to run a file system check.

Any other ideas?