mysql mount ran out of space... all graphs NAN now

[maw]

Hi all,

My /var(mysql DB) mount ran out of available space causing cacti to fail for 2days straight. Since rectifying the space issue, the poller is no longer functioning correctly. I have rebuilt the poller cache twice but the 'cacti log' file isnt updating anymore and my graphs show NAN now.

The reason the mount ran out of space was due to the mysql option "log-bin" being enabled. I have since commented that line out of config file (/etc/my.cnf). Additionally, I zipped up my original apache log file. These are the only two changes I made on purpose other then the disk running out of space. I attached the bottom part of my cacti.log since it stopped updating. Please note the cronjob is still executing.

Done anyone know how to fix this?

Thanks,
maw

[maw]

I restarted my server and checked the cacti.log again. It has been updated with a barrage of SQL errors. eg:

03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (340, 'hdd_used', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='5'"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (340, 'hdd_total', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='5'"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (339, 'hdd_used', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='3'"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (339, 'hdd_total', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='3'"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (338, 'hdd_used', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='2'"

[maw]

any suggestions of how to fix the mysql errors?

[BSOD2600]

Try clearing the poller cache.

[maw]

as in deleting everything in that table?

[maw]

Try clearing the poller cache.

Once making space available I tried this right away. I took a look in my /cacti/rra folder and noticed the 'date modified' for all files is about the same time mysql ran out of space.

[BSOD2600]

as in deleting everything in that table?

Yea. run:
mysql -u root -p cacti
truncate table poller_output;
quit

[maw]

oh snap... good work BSOD2600.

Things look happy now... any reason that table isnt cleaned when poller cache is rebuilt from UI?

Thanks for the help.

[gandalf]

Most often, increasing php.ini's memory size to at least 64M helps
Reinhard