mysql mount ran out of space... all graphs NAN now

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Developers, Moderators

Post Reply
maw
Posts: 34
Joined: Wed Mar 07, 2007 2:26 am
Location: Vancouver, Canada

mysql mount ran out of space... all graphs NAN now

Post by maw »

Hi all,

My /var(mysql DB) mount ran out of available space causing cacti to fail for 2days straight. Since rectifying the space issue, the poller is no longer functioning correctly. I have rebuilt the poller cache twice but the 'cacti log' file isnt updating anymore and my graphs show NAN now.

The reason the mount ran out of space was due to the mysql option "log-bin" being enabled. I have since commented that line out of config file (/etc/my.cnf). Additionally, I zipped up my original apache log file. These are the only two changes I made on purpose other then the disk running out of space. I attached the bottom part of my cacti.log since it stopped updating. Please note the cronjob is still executing.

Done anyone know how to fix this?

Thanks,
maw
Attachments
pic1.JPG
pic1.JPG (103.38 KiB) Viewed 3383 times
SNAG-0295.jpg
SNAG-0295.jpg (56.93 KiB) Viewed 3387 times
SNAG-0294.jpg
SNAG-0294.jpg (47.28 KiB) Viewed 3387 times
cacti.log.txt
(3.3 KiB) Downloaded 186 times
:: FreeBSD6.2 - Apache 1.3.37+mod_ssl - PHP 5.2.1 - MySQL 5.0 ::
maw
Posts: 34
Joined: Wed Mar 07, 2007 2:26 am
Location: Vancouver, Canada

Post by maw »

I restarted my server and checked the cacti.log again. It has been updated with a barrage of SQL errors. eg:

03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (340, 'hdd_used', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='5'"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (340, 'hdd_total', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='5'"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (339, 'hdd_used', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='3'"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (339, 'hdd_total', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='3'"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Exec Failed "insert into poller_output (local_data_id, rrd_name, time, output) values (338, 'hdd_used', '2007-03-26 23:55:20', '0')"
03/26/2007 11:55:21 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "select field_value from host_snmp_cache where host_id=39 and field_name='hrStorageAllocationUnits' and snmp_index='2'"
:: FreeBSD6.2 - Apache 1.3.37+mod_ssl - PHP 5.2.1 - MySQL 5.0 ::
maw
Posts: 34
Joined: Wed Mar 07, 2007 2:26 am
Location: Vancouver, Canada

Post by maw »

any suggestions of how to fix the mysql errors?
:: FreeBSD6.2 - Apache 1.3.37+mod_ssl - PHP 5.2.1 - MySQL 5.0 ::
User avatar
BSOD2600
Cacti Moderator
Posts: 12171
Joined: Sat May 08, 2004 12:44 pm
Location: USA

Post by BSOD2600 »

Try clearing the poller cache.
maw
Posts: 34
Joined: Wed Mar 07, 2007 2:26 am
Location: Vancouver, Canada

Post by maw »

as in deleting everything in that table?
:: FreeBSD6.2 - Apache 1.3.37+mod_ssl - PHP 5.2.1 - MySQL 5.0 ::
maw
Posts: 34
Joined: Wed Mar 07, 2007 2:26 am
Location: Vancouver, Canada

Post by maw »

BSOD2600 wrote:Try clearing the poller cache.
Once making space available I tried this right away. I took a look in my /cacti/rra folder and noticed the 'date modified' for all files is about the same time mysql ran out of space.
:: FreeBSD6.2 - Apache 1.3.37+mod_ssl - PHP 5.2.1 - MySQL 5.0 ::
User avatar
BSOD2600
Cacti Moderator
Posts: 12171
Joined: Sat May 08, 2004 12:44 pm
Location: USA

Post by BSOD2600 »

maw wrote:as in deleting everything in that table?
Yea. run:
mysql -u root -p cacti
truncate table poller_output;
quit
maw
Posts: 34
Joined: Wed Mar 07, 2007 2:26 am
Location: Vancouver, Canada

Post by maw »

oh snap... good work BSOD2600.

Things look happy now... any reason that table isnt cleaned when poller cache is rebuilt from UI?

Thanks for the help.
:: FreeBSD6.2 - Apache 1.3.37+mod_ssl - PHP 5.2.1 - MySQL 5.0 ::
User avatar
gandalf
Developer
Posts: 22383
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Post by gandalf »

Most often, increasing php.ini's memory size to at least 64M helps
Reinhard
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest