No graph data since /var partition filled

[william]

Hi,

OS is FreeBSD 6.1-RELEASE.

Over night the /var parition filled up, this morning I made space, unmounted, fsck -y, mounted and then rebooted the server.

It seems all my graphs are now not working, in the cacti.log I'm getting ALOT of:

s (2831, 'cisco_memfree', '2008-04-15 10:05:37', '152931668')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2832, 'cisco_memused', '2008-04-15 10:05:37', '18815372')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2833, 'traffic_in', '2008-04-15 10:05:37', '3220684261')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2833, 'traffic_out', '2008-04-15 10:05:37', '2394334890')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2834, 'traffic_in', '2008-04-15 10:05:37', '1377339251')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2834, 'traffic_out', '2008-04-15 10:05:37', '69746617')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2835, 'traffic_in', '2008-04-15 10:05:37', '0')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2835, 'traffic_out', '2008-04-15 10:05:37', '0')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2836, 'traffic_in', '2008-04-15 10:05:37', '908658951')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2836, 'traffic_out', '2008-04-15 10:05:37', '12476422')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2837, '5min_cpu', '2008-04-15 10:05:37', '12')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2838, 'traffic_in', '2008-04-15 10:05:37', '1569018647')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2838, 'traffic_out', '2008-04-15 10:05:37', '698559562')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2839, 'traffic_in', '2008-04-15 10:05:37', '3888500664')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2839, 'traffic_out', '2008-04-15 10:05:37', '1370857620')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2840, 'traffic_in', '2008-04-15 10:05:37', '26859552')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2840, 'traffic_out', '2008-04-15 10:05:37', '3942735725')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2841, 'traffic_in', '2008-04-15 10:05:37', '2414358441')'
04/15/2008 10:05:37 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (2841, 'traffic_out', '2008-04-15 10:05:37', '3231346671')'
04/15/2008 10:05:38 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'145', SQL:"select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) "


Can someone please assist me in resolving my issue?

Cheers.

[rony]

Run a repair on your Cacti database and all the tables in mysql.

Also, I would suggest that you use the System -> System Utilities -> Rebuild Poller Cache and run the following query:

Code: Select all

truncate poller_output;

Do these activities when the poller is NOT running.

[william]

How do I run a repair on the cacti mysql db?

[william]

google says:

mysqlcheck --auto-repair --databases cacti

going to enable the poller and see what happens

[william]

rebuilt the cache and ran the query, now everything is updating Thanks Tony!!!

[perjaka]

Run a repair on your Cacti database and all the tables in mysql.

Also, I would suggest that you use the System -> System Utilities -> Rebuild Poller Cache and run the following query:

Code: Select all

truncate poller_output;

Do these activities when the poller is NOT running.

These steps also fixed out problem. We had a server lock up and an admin rebooted the box by unplugging it. After it came up 10 minutes later we had the same issue as described in this thread.

Basically we enabled logging and found it riddled with this: "08/27/2009 10:30:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'145', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (....<bunch of values>')'"

Such a simple measure repaired months and over 1000 potentially lost graphs!

This thread = life saver.