Graphs quit updating -- RESOLVED

gshackel · Post by **gshackel** » Thu Sep 21, 2006 5:37 pm

Cacti has been running fine for months. Earlier we had a DoS attack that generated lots of traffic on the LAN and Cacti stopped graphing data. The poller is running and debugging graphs from the GUI produces a result of OK. Not sure where to go from here? Can anyone provide suggestions on what to check next?
OS FreeBSD 4.11
Cacti Version 0.8.6h

Thanks,
Grant Shackelford
Network Engineer

jasyn · Post by **jasyn** » Thu Sep 21, 2006 9:22 pm

might be a dumb question .. but have you restarted apache?

cigamit · Post by **cigamit** » Thu Sep 21, 2006 10:02 pm

Have you tried changing the logging level to DEBUG and then parse through the results? What results are being returned to Cacti via the poller?

Are the graphs still showing the old data, just not adding the new? Are the RRD files modified times being updated? Have you checked permissions?

A little information goes a long way, but a lot of information can get the problem fixed a lot faster.

gshackel · Post by **gshackel** » Fri Sep 22, 2006 8:37 am

To answer some questions posted.

I checked all of the processes and even tried to reboot the system and it did not help.

Apache is running. Graphs display old data, but do not update new data.

To clarify - the timeline on the grpahs gets updated, just no new data.

RRD file modification times do not change. They are stuck at last update time of about noon EDT yesterday.

Also, the mysql database seems to be okay and has new data added.

This happened once before several months ago and a former colleague got information from someone in these forums on how to clear out some data from the database that was apparently causing a problem. Very similar symptoms now.

I did turn on debugging and look through the logs. Nothing stood out to me. I will paste some of the output below.

bash-2.05b# tail cacti.log
09/22/2006 09:31:20 AM - CMDPHP: Poller[0] Host[78] DS[4322] SNMP: v2: sdf-core-csw2.mgmt.hosting.com, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.39, output: 332505142
09/22/2006 09:31:20 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (4322,'traffic_out','2006-09-22 09:30:03','332505142')"
09/22/2006 09:31:20 AM - CMDPHP: Poller[0] Host[78] DS[4325] SNMP: v2: sdf-core-csw2.mgmt.hosting.com, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.125, output: 21099164
09/22/2006 09:31:20 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (4325,'traffic_in','2006-09-22 09:30:03','21099164')"
09/22/2006 09:31:20 AM - CMDPHP: Poller[0] Host[78] DS[4325] SNMP: v2: sdf-core-csw2.mgmt.hosting.com, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.125, output: 320462335
09/22/2006 09:31:20 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (4325,'traffic_out','2006-09-22 09:30:03','320462335')"
09/22/2006 09:31:22 AM - CMDPHP: Poller[0] Host[80] SNMP: Host did not respond to SNMP
09/22/2006 09:31:22 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "update host set status = '1', status_event_count = '72412', status_fail_date = '2006-01-13 02:10:00', status_rec_date = '2005-04-25 04:15:00', status_last_error = 'Host did not respond to SNMP', min_time = '0.71716', max_time = '999.99999', cur_time = '2.25210', avg_time = '7.95633', total_polls = '160152', failed_polls = '73356', availability = '54.196013786902' where hostname = 'staging.ns.hosting.com'"
09/22/2006 09:31:22 AM - CMDPHP: Poller[0] Time: 79.0530 s, Theads: N/A, Hosts: 15
09/22/2006 09:31:22 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_time (poller_id, start_time, end_time) values (0, NOW(), NOW())"
bash-2.05b#

Thanks to all who replied for your help,
Grant Shackelford

gshackel · Post by **gshackel** » Fri Sep 22, 2006 8:41 am

One more item:

Yes, I did check the permissions for the directory where the rrd files are stored. The directory is owned by the cacti user and is mode 777 now (was 755 previously) just to make sure there are no permission issues.

All of the files in the directory have the same ownership and permissions. As far as I can tell, no midifications were made to the system when it stopped working.

Thanks,
Grant Shackelford

Post by **gandalf** » Fri Sep 22, 2006 8:52 am

Please try my link on "NaN Debugging". If it does not help, please respond to let me extend it.
Reinhard

gshackel · Post by **gshackel** » Fri Sep 22, 2006 9:17 am

Looked at the NaN debugging doc. Great resource. Sorry I didn't find this before. Everything looks good until I get to step 3 and there are no occurences of the string "rrdtool" in the log files. Looks like the poller is working fine as far as retrievig data. It also appears that the database is being updated. For some reason, rrdtool does not seem to be called to update the rrd files.

Thanks,
Grant Shackelford

Post by **gandalf** » Sat Sep 23, 2006 3:34 am

Will update my howto in insert a new step about "Updating MySQL table poller_output with results from SNMP/Script". Just walked into that yesterday. But I'm quite sure, that this one will be a bit complicated to verify (needs some database checks, not that familiar to many users). Retry my howto tomorrow and let me know
Reinhard

gshackel · Post by **gshackel** » Sat Sep 23, 2006 8:58 pm

FYI - In the interim, I installed Cacti on a new system and imported the database from the old system. The new system had the same problem. I dumped the old database and created a new database from teh cacti template and the new system polls and updates graphs just fine. I think there may be some corrupt data in the old database that needs to be removed to get the system back in working order.

Thanks for the feedback,
Grant Shackelford

Post by **rony** » Sat Sep 23, 2006 11:39 pm

Export and email me your database, I will take a look at it.

bdbrumm · Post by **bdbrumm** » Mon Oct 02, 2006 3:03 pm

Per this thread: http://forums.cacti.net/viewtopic.php?t=16379
I had a very similar issue.

Perhaps you need to check out your poller_output table and
force a truncate?

gshackel · Post by **gshackel** » Tue Oct 03, 2006 12:35 pm

Thanks for the information. I followed the steps in your post and resolved the issue. Clearing the poller_output table seems to have done the trick. I also changed the php.ini settings for memory to 64M (they were still set to 8M). RRD files are being updated and graphs are being updated again. Thanks to everyone for your suggestions.

Grant Shackelford

Post by **rony** » Tue Oct 03, 2006 12:37 pm

Lol, guess I don't have to look at your database?

gshackel · Post by **gshackel** » Tue Oct 03, 2006 12:46 pm

Tony,
Thanks very much for your assistance, but there is no need to look at the database now. I appreciate the help!

Thanks,
Grant Shackelford

carock · Post by **carock** » Thu Oct 05, 2006 1:55 pm

I had a similar problem this morning.

It appears the cacti log file grew to 2.1 Gig in size and was no longer able to be written to. For some reason because the log file was not being written to, the RRA's were also not being updated. RRA's not updating caused my graphs to display no activity since the error.

I moved the log file and cacti created a new one. Now the RRA's are getting updated.

Chuck

Cacti

Graphs quit updating -- RESOLVED

Graphs quit updating -- RESOLVED

Status Update

Issue Resolved

Who is online