Cacti no longer creating graphs

[UKS-Gavin]

Hi All,

A customer of ours has cacti setup and for some reason it is no longer creating the RRD's.

Running the debian stable version of cacti on Debian Sarge 3.1 with a custom kernel.

The system stopped graphing approximately a week ago. I've checked the file permissions and these appear fine, looking at the cacti log files i'm seeing no errors. I can see the poller running and capturing the SNMP data but it doesn't seem to update the RRA files.

To confirm that I've moved the RRD's to a temp directory, ran the poller by hand and it does not recreate the files.

Any pointers as to what to check would be greatly appreciated.

Regards,
Gavin Conway
UKSolutions.Net

[Linegod]

- clear the poller cache
- Increase the log level
- check the cacti log for errors
- check the system logs for errors (/var/log/messages, etc)

[gandalf]

Additionally,
- check file space (cacti.log file size, mysql, rrd files partition ...)
- check crontabs
- see my link on "NaN Debugging"
Reinhard

[UKS-Gavin]

Thanks for that, in the end this was the output_poller table in mysql.

Is there any disadvantage to putting a bi-weekly/montly script in place to truncate that table?

Cheers,
Gavin

[rony]

Is there any disadvantage to putting a bi-weekly/montly script in place to truncate that table?

You shouldn't....

The poller does this unless it exits abnormally. You should figure out why the poller isn't exiting correctly. Check out the cacti.log for more information.

[UKS-Gavin]

Is there any disadvantage to putting a bi-weekly/montly script in place to truncate that table?

You shouldn't....

The poller does this unless it exits abnormally. You should figure out why the poller isn't exiting correctly. Check out the cacti.log for more information.

I've looked into this, the system was running fine for nearly 2 months with no external influences and then started reporting NaN. I'll monitor it to see how it goes but if I do choose to put the truncate in will it result in any detrimental behaviour?

Thanks,
Gavin

[rony]

Yes, loss of data.

What you propose is just a half sticky bandaid to the real problem. You need to figure out what the real problem is before you just willy nilly stick a bandaid on it.

[UKS-Gavin]

Yes, loss of data

Could you give some suggestions as to what would cause the poller to crash unexpectedly?

Thanks,
Gavin

[rony]

1. The max memory and execution time not being set high enough in your php.ini.
2. The cacti.log file exceeding the file systems max file size (typically 2 gigs).
3. Database corruption.

Ultimatly you need to look at your log and see if there is anything that would explain why this is happening.

FYI, this issue with the poller_output table can creep up on you. I would suggest that you have crontab execute a script that executes the poller than after the poller exits, counts the number of rows in the poller_output table. If the number is greater than 0, copy the cacti.log and email someone to alert them to look into the issue.

[UKS-Gavin]

Thanks for that Tony, I'll get that put in place now. Much appreciated.