All of my graphs have stopped updating, it seems like RRD files get updated with values by the poller but the graphs stopped showing new data since yesterday.
I have checked and tried every single method I could find, that includes:
- Rebuilt poller cache via web and CLI.
- Stop, start poller
- Crontab and both poller timing values on web are all on 5 minutes
- RRD's are all owned by apache
- Graph debug mode shows normal stuff and OK at the end.
- rrdtool lastupdate localhost_cpu_7.rrd: "cpu 1396943102: 3" (this time is more than 12 hours after data stopped on the graphs).
- Changed spine threads from 5 to 10 and back to 5.
- Ran poller manually, everything seems fine except for weathermap complaining about the maps: "DataCenter had no valid data, according to WeatherMapDataSource_rrd"
- Boost had some log entries about not having permissions to it's cached images; Granting +x on the boost cache didn't help, disabling boost didn't help.
- Waited a whole night to try and let it fix itself.
- Realtime is working fine.
- Restarted every relevant service (mysql, apache etc)
- Rebooted the server.
- All devices are in "Up" status.
- Poller output table empty (CLI script)
- Repaired database (CLI script)
- 1.5GB free RAM
- 55GB free disk space
- CPU usage is mostly low
Prior to the failure, cron job was on 5 minutes, same for settings page, however poller interval on settings page was set to 1 minute.
This generated some errors that worried me, I saw best practice to have them all match so I changed it so they're all set to 5 minutes, added +5 spine threads to the already used 5 and rebuilt the poller cache.
Excuse my french but wtf am I missing/doing wrong?
Update: Rebuilt poller cache about 10 times more, set poller interval to 1 minute, disabled boost and graphs are getting data now.
I'd *really* like to know what is happening since my boss started saying stuff like "I'm not sure we can trust this software".
So please, if anyone can shed some light on what may have caused this or how exactly the relevant mechanisms work I'd be grateful as I have a similar problem on a different machine.