Graphs not getting populated but real time graphs works

[plasky12]

The permissions to the rra folder is okay. No changes have been done for quite sometime.
Strange part is the real time graphs are giving proper outputs but the graph isnt getting populated?
Maybe add the graphs again? Or some issue with the polling for the device?. There seems to be no snmp timeouts as well.
Is there anyway i can disable polling only for one device? And then enable it after 20 mins to see if the polling is giving correct data?

[plasky12]

The permissions to the rra folder is okay. No changes have been done for quite sometime.
Strange part is the real time graphs are giving proper outputs but the graph isnt getting populated?
Maybe add the graphs again? Or some issue with the polling for the device?. There seems to be no snmp timeouts as well.
Is there anyway i can disable polling only for one device? And then enable it after 20 mins to see if the polling is giving correct data?

[plasky12]

The permissions to the rra folder is okay. No changes have been done for quite sometime.
Strange part is the real time graphs are giving proper outputs but the graph isnt getting populated?
Maybe add the graphs again? Or some issue with the polling for the device?. There seems to be no snmp timeouts as well.
Is there anyway i can disable polling only for one device? And then enable it after 20 mins to see if the polling is giving correct data?

[TheWitness]

Did I hear an echo

[TheWitness]

Well, you can first put a single device into debug to see what's coming back from the poller. There is also a an open bug re inserts. Is your ooller_output table memory?

[plasky12]

poller output table all look ok.
I put the device on debug mode but i dont find anything in the logs.
made it run 2 poller cycles, but nothing. I am really confused and stuck right now . Not sure what is going wrong.

[plasky12]

Hello anyone?
Can i delete the existing rrd file and the graphs associated with that device?
And then add the graphs again so that i can check if new rrd files are created and data is getting updated?
Any suggestions are most welcome.
Thanks

[TheWitness]

Of course, and once the new file is created, you can stitch/merge old and new with the CLI splice_rrd.php.

[plasky12]

Hi,

So i tried deleting the graphs and associated rrd files and recreated them. But still seems to be no data being plotted.
In fact i get this error when any new graph is added that -'' failed to open data file poller may not have run yet".
Also i see this in the logs when the graphs are trying to be plotted

2022-06-19 00:25:07 - CMDPHP PHP ERROR NOTICE Backtrace: (/graph_json.php[222]:rrdtool_function_graph(), /lib/rrd.php[1589]:variable_nth_percentile(), /lib/graph_variables.php[473]:nth_percentile(), /lib/graph_variables.php[33]:rrdtool_function_stats(), /lib/graph_variables.php[125]:CactiErrorHandler())
2022-06-19 00:25:07 - ERROR PHP NOTICE: Undefined offset: 27189 in file: /var/www/html/cacti-1.2.18/lib/graph_variables.php on line: 125
2022-06-19 00:25:07 - CMDPHP PHP ERROR NOTICE Backtrace: (/graph_json.php[222]:rrdtool_function_graph(), /lib/rrd.php[1589]:variable_nth_percentile(), /lib/graph_variables.php[473]:nth_percentile(), /lib/graph_variables.php[33]:rrdtool_function_stats(), /lib/graph_variables.php[100]:CactiErrorHandler())
2022-06-19 00:25:07 - ERROR PHP NOTICE: Undefined offset: 27189 in file: /var/www/html/cacti-1.2.18/lib/graph_variables.php on line: 100
2022-06-19 00:25:07 - CMDPHP PHP ERROR NOTICE Backtrace: (/graph_json.php[222]:rrdtool_function_graph(), /lib/rrd.php[1589]:variable_nth_percentile(), /lib/graph_variables.php[473]:nth_percentile(), /lib/graph_variables.php[33]:rrdtool_function_stats(), /lib/graph_variables.php[125]:CactiErrorHandler())
2022-06-19 00:25:07 - ERROR PHP NOTICE: Undefined offset: 27189 in file: /var/www/html/cacti-1.2.18/lib/graph_variables.php on line: 125
2022-06-19 00:25:07 - CMDPHP PHP ERROR NOTICE Backtrace: (/graph_json.php[222]:rrdtool_function_graph(), /lib/rrd.php[1589]:variable_nth_percentile(), /lib/graph_variables.php[473]:nth_percentile(), /lib/graph_variables.php[33]:rrdtool_function_stats(), /lib/graph_variables.php[100]:CactiErrorHandler())
2022-06-19 00:25:07 - ERROR PHP NOTICE: Undefined offset: 27189 in file: /var/www/html/cacti-1.2.18/lib/graph_variables.php on line: 100
2022-06-19 00:25:07 - CMDPHP PHP ERROR NOTICE Backtrace: (/graph_json.php[222]:rrdtool_function_graph(), /lib/rrd.php[1589]:variable_nth_percentile(), /lib/graph_variables.php[473]:nth_percentile(), /lib/graph_variables.php[33]:rrdtool_function_stats(), /lib/graph_variables.php[125]:CactiErrorHandler())
2022-06-19 00:25:07 - ERROR PHP NOTICE: Undefined offset: 27189 in file: /var/www/html/cacti-1.2.18/lib/graph_variables.php on line: 125
2022-06-19 00:25:07 - CMDPHP PHP ERROR NOTICE Backtrace: (/graph_json.php[222]:rrdtool_function_graph(), /lib/rrd.php[1589]:variable_nth_percentile(), /lib/graph_variables.php[473]:nth_percentile(), /lib/graph_variables.php[33]:rrdtool_function_stats(), /lib/graph_variables.php[100]:CactiErrorHandler())
2022-06-19 00:25:07 - ERROR PHP NOTICE: Undefined offset: 27189 in file: /var/www/html/cacti-1.2.18/lib/graph_variables.php on line: 100

Could you suggest?

[TheWitness]

Move to either 1.2.21 or the 1.2.x branch and try again.

[plasky12]

Hi,

thanks for the suggestions. I have managed to resolve it by deleting the old rrd files and graphs and adding the graphs again. The rrd files are now getting updated and graphs are getting plotted as usual.
Just one final question. I am using cacti version 1.2.18. What is the maximum number of devices i can monitor keeping in mind the performance? Right now we monitor close to 5000 routers . Is there a limit?
And Also how do i find the average pooling performance of all the devices being monitored in ? basically the average time calculated?

[TheWitness]

I'm monitoring quite a few, and I know of several companies in the 20-30k range. I would not be running anything post 1.2.15 right now unless it's 1.2.21+. There were lot's of issues injected between those releases as we tried to stabilize template editing and it's impact on Data Sources as well as some performance related changes. The 1.2.21 release is much better, though there were some minor bugs introduced with "Remember me" functionality and Web Basic authentication the first of which is resolved, and the second we are waiting on a pull request for. If you stay at 1.2.18, just remember, you have been warned. Review the CHANGELOG for 1.2.19++.

[plasky12]

ok thanks for the warning . I will keep this mind.
The performance right now for us seems okay with around 5000 routers being monitored. Just wanted to if there is any threshold limit where the performance would affected? Any approximate rough number?
Also i wanted to check the average pooling stats and time.
Is this one? Should i take 20.32 as the average pooling value for all the devices?

2022-06-21 15:03:31 - SYSTEM BOOST STATS: Time:20.32 RRDUpdates:75021
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.43 ProcessNumber:7 RRDUpdates:6328
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.42 ProcessNumber:2 RRDUpdates:5530
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.40 ProcessNumber:4 RRDUpdates:4497
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.30 ProcessNumber:10 RRDUpdates:6227
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.27 ProcessNumber:13 RRDUpdates:3458
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.40 ProcessNumber:1 RRDUpdates:4855
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.25 ProcessNumber:15 RRDUpdates:1773
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.33 ProcessNumber:6 RRDUpdates:6356
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.31 ProcessNumber:3 RRDUpdates:3620
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.18 ProcessNumber:9 RRDUpdates:5778
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.22 ProcessNumber:8 RRDUpdates:6344
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.15 ProcessNumber:11 RRDUpdates:3553
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.04 ProcessNumber:16 RRDUpdates:6180
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:17.98 ProcessNumber:14 RRDUpdates:3322
2022-06-21 15:03:30 - SYSTEM BOOST STATS: Time:18.07 ProcessNumber:5 RRDUpdates:6356
2022-06-21 15:03:29 - SYSTEM BOOST STATS: Time:17.71 ProcessNumber:12 RRDUpdates:844
2022-06-21 14:59:30 - SYSTEM BOOST STATS: Time:20.68 RRDUpdates:52946

[TheWitness]

With that few Data Sources, you could move updates to once every 4 or 6 hours. So, nothing to panic about. My production install has 1.6M Data Sources and boost runs about once an hour with more than 20M updates per hour. So, you have a lot of headroom. With 1 Minute polling, the most important thing to consider outside of boost, is the response time of certain OID's or if you are running scripts, those scripts. In my case, I have a service the polls the data separate from spine, and then spine just samples from the database. This same process is used for the WMI plugin. It gathers data from WMI/CIM devices out of band, and the the data collector just grabs the data from the database.