[fixed] Indexed values not being graphed from remote poller

[aamorris]

I have found a strange issue when creating graphs for indexed values like interfaces and disks on a remote poller.

When I add new indexed data sources via a remote poller, the data is not inserted into the RRD databases (and subsequently graphed) until I remove and re-add the graphs a second time. This only happens on the remote poller and indexed values. Non-indexed data sources (CPU, Memory, uptime, etc) work fine on the first add.

I have to add the graphs, remove the graphs (I also remove the unused data sources), and re-add the graphs.

I know the poller can query the data because viewing the data in real-time works fine.

Cacti 1.1.38
Spine 1.1.38
Ubuntu 16.04.5 LTS
mysql-server-5.7 5.7.22-0ubuntu0.16.04.1
php7.0 7.0.30-0ubuntu0.16.04.1

[aamorris]

***EDIT***
Nevermind... I am still seeing the issue.
***EDIT***

I believe I corrected the issue. It appears the root cause was due to polling taking too long. My environment is on the larger side, the primary poller has ~300 hosts and 41,000 data sources.

There were no errors related to polling timeout, but I was averaging 240s. Well under 300s, but I guess too close to the edge.

I increased the number of script servers and it significantly improved the polling time. Down to ~80s.

We had already performed a number of performance optimizations in the past, but we never had reason to touch the script servers.

I think this become an issue recently due to a growing number of bare metal hosts where we are graphing the individual CPUs and the hosts have 40+ cores. These data sources are processed via a PHP script, not directly via SNMP.