Cacti server performance

[MrBr]

I'm receiving "There are '15' detected as overrunning a polling process, please investigate" in Poller log file
This error message repeated every poller cycle
I've investigated

All of the mentioned data sources are responsive and provide the graphs without any problems.
Running the Data Input Method commands from CLI under cactiuser also did not revealed any problems.

I have to note that, most of my monitored hosts are scattered across the globe and have high response time, ~400-800 ms rtd
The server itself is ~30% loaded

Is there any additional checks that I should perform?
Is Spine the only option ?
How many hosts/rrds single server can handle?

General info:
Current server hosts
Cacti Version 0.8.7i
~300 hosts
~5000 DSs
Poller and Cron 1 min
cmd.php poller
RHEL Linux based

[gandalf]

Spine will speed up net polling performance, but will not be able to reduce rtt
Running spine in verbosity=3 will print timings for each host; so you will know which one is slow.
You may want to tweak timouts for those hosts, then. But when NOT running snmpbulkwalk, each SNMP request will last the rtt to that very host. This may lead to exceeding poller interval in case there are many slow responses
R.

[MrBr]

I've started transferring Cacti to Spine.
However I've encountered a "U" query result problem.
Most of my requests are based on custom scripts/snmp requests.
With cmd.php I'm successfully polling the results.
With Spine - I'm receiving "U".
Spine with debug options did not revealed any information, the data been retrieved but not processed. I'm not using PHP in any of my requests. Can you please point me to the right direction in order to solve this issue.

[gandalf]

Well, when using script (queries), snmpbulkwalk will not come in and thus won't save SNMP packet overhead. Spine will usually perform much faster, in case CPU usage is key. But in case rtt is key, spine may not provide that much of a benefit.
Concerning the "U" issue, we had differences e.g. in handling numeric data followed by strings (e.g. 25 Celsius)
R.

[MrBr]

To clarify the "U" issue.
If I'm receiving the following result from custom SNMP request. "sourceA:52 sourceB:56"
cmd.php will handle this sources correctly, spine - won't.
Is this right?
If yes, how the spine can handle multiple results in single SNMP request?

[gandalf]

This is not a known spine issue. In fact, those types of data is running fine in our installation. Which version of spine are you running? Did you compile spine on your own?
R.

[MrBr]

I'm using cacti-spine-0.8.7i and compiled from the sources downloaded from cacti.net
I'm using centos5 as my os, i did not found precompiled sipe binary.

[gandalf]

Sounds good.
One issue: Did you append the second parameter of the input method after a first graph has been created?
To cut it short: please rerun rebuild_poller_cache.php from cli directory for alll hosts
R.

[MrBr]

As I mentioned earlier, I'm migrating an already existing Cacti server from cmd.php to spine.
I've created a complete replica of the production server and changed the poller on the new one (test server). The graphs, data sources, rrds and everything else are exactly the same as on the original server, no changes been made. rebuild_poller_cache.php been executed after the poller change.
I'll rerun it once again tomorrow, but I don't think it'll help

[gandalf]

... rebuild_poller_cache.php been executed after the poller change.
I'll rerun it once again tomorrow, but I don't think it'll help

Well, I suppose you're right, then. This was an issue I was running into some time ago. To be honest, I'm out of ideas now. I need access to your installation or a mysqldump along with the script being used to be able to reproduce
R.

[MrBr]

Unfortunately i cannot provide neither the access nor the full dump, commercial issues.

However, I can shrink down the deployment to single test host. Then I can send the dump to you.
The other problem is that I'm using custom SNMP OID, that provides custom response. How can we overcome this?

P.S. I'll try to reproduce the problem with local script that provides the same output

P.P.S. Executing "php rebuild_poller_cache.php" did not helped, the problem still there