won't create graphs, poller intermittent

[cactusesalltheway]

centos 6.8 Final/Cacti 0.8.7/php5
multicore, 16gb virtsh vm

Approximately 500 devices are being polled ok and their graphs work, but there are gaps.

We rebooted to add ram and now when I add a new graph using the php scripts, I see snmp traffic on the remote and it adds the devices on the front end, but no graph shows up in /rra and debut mode says 'no file or directory' with the placeholder image. Owner is cacti for all folders and the permissions are wide open on /rra and everything. rrdtool is running fine. Plenty of HD space. 3/16gb ram used.
If I add 2 hosts and one of them is offline, I get a snmp walk timeout for host for the one that is down, but the one that is up there is no error and the graph doesn't get created although it shows up in the front end.

I look at /var/log/cacti/cacti.log and there are only some errors from 5 days ago when we rebooted:
01/12/2017 02:17:00 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating

01/12/2017 02:16:59 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed!, Error:'2006', SQL:"select field_value from host_snmp_cache where host_id=720 and field_name='hrStorageAllocationUnits' and snmp_index='10'"

Then nothing else, no other logs since 1/12, which I think is odd of itself considering I've been repeatedly trying to a couple days now.

Then there is the poller.log which shows these two clusters of events over and over:
OK u:0.15 s:0.26 r:56.88
OK u:0.15 s:0.26 r:56.88
Waiting on 1 of 1 pollers.
Waiting on 1 of 1 pollers.

I think spine was supposed to be running, now it's not. Even the graphs that don't show up appear in the front end system utilities with full snmp cache and data sources.

The application is not telling me anything useful and I'm having trouble finding any documentation beyond 'turn on debug mode', please advise. I need to know what I have to do to bring it back up after reboot besides start httpd and I need tools to test what is working and what is not.

Thanks.

[cactusesalltheway]

Guess no one around here knows anything?

I would be happy just to find some documentation on the meaning of this message:

"Waiting on 1 of 1 pollers."

Can anyone interpret this? How can a single poller wait on itself?

How can this pretty normal error message not be in the documentation?

[cigamit]

Cacti 0.8.7

That is probably the largest issue why no one is commenting. That particular version was released on 23-Oct-2007. There have been any number of fixes since then, so we may be chasing a rabbit down a hole that has already been plugged.

The poller process itself is a host process that launches either spine or cmd.php processes to do the actual polling. That is why it is saying waiting on 1/1. It seems like you have script server processes timing out, and the poller is waiting on them to time out. Either figure out why they are timing out, or up the number of script servers so that additional scripts can be ran without waiting (the others will still timeout though)

[cactusesalltheway]

Thanks, that's helpful.

In the meantime I figured a bunch out by reading "The Witness"'s posts.

I understand we are not up to date and that makes sense, I don't expect you guys to bend over backwards.

I do have a short list of questions though, monitoring is working well for 4/5 of our hosts.

-At the moment I'm trying to get the snmp timeout setting to apply to all graphs so I don't have to go into 600 individual graphs and set it. Any ideas on that?

-Other question, do I understand correctly that if when adding a graph, if the first ping/snmp request fails, the graph will remain blank forever?

-I also have the 'Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 21, [then 20/21 rotating issues], manually clearing table and numerous attempts at tuning threads/procs/scripts hasn't resolved.

-I have a bunch of graphs that showed up that have 'host_id' as their host_id, they cause a bunch of errors. I can just delete these manually from mysql right?

-in poller.log, it resets every minute after each run, and i only see 1/5 of our hosts in there mentioned specifically, why does it sometimes just say 'ok u:0.18...' and other times 'rrdtool update host.com'. Why don't all hosts get listed in debug, or why are some listed sometimes and other times not?

-Also when adding new graphs sometimes the filename of the rrd is different from what shows up in the graph's data source, any idea what could cause that? Would the re-apply suggested names to graphs work to fix that?

-In the meantime I read that I had to have apache set as group, so I tried that, not only did it not fix it, but it didn't even create an rrd in rra, and when I set it back to cacti group, it remained the same. So this actually made it worse and I can't figure out why it can't be reversed. I set perms to 777 for the entire rra directory just to make sure that wasn't the issue. The host is up and even when the rra wasn't created, it showed in the devices view the snmp response. But debug says the file isn't there, and it sure isn't.

Are any of these diverse issues supposed to be resolved by updating to 8.8 or changing rrd tool version from 1.3? (this is why I hate inheriting systems from other people, they do a half ass job and you are left in a haunted house)

That's it for now, I might be back with more. Thanks for taking a look if you got this far.

[cactusesalltheway]

I'm also getting about a hundred of these errors a second:

Jan 25 13:39:22 mon04 Cacti[31830]: CMDPHP: ERROR: SQL Cell Failed!, Error:'1064', SQL:"select field_value from host_snmp_cache where host_id=|host_id| and field_name='hrStorageAllocationUnits' and snmp_index='1'"


Question: how is cmd.php throwing this error if we're using spine?

We also have a ton of graphs that show up with the name host_description, but there is no host with that name in any of the lists, and no graphs with that name in /rra

Or a bigger question, is this 8.7 install just too far gone? Am I better off starting from scratch than cleaning out the stables?