Gaps in all my graphs

[pompeychimes]

n00b here...

As of a few days ago all of my graphs are have gaps. The gaps are the same on all my graphs and occur every 10 - 20 mins. I see the below error in the log...

08/06/2012 11:05:01 AM - POLLER: Poller[0] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 600 seconds have passed since the last poll!

Yesterday, the tech support page compelled me to upgrade the "memory_limit" to 180M and I did. This made no difference.

The above error makes me think the poller isn't finishing in time before the scheduled tasks tries to start it again.

Tech support attached.

Thanks,

James

Operating System: W2K3 SP1
Webserver: IIS 6
Cacti: 0.8.8a
Spine:
MySQL:
PHP: 0.6.2
RRDTool (Cygwin or Win32 build):
Net-SNMP: 5.6.1.1
Cygwin (cygwin1.dll version):
Plugin Architecture:

[BSOD2600]

Read through the debugging guide linked in http://forums.cacti.net/viewtopic.php?f=3&t=11747 ?

When the poller takes longer, is anything else on the server consuming all the cpu/resources?

nothing was attached...

[pompeychimes]

The forum wouldn't let me upload a html file.

Unless you're referring to "HOWTO: Debug Windows NTFS permission problems." I'm not seeing the debugging guide you mentioned.

Nothing's jumping out at me in regards to CPU resources. There are momentary spike but nothing excessive.

[pompeychimes]

At 1:40 I get this in the log and nothing else...

08/07/2012 01:40:32 PM - CMDPHP: Poller[0] Host[63] DS[5119] WARNING: Result from SERVER not valid. Partial Result: U
08/07/2012 01:40:32 PM - CMDPHP: Poller[0] Host[62] DS[5116] WARNING: Result from SERVER not valid. Partial Result: U
08/07/2012 01:40:31 PM - CMDPHP: Poller[0] Host[61] DS[5113] WARNING: Result from SERVER not valid. Partial Result: U
08/07/2012 01:40:31 PM - CMDPHP: Poller[0] Host[60] DS[5110] WARNING: Result from SERVER not valid. Partial Result: U
08/07/2012 01:40:30 PM - CMDPHP: Poller[0] Host[57] DS[5102] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:30 PM - CMDPHP: Poller[0] Host[57] DS[5102] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:30 PM - CMDPHP: Poller[0] Host[57] DS[5103] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:30 PM - CMDPHP: Poller[0] Host[57] DS[5103] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:29 PM - CMDPHP: Poller[0] Host[56] DS[5068] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:29 PM - CMDPHP: Poller[0] Host[56] DS[5068] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:29 PM - CMDPHP: Poller[0] Host[56] DS[5067] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:29 PM - CMDPHP: Poller[0] Host[56] DS[5067] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:29 PM - CMDPHP: Poller[0] Host[55] DS[5032] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:29 PM - CMDPHP: Poller[0] Host[55] DS[5032] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:29 PM - CMDPHP: Poller[0] Host[55] DS[5033] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:29 PM - CMDPHP: Poller[0] Host[55] DS[5033] WARNING: Result from SNMP not valid. Partial Result: U
08/07/2012 01:40:28 PM - CMDPHP: Poller[0] Host[53] DS[4996] WARNING: Result from SERVER not valid. Partial Result: 08/07/2012 01:40:22
08/07/2012 01:40:22 PM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
08/07/2012 01:40:01 PM - SYSTEM STATS: Time:298.6532 Method:cmd.php Processes:4 Threads:N/A Hosts:64 HostsPerProcess:16 DataSources:13371 RRDsProcessed:4882
08/07/2012 01:40:01 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.

At 1:45 I get this followed by what appears to be the normal stream of log entries once the poller starts...

08/07/2012 01:45:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 952, Data Sources: traffic_in(DS[4929]), traffic_out(DS[4929]), traffic_in(DS[4930]), traffic_out(DS[4930]), traffic_in(DS[4931]), traffic_out(DS[4931]), traffic_in(DS[4932]), traffic_out(DS[4932]), traffic_in(DS[4933]), traffic_out(DS[4933]), traffic_in(DS[4934]), traffic_out(DS[4934]), traffic_in(DS[4935]), traffic_out(DS[4935]), traffic_in(DS[4936]), traffic_out(DS[4936]), traffic_in(DS[4937]), traffic_out(DS[4937]), traffic_in(DS[4938]), traffic_out(DS[4938]), Additional Issues Remain. Only showing first 20

[BSOD2600]

All of those warnings from the hosts... are there any common trends? same scripts/templates? those DS[] fields are links to each data source...

Sounds like you've got a rouge script timing out which is making the poller lapse.

[pompeychimes]

With my limited experience the only commonality I can see is that all the hosts are high end switches (6500, Nexus, etc...).

Last night I switched over to the spine poller and the gaps went away. I have since switched back and the gaps came back.

How do i identify the rogue script?

[BSOD2600]

From the output above, what script is DS[5119], DS[5116], DS[5113], and DS[5110] using? Sure your devices aren't falsely interpreting the wave of snmp requests as an attack?

[pompeychimes]

If i'm looking at the right thing it appears all of these are using this script ss_fping.php

If I run the script manually from the command line I get this...

C:\Inetpub\wwwroot\cacti\scripts>php ss_fping.php 172.31.80.20 5
min:56.0000 avg:57.6000 max:60.0000 dev:1.6248 loss:0.0000
C:\Inetpub\wwwroot\cacti\scripts>

A handful of the devices are smart enough to repel attacks but not the majority of them.

[BSOD2600]

Lets backup. Before a few days ago, were all of these scripts running normally and completing in time? If so, what changed on the Cacti OS? Rebooted the server yet?

For a single polling cycle, change the cacti logging level to debug and look through it when various scripts/queries fail. An common reason?

[pompeychimes]

Thanks for your help thus far. We've decided to deploy a new Cacti Server. At present it shares a server with other apps and processes. We want to get it on its own box so as to eliminate any other variables now and in the future.