Graphs stop at midnight

[PriSupport]

Hello there,
I have a fresh install and was working fine, no major issues on performance but all of a sudden my graphs stop being generated at midnight.
Here is the support info:

General Information
Date Wed, 08 Aug 2012 13:11:28 -0700
Cacti Version 0.8.8a
Cacti OS win32
SNMP Version NET-SNMP version: 5.6.1.1
RRDTool Version RRDTool 1.2.x
Hosts 7
Graphs 54
Data Sources SNMP: 7
SNMP Query: 47
Total: 54
Poller Information
Interval 300
Type cmd.php
Items Action[0]: 101
Total: 101
Concurrent Processes 10
Max Threads 1
PHP Servers 1
Script Timeout 25
Max OID 10
Last Run Statistics Time:4.7070 Method:cmd.php Processes:10 Threads:N/A Hosts:8 HostsPerProcess:1 DataSources:101 RRDsProcessed:54
PHP Information
PHP Version 5.3.15
PHP OS WINNT
PHP uname Windows NT SUPPORT 5.1 build 2600 (Windows XP Professional Service Pack 3) i586
PHP SNMP Installed
max_execution_time 30
memory_limit 128M

When I run poller.php manually my log shows SNMP timeouts:

08/08/2012 01:06:34 PM - RECACHE STATS: RecacheTime:1.5404 HostsRecached:1
08/08/2012 01:06:32 PM - PCOMMAND: Poller[0] Host[6] WARNING: Recache Event Detected for Host
08/08/2012 01:06:31 PM - SYSTEM STATS: Time:5.3131 Method:cmd.php Processes:10 Threads:N/A Hosts:8 HostsPerProcess:1 DataSources:101 RRDsProcessed:54
08/08/2012 01:06:31 PM - CMDPHP: Poller[0] Host[9] DS[78] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:31 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.5.1', and OID:''
08/08/2012 01:06:31 PM - CMDPHP: Poller[0] Host[8] DS[74] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:31 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.4.2', and OID:''
08/08/2012 01:06:30 PM - CMDPHP: Poller[0] Host[7] DS[68] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:30 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.1.1', and OID:''
08/08/2012 01:06:30 PM - CMDPHP: Poller[0] Host[10] DS[87] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:30 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.13.1', and OID:'.1.3.6.1.2.1.2.2.1.16.6'
08/08/2012 01:06:30 PM - CMDPHP: Poller[0] Host[10] DS[87] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:30 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.13.1', and OID:'.1.3.6.1.2.1.2.2.1.10.6'
08/08/2012 01:06:30 PM - CMDPHP: Poller[0] Host[10] DS[82] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:30 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.13.1', and OID:''
08/08/2012 01:06:28 PM - CMDPHP: Poller[0] Host[6] DS[62] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:28 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.10.1', and OID:''
08/08/2012 01:06:28 PM - CMDPHP: Poller[0] ASSERT: '835878724<5853062' failed. Recaching host '192.168.10.1', data query #1
08/08/2012 01:06:27 PM - CMDPHP: Poller[0] Host[4] DS[89] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:27 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.1.2', and OID:''
08/08/2012 01:06:27 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SNMP not valid. Partial Result: U
08/08/2012 01:06:27 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.1.2', and OID:''

immediately followed by:

POLLER: Poller[0] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 360025 seconds have passed since the last poll

Poller is set to 5min while Maximum Concurrent Poller Processes = 10
Not sure what else to tweak here with the respect to scheduling, it was based on docs.

Thanks again.

[pompeychimes]

For my information can you tell me who you run the poller manually?

[PriSupport]

I cleared the cacti log to get a clean start then CMD to

Code: Select all

C:\PHP\

and ran

Code: Select all

php.exe c:\www\cacti\poller.php

granted, adjust the path to your own 'poller.php'
After I run the poller manually graphs restart but the error code about timeout and Poller Interval do not go away and then graphing stops at midnight. I know it is not the device(s), as I have MRTG running just fine on the same box pulling the same devices.

Seems like we have similar issues...

[phalek]

So can you paste a screenshot of the scheduled task you created for the poller ?

[PriSupport]

Here they are.

cacti_task.jpg (24.53 KiB) Viewed 3455 times

cacti_schedule.jpg (43.52 KiB) Viewed 3455 times

Also, I noticed that when trying to edit the task I am getting an error about rights issue?! Oddly to remedy this I had to grant Administrators group access to "C:\WINDOWS\Tasks" folder:

Code: Select all

cmd> C:\windows> CACLS TASKS /E /G builtin\administrators:F

Not sure if this matters since cacti should not be 'editing' the task itself...

Thanks for the support.

[phalek]

hm, i would actually set the start time to 00:00 AM. did you notice if the graphs do start graphing again automatically after 11:41 AM ?

[PriSupport]

You mean 12:00AM since 00:00 is not valid.
Not sure about 11:41AM but that time stamp was just the last edit/creation time of the task not really by any plan. The graphs start when I run the command. Today, (after I ran the same command yesterday) they are being created past midnight for some reason but like I said this is a fairly fresh install and I have had the issue come up 2 times in last say 3 days or so. Could it be the scheduler rights are getting reset somehow and affecting executing of the task?
Strange...

[BSOD2600]

Midnight is too much of a coincidence for things stop working at 12AM.

the server in a domain and some maintenance scripts were executed?
as all snmp queries are now failing, something blocking network access?

[PriSupport]

Yes the machine is in domain but no scripts are executee on this machine. Snmp queries are failing? Not sure what to say why, maybe from cacti perspective...as I said, the same machine pulls snmp requests using MRTG and graphs are fine without any interruptions. Could it be MRTG pulling from same device that is conflicting? Would hate to stop mrtg and loose continuity...as you can tell I am trying to step up from MRTG to Cacti.

It seems that cacti is working though midnight as of last night and this would be 1st day that it is going past midnight.
However, I still see errors in logs:

08/09/2012 10:06:05 AM - SYSTEM STATS: Time:4.7046 Method:cmd.php Processes:10 Threads:N/A Hosts:8 HostsPerProcess:1 DataSources:101 RRDsProcessed:54
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] Host[9] DS[78] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.5.1', and OID:''
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] Host[8] DS[74] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.4.2', and OID:''
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] Host[10] DS[87] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.13.1', and OID:'.1.3.6.1.2.1.2.2.1.16.6'
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] Host[10] DS[87] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.13.1', and OID:'.1.3.6.1.2.1.2.2.1.10.6'
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] Host[7] DS[68] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.1.1', and OID:''
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] Host[10] DS[82] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:05 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.13.1', and OID:''
08/09/2012 10:06:02 AM - CMDPHP: Poller[0] Host[6] DS[62] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:02 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.10.1', and OID:''
08/09/2012 10:06:02 AM - CMDPHP: Poller[0] Host[4] DS[89] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:02 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.1.2', and OID:''
08/09/2012 10:06:02 AM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SNMP not valid. Partial Result: U
08/09/2012 10:06:02 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.1.2', and OID:''
08/09/2012 10:01:05 AM - SYSTEM STATS: Time:4.8518 Method:cmd.php Processes:10 Threads:N/A Hosts:8 HostsPerProcess:1 DataSources:101 RRDsProcessed:54

Thanks again for your time and hanging in with the newbie.

[BSOD2600]

Turned the logging level to debug and looked exactly what is going on yet?
Are all those DS's standard Cacti templates or custom?

[PriSupport]

So far graphs are going good 2 days in a row.

All default templates.
Not much 'error' related logs just more of the same timeouts. Not sure how to take the errors of 'Result from SNMP not valid'...normal?latency?
Here is a small snip:

08/10/2012 04:41:05 PM - CMDPHP: Poller[0] Host[10] DS[87] WARNING: Result from SNMP not valid. Partial Result: U
08/10/2012 04:41:05 PM - CMDPHP: Poller[0] Host[7] RECACHE DQ[1] OID: .1.3.6.1.2.1.1.3.0
08/10/2012 04:41:05 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.13.1', and OID:'.1.3.6.1.2.1.2.2.1.16.6'
08/10/2012 04:41:05 PM - CMDPHP: Poller[0] Host[7] RECACHE: Processing 1 items in the auto reindex cache for '192.168.1.1'.
08/10/2012 04:41:05 PM - CMDPHP: Poller[0] Host[7] SNMP: Host responded to SNMP
08/10/2012 04:41:05 PM - CMDPHP: Poller[0] Host[10] DS[87] SNMP: v1: 192.168.13.1, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.6, output: U
08/10/2012 04:41:05 PM - CMDPHP: Poller[0] Host[10] DS[87] WARNING: Result from SNMP not valid. Partial Result: U
08/10/2012 04:41:05 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.13.1', and OID:'.1.3.6.1.2.1.2.2.1.10.6'

My ISP controls the routers so I can't confirm all interfaces but it points to this ID:

interfaces.jpg (31.06 KiB) Viewed 3443 times

Thank you.

[BSOD2600]

Not much 'error' related logs just more of the same timeouts. Not sure how to take the errors of 'Result from SNMP not valid'...normal?latency?

Manually run a snmpwalk against the OIDs in question, so for example .1.3.6.1.2.1.2.2.1.10.6 and see what the output is. I bet for this specific case, the index changed, which makes it invalid.

[PriSupport]

Not sure if it was but I realized that some of interfaces of these routers are not in use so I eliminated them as data sources. Looking good now. I do think that rights to scheduler were throwing of the scheduled task so I will have to keep an eye out. Finally some solid graphs, thank you gentlemen for your patience.