Graphs are blank

[chaachee]

I had everything working fine until a few days ago, graphing of interface traffic against my PIX firewall started failing.

Looks like a value of 429496700 is added to the mySQL poller_reindex table that causes this to want to recache the host. Clearing the value from mySQL simply gets added next poll.

Cacti Version 0.8.6f
rrdtool 1.2.10
MySQL 4.1.13a-nt

The log debug shows this (the important details in bold):

08/23/2005 09:54:26 AM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: c:\php\php-win.exe, ARGS: -q d:\inetpub\wwwroot\cacti/cmd.php 0 2]
08/23/2005 09:54:26 AM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: c:\php\php-win.exe, ARGS: -q d:\inetpub\wwwroot\cacti/cmd.php 3 4]

[snip]

08/23/2005 09:54:27 AM - PHPSVR: Poller[0] DEBUG: FILENM: d:\inetpub\wwwroot\cacti\script_server.php
08/23/2005 09:54:27 AM - PHPSVR: Poller[0] PHP Script Server has Started - Parent is cmd
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] PHP Script Server Started Properly
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[3] SNMP: Host responded to SNMP
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[3] RECACHE: Processing 1 items in the auto reindex cache for '208.59.224.129'.
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[2] SNMP: Host responded to SNMP
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[2] RECACHE: Processing 1 items in the auto reindex cache for '192.168.1.1'.
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[3] DS[10] SNMP: v1: 208.59.224.129, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, output: 4114176487
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] ASSERT: '429496700<429496700' failed. Recaching host '192.168.1.1', data query #1.

08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[3] DS[10] SNMP: v1: 208.59.224.129, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.1, output: 2740115217
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[2] NOTICE: Spike Kill in Effect for '192.168.1.1'.
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[2] DS[8] SNMP: v1: 192.168.1.1, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, output: 2755236966
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[3] DS[11] SNMP: v1: 208.59.224.129, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, output: 3627323573
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[3] DS[11] SNMP: v1: 208.59.224.129, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, output: 4287231038
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[4] SNMP: Host responded to SNMP
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[2] DS[9] SNMP: v1: 192.168.1.1, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, output: 4044161255
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[4] RECACHE: Processing 1 items in the auto reindex cache for 'csvpn01'.
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[2] DS[9] SNMP: v1: 192.168.1.1, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, output: 1902722092
08/23/2005 09:54:27 AM - PHPSVR: Poller[0] DEBUG: PHP Script Server Shutdown request received, exiting
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Time: 0.8485 s, Theads: N/A, Hosts: 1
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[4] DS[12] SNMP: v1: csvpn01, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, output: 374405207
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[4] DS[12] SNMP: v1: csvpn01, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.1, output: 36379778
08/23/2005 09:54:27 AM - CMDPHP: Poller[0] Host[4] DS[13] SNMP: v1: csvpn01, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, output: 85089575
08/23/2005 09:54:28 AM - CMDPHP: Poller[0] Host[4] DS[13] SNMP: v1: csvpn01, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, output: 228290306
08/23/2005 09:54:28 AM - PHPSVR: Poller[0] DEBUG: PHP Script Server Shutdown request received, exiting

[snip]

08/23/2005 09:54:28 AM - POLLER: Poller[0] Host[2] WARNING: Recache Event Detected for Host
08/23/2005 09:54:28 AM - POLLER: Poller[0] Host[2] RECACHE: Re-cache for Host, data query #1
08/23/2005 09:54:29 AM - POLLER: [/b]Poller[0] Host[2] RECACHE: Re-cache successful.
08/23/2005 09:54:29 AM - RECACHE: Poller[0] STATS: Time: 0.5919 s, Hosts Recached: 1

[chaachee]

Can anyone provide any information as to what's going on here?

I removed and recreated the device and all associated graphs, but the problem returned as soon as the device was first polled. The main indication remains this error in the log file:

POLLER: Poller[0] Host[5] WARNING: Recache Event Detected for Host

Also, polling that occured while the device was removed logged this error in the file:

CMDPHP: Poller[0] ERROR: Either there are no items in the cache or polling is disabled

I have 3 devices being polled, all Cisco hardware, but only the PIX has the problem (and it used to work fine).

[BSOD2600]

1) do you have polling disabled?
2) recaching isn't bad per say... just takes more time. On host 2 and 3, look at the Associated Data Queries. What Re-Index Methods are you using?
3) Are the interfaces truly changing that often on the devices?

[chaachee]

1. No, polling is unabled. I simply removed the device that was giving me blank graphs and all graphs associated with it.

2. Re-caching wouldn't be a problem is something showed up... eventually. Right now, nothing shows up ever because 429496700<429496700 (whatever that means) everytime.

3. Not sure what you mean here. The polls happen every 5 minutes, but always to the same interfaces.

[MrBill]

My issue is with a v7 PIX. Stopped getting graphs on Aug 21.
For each poll cycle I get the following entries:
--------------------------
08/29/2005 03:15:05 PM - RECACHE: Poller[0] STATS: Time: 1.1695 s, Hosts Recached: 1
08/29/2005 03:15:04 PM - POLLER: Poller[0] Host[3] WARNING: Recache Event Detected for Host
08/29/2005 03:15:04 PM - SYSTEM STATS: Time: 3.0589 s, Method: cmd.php, Processes: 1, Threads: N/A, Hosts: 4, Hosts/Process: 4, Data Sources 23, RRDs Processed 14
08/29/2005 03:15:03 PM - CMDPHP: Poller[0] ASSERT: '429496700<429496700' failed. Recaching host '10.0.99.1', data query #1
------------------------
I've tried clearing the poller cache but the function doesn't seem to work on my box.
I know the Cacti version is 0.8.6f. I don't know the exact versions of MySQL and RRD, but they were the most current (stable) as of mid-July.

Any ideas anybody?

[MrBill]

I think the error message is complaining because the uptime for the firewall hasn't changed. "429496700" is the reported uptime returned by the PIX.
If I look at the box through the Devices link, the uptime never changes. If I look at another PIX with a different ver, the reported uptime does change.
The device with the issue is v7.0(1). My 6.3(4) PIX is not having any issue.
So, a combination of issues are colliding: the PIX is not sending a correct (updated) uptime, and Cacti is interpreting this data to mean the device has stalled.

I don't have time at the moment to browse the MIB tree to determine if the
value returned as uptime is actually uptime. Hopefully I'll have time next week to work on this further and determine a workaround.

[Sonny]

I am getting the exact same issue - my PIX is v7.0(2) and I'm using same version of Cacti as you are.

My uptime according to the PIX is 49 days 17 hours - the same as it was reporting yesterday (taken from a 'sh ver'). I've reported this as a bug to Cisco, but a quick Google turned up one result saying this was an issue with failover units (I run failover as well)...Interesting - will report the results once I hear back form Cisco.

[MrBill]

My box with the issue does not have any failover configured.
Performing a MIB browse on the box shows the Uptime stuck at 49 days.
But if you look at the uptime reported by ASDM, that uptime is correct.

Opening a case with TAC was going to be my next course of action, but
since you have done so I will hold-off on any further action.

[Sonny]

Sorry, my CCO contract has expired so TAC won't answer my enquiry...have posted on the Cisco forums, but until I get my contract with Cisco updated again I doubt I'll get a quick answer...

[Sonny]

Got this back from Cisco forums:

This is a known bug (CSCei20682), due to the time being stored in milliseconds in a 32-bit variable and 49 days 17 hours is where you hit the limit.

Already fixed in an upcoming 7.0 release, so keep checking the release notes here:

http://www.cisco.com/cgi-bin/tablebuild.pl/pix-interim

and you should see it listed. For the time being unfortunately you'll have to reboot the PIX to get the uptime back to normal.

[chaachee]

This has been fixed by Cisco in Cisco PIX Security Appliance Version 7.0(4)