Cisco PIX stuck in "Unknown" status

[scarolan]

I have added a Cisco PIX to Cacti but it never comes out of "Unknown" status even after creating graphs. I have tried deleting and re-adding the device, deleting and re-adding graphs, and switching the Downed Device Detection to various settings, all to no avail. Can anyone help? I searched Google and these forums but was unable to find an answer. Our other switches and routers monitor just fine. Here is some more information:

System:Cisco PIX Firewall Version 6.1(4)
Only seems to run SNMP v1
I can snmpwalk and ping from the cacti host. The traffic counters are included in the snmpwalk output.
RRD files exist but seem to contain no data, graphs all say "NaN"

Your help is most appreciated, hopefully someone with more expertise than me has an idea how to troubleshoot this.

[gandalf]

Are you running spine? If yes, please see 2nd link of my sig to learn how to debug a single host using spine. Post those results, please
R.

[scarolan]

I was not using spine previously so decided to give it a shot. Here is what I have installed now:

[root@aus02syslog log]# rpm -q cacti
cacti-0.8.7g-1.el5.rf
[root@aus02syslog log]# rpm -q cacti-spine
cacti-spine-0.8.7a-1.el5.rf

The path to spine is /usr/bin/spine. I have this set up in my PATHS settings. When I turn debugging up and run with spine the cacti.log only contains this:

Code: Select all

08/12/2010 02:00:01 PM - POLLER: Poller[0] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300', Max Runtime '298', Poller Runs: '1'
08/12/2010 02:00:01 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/spine, ARGS:  0 119]
08/12/2010 02:00:01 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/spine, ARGS:  126 126]

Data collection stopped when I changed the setting to spine, so I put it back to cmd.php and left DEBUG on, and found all my other hosts are being polled but not Host[126] which is the problem host that stays in Unknown. Host[126] is not in the cacti.log file at all.

Here's what shows up in poller.log when I try to use spine:


08/12/2010 02:25:01 PM - POLLER: Poller[0] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300', Max Runtime '298', Poller Runs: '1'
08/12/2010 02:25:01 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/spine, ARGS: 0 119]
08/12/2010 02:25:01 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/spine, ARGS: 126 126]
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
Waiting on 2 of 2 pollers.
[etc]

[scarolan]

Some progress was made:

1. I had to install spine from source because the RPM in my repo was not compiled right. Spine is now working.
2. I deleted and re-added the PIX and now it shows "Up" in the device status.

I added two graphs, will wait a bit and see if the data sources can gather any data. I will post here when I have more info.

[scarolan]

Here's an update on this issue:

* Spine is correctly configured and polling all my hosts
* The problem host now shows "Up" instead of "Unknown"
* When I look at the Data Source info for this host it looks different than our other routers and switches. For some reason it says "Data query data sources must be created through New Graphs". My working hosts say something like "In/Out bits" and "IfDescr"
* The problem host is not generating any data, there is no rrd file, and graphs show as broken images. When I run it in debug mode I don't see anything about this host in the cacti.log output.

Any suggestions? As I mentioned above snmpwalk works fine, I can ping the host and cacti sees it as up.

[scarolan]

This is what was showing up on the Data Source debug output:

Code: Select all

/usr/bin/rrdtool create \
/var/www/cacti-0.8.7e/rra/mypix_traffic_in_473.rrd \
--step 300  \
DS:traffic_in:COUNTER:600:0:100000000 \
DS:traffic_out:COUNTER:600:0:100000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

But it never creates an rrd file. I have run the above command manually and it did go ahead and create the RRD file. Now I have a graph image, but no data. cacti.log and poller.log still don't show any data for this host.

[gandalf]

As given in 2nd link of my sig, do a single debug run on that host and post results, please
R.

[scarolan]

Here you go Gandalf. Sorry for the huge amount of data; I'm not sure what part of this you need. The host we are concerned with is called "austinpix", but I don't see it anywhere in these files.

[edit] There seems to be a limitation on the amount of data I can post here. I'm going to submit these files as attachments instead.

[edit 2]Now I am getting this error message:

Mistake! There was an attempt of an automatic insert of the message in a forum. Your message is sent to hell. Try still times who knows - can it will turn out? Still probably, that you too long wrote the message - then pass to page back, copy the text, update page, insert the copied text and press button "Send".

[edit 3]Ok, I give up. I can't upload the files or copy and paste them. I have placed the files on my server here:

http://code200.com/cacti.log
http://code200.com/poller.log

[gandalf]

As you are running spine, please use the mentioned hint about producing a spine debug log against a single host. This will help.
BTB: Editing a post for minutes makes phpBB angry ...
R.

[scarolan]

Sorry for the repeated edits, I was having a very hard time copying the large cacti.log output. Anyway here is the spine output for my problem host. I confirmed via the debug instructions that the <id> is in fact 131.

Code: Select all

[root@aus02syslog cacti]# spine --verbosity=5 131 131
SPINE: Using spine config file [/etc/spine.conf]
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The path_php_server variable is /var/www/cacti-0.8.7e/script_server.php
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The path_cactilog variable is /var/www/cacti/log/cacti.log
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The log_destination variable is 1 (FILE)
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The path_php variable is /usr/bin/php
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The availability_method variable is 2
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The ping_recovery_count variable is 3
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The ping_failure_count variable is 2
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The ping_method variable is 2
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The ping_retries variable is 1
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 400
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The snmp_retries variable is 3
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 0
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The boost_redirect variable is 0
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The log_pstats variable is 0
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The threads variable is 1
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The polling interval is 300 seconds
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The number of concurrent processes is 1
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The script timeout is 25
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The number of php script servers to run is 1
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: StartHost='131', EndHost='131', TotalPHPScripts='0'
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Required
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size is 10
08/18/2010 02:40:59 PM - SPINE: Poller[0] Version 0.8.7g starting
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: Spine is running asroot.
08/18/2010 02:40:59 PM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
08/18/2010 02:40:59 PM - SPINE: Poller[0] DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.
08/18/2010 02:41:00 PM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(s)
08/18/2010 02:41:00 PM - SPINE: Poller[0] NOTE: Spine did not detect multithreaded device polling.
08/18/2010 02:41:00 PM - SPINE: Poller[0] NOTE: Spine is behaving in a 0.8.7g manner
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads is 0
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
08/18/2010 02:41:00 PM - SPINE: Poller[0] Host[0] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
08/18/2010 02:41:00 PM - SPINE: Poller[0] Host[131] SNMP Result: Host responded to SNMP
08/18/2010 02:41:00 PM - SPINE: Poller[0] Host[131] TH[1] Host has no information for recache.
08/18/2010 02:41:00 PM - SPINE: Poller[0] Host[131] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
08/18/2010 02:41:00 PM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
08/18/2010 02:41:00 PM - SPINE: Poller[0] Time: 0.3605 s, Threads: 1, Hosts: 2

[gandalf]

Code: Select all

[root@aus02syslog cacti]...
08/18/2010 02:41:00 PM - SPINE: Poller[0] Host[131] SNMP Result: Host responded to SNMP
08/18/2010 02:41:00 PM - SPINE: Poller[0] Host[131] TH[1] Host has no information for recache.
08/18/2010 02:41:00 PM - SPINE: Poller[0] Host[131] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function

That's the interesting part. It shows, that the downed host detection is passed fine (... responded to SNMP). In a normal case, the next entries should show the items of the poller items table being executed one by one. That's missing here.
But before I call Larry for help (the guy who wrote SPINE), I want to know if there are valid entries for THIS HOST under "System Utilities -> View Poller cache". Please post those entries.
BTB: Which value is used as "max OIDs" for that very host?
R.

[scarolan]

Thanks for looking at this. There are no entries at all for host id 131 under "View Poller Cache".

Max OIDs for this host is set at 10. Some other info, not sure if it's helpful. this is how I set things up:

1. Added host via the Devices link. Set snmp to v1, configured downed host detection as SNMP, and entered correct community name.
2. Host is added fine, and shows some basic SNMP data at the top of the page. (Changed the name to protect the innocent:)

System:Cisco PIX Firewall Version 6.1(4)
Uptime: 238250600 (27 days, 13 hours, 48 minutes)
Hostname: MYPIX
Location: Office
Contact: me@company.com

3. Did "create graphs for this host" and made an In/Out bits with total bandwidth for these two interfaces:

1 Up PIX Firewall outside interface ethernetCsmacd(6) 100000000 00:50:54:FF:37:3C
2 Up PIX Firewall inside interface ethernetCsmacd(6) 100000000 00:50:54:FF:37:3D

4. Graphs do not show up; as I mentioned above they are missing the RRD file.

5. Manually created the RRD file using the command that shows up on data source debug mode. Now a blank graph shows up.


Here's the data source debug:

/usr/bin/rrdtool create \
/var/www/cacti-0.8.7e/rra/austinpix_traffic_in_473.rrd \
--step 300 \
DS:traffic_in:COUNTER:600:0:100000000 \
DS:traffic_out:COUNTER:600:0:100000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

And here's the graph debug output:

RRDTool Command:
/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title='austinpix - Traffic - PIX Firewall in/2' \
--rigid \
--base=1000 \
--height=250 \
--width=800 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label='bits per second' \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/var/www/cacti-0.8.7e/rra/austinpix_traffic_in_473.rrd":traffic_in:AVERAGE \
DEF:b="/var/www/cacti-0.8.7e/rra/austinpix_traffic_in_473.rrd":traffic_out:AVERAGE \
CDEF:cdefa=a,8,* \
CDEF:cdeff=b,8,* \
AREA:cdefa#00CF00FF:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s" \
COMMENT:"Total In\: 0 bytes\n" \
LINE1:cdeff#002A97FF:"Outbound" \
GPRINT:cdeff:LAST:"Current\:%8.2lf %s" \
GPRINT:cdeff:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdeff:MAX:"Maximum\:%8.2lf %s" \
COMMENT:"Total Out\: 0 bytes"
RRDTool Says:
OK

Oddly the poller never seems to actually gather any data.

Let me know if there are any other troubleshooting steps I should take.

[gandalf]

Until we resolve the issue, you should not create any rrd file manually. Best would be to delete the file. If all is fine, cacti will create it for you.

Then, you always need a single graph at minimum. In your case, this was a traffic graph. To single down an issue with traffic graphs, I recommend that you create a non-Data Query graph for the device and wait at least for 10 minutes.

In the meantime, please verify that this new entry propagated to "System Utilities -> View Poller Cache". If yes, I expect that graph and rrd file to be created within 10 minutes and device status to be up.

In this case, we should assume that there's something wrong with the interface query. But please make sure, that the related interface OIDs are snmpwalk-able from Cacti cli (you know, it's a PIX, it may reject SNMP queries against it's interfaces)

R.

[scarolan]

Which non-Data Query graph should I create? I selected "Unix - Ping Latency", but am not sure that is what you intended.

[scarolan]

Well whaddya know, a graph appeared for my ping latency:

austinpix - Ping Host Script: perl /var/www/cacti-0.8.7e/scripts/ping.pl 10.100.0.253
RRD: /var/www/cacti-0.8.7e/rra/austinpix_ping_474.rrd