[solved] Graphs not working for Cisco devices

[rsutton]

Cacti 8.7d
Win2k3/IIS6

I finally got all my servers in to cacti and now I have added a Cisco Pix a Cisco 3000 concentrator and a few Cisco switches. The problem is that I am not able to get graphs for any of the Cisco stuff! Here is some info from what I've checked; this is in my log file every poll cycle:

03/17/2009 11:41:19 AM - CMDPHP: Poller[0] Host[31] DS[203] WARNING: Result from SNMP not valid. Partial Result: U

I don't know what that means but if I snmpwalk the OID I am able to see the data. When I turn on graph debug I get this output:
RRDTool Command:

c:/cacti/rrdtool.exe graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="FBPIX - Traffic - 10.10.10.1 (PIX Firewall in)" \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="bits per second" \
--slope-mode \
--font TITLE:12:C:\windows\fonts\times.ttf \
--font AXIS:8:C:\windows\fonts\times.ttf \
--font LEGEND:10:C:\windows\fonts\times.ttf \
--font UNIT:8:C:\windows\fonts\times.ttf \
DEF:a="C\:/Inetpub/wwwroot/cacti/rra/fbpix_traffic_in_226.rrd":traffic_in:AVERAGE \
DEF:b="C\:/Inetpub/wwwroot/cacti/rra/fbpix_traffic_in_226.rrd":traffic_out:AVERAGE \
CDEF:cdefa=a,8,* \
CDEF:cdefe=b,8,* \
AREA:cdefa#00CF00FF:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97FF:"Outbound" \
GPRINT:cdefe:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s\n"

RRDTool Says:

ERROR: opening 'C:/Inetpub/wwwroot/cacti/rra/fbpix_traffic_in_226.rrd': No such file or directory

I had that ERROR opening problem on some of my Windows servers but I fixed it by fixing my NTFS permissions, I'm not sure why I am getting it again.

*edit* I am using the built in Cacti templates for Grabbing Cisco interface traffic.

[BSOD2600]

Click on that DS[203] link -- what does it go to? What is the specific OID which cacti is trying to query for it (look in the poller/snmp caches).

Post an SNMPWALK of the OID from DS[203].

[rsutton]

The DS[203] link goes to "FBPIX - 5 Minute CPU " data source, which also isn't working although I figured I would look in to that after I get the important (traffic) data working. I would rather look in to this later as I need to get traffic data working. I am not getting any other errors as far as I can tell that pertain to traffic data.

[TheWitness]

Some of the OID's are very PIX version specific. I don't run them, so I can't comment. A little research here goes a long way.

TheWitness

[rsutton]

Where can I find supported Cisco models/OS information for the built in Cisco Template?

I am using a Cisco Pix 515 with 6.4. I also am using a Catalyst 2900 switches and a Cisco 3000 VPN appliance. All very standard equipment.

[BSOD2600]

As with all cacti debugging, when stuff stops working the first step is to turn on the debugging tools cacti provides. First thats graph/data source debugging turned on. Next, changing the cacti logging level to debug for a polling cycle and sorting through the data to find out what a specific query/oid/script is not working.

Have you done this yet with the Cisco devices?
Please give specific examples of what types of templates are not working and what you've done to troubleshoot the issue.

[rsutton]

The templates I'm using on my Pix are:

Host template - Cisco Router
Data Template - Interface - Traffic Get SNMP Data (Indexed)
The graphs I'm tryign to create are:
FBPIX - Traffic - 10.10.10.1 (PIX Firewall in) 217 Interface - Traffic (bits/sec) 120x500
FBPIX - Traffic - 64.x.x.x (PIX Firewall ou) 216 Interface - Traffic (bits/sec)

I have set the logging level to debug, here are the lines in the log pertaining to the pix:

03/18/2009 10:36:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe update C:\Inetpub\wwwroot\cacti\rra\fbpix_traffic_in_225.rrd --template traffic_in:traffic_out N:212151273:721971045

03/18/2009 10:36:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe create C:/Inetpub/wwwroot/cacti/rra/fbpix_traffic_in_225.rrd --step 300 DS:traffic_in:COUNTER:600:0:9000000000000000 DS:traffic_out:COUNTER:600:0: 100000000 RRA:AVERAGE:0.5:1:600 RRA:AVERAGE:0.5:6:700 RRA:AVERAGE:0.5:24:775 RRA:AVERAGE:0.5:288:797 RRA:MAX:0.5:1:600 RRA:MAX:0.5:6:700 RRA:MAX:0.5:24:775 RRA:MAX:0.5:288:797

03/18/2009 10:36:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe update C:\Inetpub\wwwroot\cacti\rra\fbpix_5min_cpu_203.rrd --template 5min_cpu 1237397781:U

03/18/2009 10:36:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe update C:\Inetpub\wwwroot\cacti\rra\fbpix_traffic_in_226.rrd --template traffic_in:traffic_out N:387130362:3305666044

03/18/2009 10:36:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe create C:/Inetpub/wwwroot/cacti/rra/fbpix_traffic_in_226.rrd --step 300 DS:traffic_in:COUNTER:600:0:9000000000000000 DS:traffic_out:COUNTER:600:0: 100000000 RRA:AVERAGE:0.5:1:600 RRA:AVERAGE:0.5:6:700 RRA:AVERAGE:0.5:24:775 RRA:AVERAGE:0.5:288:797 RRA:MAX:0.5:1:600 RRA:MAX:0.5:6:700 RRA:MAX:0.5:24:775 RRA:MAX:0.5:288:797

I do not see any errors in there, but I am too familiar with the lingo. The only thing I have done to the data template is increase the max data value.

[BSOD2600]

The templates I'm using on my Pix are...

Alright, those appear to be standard Cacti template so they should work...

I have set the logging level to debug, here are the lines in the log pertaining to the pix:

I don't see any entries regarding cacti actually acquiring the data. Possibly look for DS[225], DS[226] in the log. This is likely the point of failure. Also if cacti keeps trying to create rrd files each polling cycle, you have incorrect NTFS permissions.

The only thing I have done to the data template is increase the max data value.

You shouldn't need to do that. The interface template takes the ifSpeed and sets that as the DS maximum automatically.

Please post the verbose query of the snmp interface.

[rsutton]

Possibly look for DS[225], DS[226] in the log. This is likely the point of failure.

I found both of these referenced in the log. Here is the output:
03/18/2009 01:01:22 PM - CMDPHP: Poller[0] Host[31] DS[226] SNMP: v1: FBPIX.mydomain.COM, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, output: 871459933
03/18/2009 01:01:22 PM - CMDPHP: Poller[0] Host[31] DS[203] SNMP: v1: FBPIX.mydomain.COM, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.2.1.58.0, output: U
03/18/2009 01:01:22 PM - CMDPHP: Poller[0] Host[31] DS[203] WARNING: Result from SNMP not valid. Partial Result: U
03/18/2009 01:01:22 PM - CMDPHP: Poller[0] Host[31] DS[226] SNMP: v1: FBPIX.mydomain.COM, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, output: 1687297079
03/18/2009 01:01:22 PM - CMDPHP: Poller[0] Host[31] DS[225] SNMP: v1: FBPIX.mydomain.COM, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, output: 2900872321
03/18/2009 01:01:22 PM - CMDPHP: Poller[0] Host[31] DS[225] SNMP: v1: FBPIX.mydomain.COM, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.1, output: 1211461064
03/18/2009 01:01:22 PM - CMDPHP: Poller[0] Host[31] RECACHE: Processing 2 items in the auto reindex cache for 'FBPIX.mydomain.COM'.

Also if cacti keeps trying to create rrd files each polling cycle, you have incorrect NTFS permissions.

Keep in mind I don't have problems on the servers I am monitoring, so I don't understand why permissions would be different for the Cisco devices. Regardless, the user that runs the poller task has read/write/modify on the entire web root . I also just double checked that he has full control on cmd.exe and php.exe, which he does.

Please post the verbose query of the snmp interface.

I believe that is what I posted above from the Cacti log. If that is not it please let me know.

You shouldn't need to do that. The interface template takes the ifSpeed and sets that as the DS maximum automatically.

That is good to know, thank you.

[TheWitness]

Simply said, the Cicso CPU should not work on anything other than an IOS device that supports that OID. The PIX is not on that list. Please keep this in mind.

Also, please note that MEDIUM is the right log level for debugging this issue. If you have a device that returns a "U" when doing snmp, but other counters work on that device, that simply means that your device does not support that OID (aka find a different template).

TheWitness

[BSOD2600]

I found both of these referenced in the log. Here is the output:

All looks good, except the 5min cpu which I guess that device doesnt use that OID. Forgot to mention to also look in that log file for 225.rrd and 226.rrd (or whatever the names of the rrd files for those two data sources). Interested if cacti is invoking an rrdtool create or update command each time.

Please post the verbose query of the snmp interface.

I believe that is what I posted above from the Cacti log. If that is not it please let me know.

Nope. Devices -> FBPIX.mydomain.COM device -> Associated Data Queries section -> Click on Verbose Query for SNMP - Interface Statistics. Since the debugging output from above spit out valid data for the interfaces, I don't think this is the problem anymore.

[rsutton]

Unrelated but my pix has been having issues and last night we replaced it with an ASA 5510. I'm reconfiguring SNMP on the ASA and I will see if I have the same issues or not.

[rsutton]

Forgot to mention to also look in that log file for 225.rrd and 226.rrd

03/19/2009 10:41:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe update C:\Inetpub\wwwroot\cacti\rra\fbpix_traffic_in_225.rrd --template traffic_out:traffic_in N:0:0
03/19/2009 10:41:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe create C:/Inetpub/wwwroot/cacti/rra/fbpix_traffic_in_225.rrd --step 300 DS:traffic_in:COUNTER:600:0:9000000000000000 DS:traffic_out:COUNTER:600:0: 100000000 RRA:AVERAGE:0.5:1:600 RRA:AVERAGE:0.5:6:700 RRA:AVERAGE:0.5:24:775 RRA:AVERAGE:0.5:288:797 RRA:MAX:0.5:1:600 RRA:MAX:0.5:6:700 RRA:MAX:0.5:24:775 RRA:MAX:0.5:288:797

03/19/2009 10:41:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe update C:\Inetpub\wwwroot\cacti\rra\fbpix_traffic_in_226.rrd --template traffic_out:traffic_in N:3046547102:495449953
03/19/2009 10:41:22 AM - POLLER: Poller[0] CACTI2RRD: c:/cacti/rrdtool.exe create C:/Inetpub/wwwroot/cacti/rra/fbpix_traffic_in_226.rrd --step 300 DS:traffic_in:COUNTER:600:0:9000000000000000 DS:traffic_out:COUNTER:600:0: 100000000 RRA:AVERAGE:0.5:1:600 RRA:AVERAGE:0.5:6:700 RRA:AVERAGE:0.5:24:775 RRA:AVERAGE:0.5:288:797 RRA:MAX:0.5:1:600 RRA:MAX:0.5:6:700 RRA:MAX:0.5:24:775 RRA:MAX:0.5:288:797

Please post the verbose query of the snmp interface.

+ Running data query [1].
+ Found type = '3' [snmp query].
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/snmp_queries/interface.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.2.1.2.2.1.1'
+ Index found at OID: 'interfaces.ifTable.ifEntry.ifIndex.1' value: '1'
+ Index found at OID: 'interfaces.ifTable.ifEntry.ifIndex.2' value: '2'
+ Index found at OID: 'interfaces.ifTable.ifEntry.ifIndex.3' value: '3'
+ Index found at OID: 'interfaces.ifTable.ifEntry.ifIndex.4' value: '4'
+ Index found at OID: 'interfaces.ifTable.ifEntry.ifIndex.5' value: '5'
+ Index found at OID: 'interfaces.ifTable.ifEntry.ifIndex.6' value: '6'
+ Index found at OID: 'interfaces.ifTable.ifEntry.ifIndex.7' value: '7'
+ Located input field 'ifIndex' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.1'
+ Found item [ifIndex='1'] index: 1 [from value]
+ Found item [ifIndex='2'] index: 2 [from value]
+ Found item [ifIndex='3'] index: 3 [from value]
+ Found item [ifIndex='4'] index: 4 [from value]
+ Found item [ifIndex='5'] index: 5 [from value]
+ Found item [ifIndex='6'] index: 6 [from value]
+ Found item [ifIndex='7'] index: 7 [from value]
+ Located input field 'ifOperStatus' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.8'
+ Found item [ifOperStatus='Up'] index: 1 [from value]
+ Found item [ifOperStatus='Up'] index: 2 [from value]
+ Found item [ifOperStatus='Up'] index: 3 [from value]
+ Found item [ifOperStatus='Down'] index: 4 [from value]
+ Found item [ifOperStatus='Down'] index: 5 [from value]
+ Found item [ifOperStatus='Down'] index: 6 [from value]
+ Found item [ifOperStatus='Up'] index: 7 [from value]
+ Located input field 'ifDescr' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.2'
+ Found item [ifDescr='Adaptive Security Appliance Null0 interface'] index: 1 [from value]
+ Found item [ifDescr='Adaptive Security Appliance outside interface'] index: 2 [from value]
+ Found item [ifDescr='Adaptive Security Appliance inside1 interface'] index: 3 [from value]
+ Found item [ifDescr='Adaptive Security Appliance Ethernet0/2 interface'] index: 4 [from value]
+ Found item [ifDescr='Adaptive Security Appliance Ethernet0/3 interface'] index: 5 [from value]
+ Found item [ifDescr='Adaptive Security Appliance management interface'] index: 6 [from value]
+ Found item [ifDescr='Adaptive Security Appliance Virtual254 interface'] index: 7 [from value]
+ Located input field 'ifName' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.31.1.1.1.1'
+ Found item [ifName='Null0'] index: 1 [from value]
+ Found item [ifName='outside'] index: 2 [from value]
+ Found item [ifName='inside1'] index: 3 [from value]
+ Found item [ifName='Ethernet0/2'] index: 4 [from value]
+ Found item [ifName='Ethernet0/3'] index: 5 [from value]
+ Found item [ifName='management'] index: 6 [from value]
+ Found item [ifName='Virtual254'] index: 7 [from value]
+ Located input field 'ifAlias' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.31.1.1.1.18'
+ Found item [ifAlias=''] index: 1 [from value]
+ Found item [ifAlias=''] index: 2 [from value]
+ Found item [ifAlias=''] index: 3 [from value]
+ Found item [ifAlias=''] index: 4 [from value]
+ Found item [ifAlias=''] index: 5 [from value]
+ Found item [ifAlias=''] index: 6 [from value]
+ Found item [ifAlias=''] index: 7 [from value]
+ Located input field 'ifType' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.3'
+ Found item [ifType='other(1)'] index: 1 [from value]
+ Found item [ifType='ethernetCsmacd(6)'] index: 2 [from value]
+ Found item [ifType='ethernetCsmacd(6)'] index: 3 [from value]
+ Found item [ifType='ethernetCsmacd(6)'] index: 4 [from value]
+ Found item [ifType='ethernetCsmacd(6)'] index: 5 [from value]
+ Found item [ifType='ethernetCsmacd(6)'] index: 6 [from value]
+ Found item [ifType='other(1)'] index: 7 [from value]
+ Located input field 'ifSpeed' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.5'
+ Found item [ifSpeed=' 0'] index: 1 [from value]
+ Found item [ifSpeed=' 1000000000'] index: 2 [from value]
+ Found item [ifSpeed=' 100000000'] index: 3 [from value]
+ Found item [ifSpeed=' 100000000'] index: 4 [from value]
+ Found item [ifSpeed=' 100000000'] index: 5 [from value]
+ Found item [ifSpeed=' 100000000'] index: 6 [from value]
+ Found item [ifSpeed=' 0'] index: 7 [from value]
+ Located input field 'ifHwAddr' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.2.2.1.6'
+ Found item [ifHwAddr='00:00:00:00:00:00'] index: 1 [from value]
+ Found item [ifHwAddr='00:22:90:C1:FC:C4'] index: 2 [from value]
+ Found item [ifHwAddr='00:22:90:C1:FC:C5'] index: 3 [from value]
+ Found item [ifHwAddr='00:22:90:C1:FC:C6'] index: 4 [from value]
+ Found item [ifHwAddr='00:22:90:C1:FC:C7'] index: 5 [from value]
+ Found item [ifHwAddr='00:22:90:C1:FC:C8'] index: 6 [from value]
+ Found item [ifHwAddr='00:00:00:00:00:00'] index: 7 [from value]
+ Located input field 'ifIP' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.2.1.4.20.1.2'
+ Found item [ifIP='10.10.10.1'] index: 3 [from regexp oid parse]
+ Found item [ifIP='12.177.21.9'] index: 2 [from regexp oid parse]
+ Found item [ifIP='192.168.1.1'] index: 6 [from regexp oid parse]
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/snmp_queries/interface.xml'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/snmp_queries/interface.xml'


So now this data is all pertaining to my Cisco ASA 5510 INSTEAD of my PIX 515, not sure if that is relevant or not.

[BSOD2600]

ah-ha, as I suspected. Cacti is trying to create the rrd files each polling cycle. You sure the NTFS permissions haven't changed lately? The scheduled task user and IIS_WPG accounts need Modify rights to the \cacti\rrd\ directory.

A test to confirm the theory. Stop the scheduled task. You manually invoke the poller with an administrative account. If the rrd files successfully get created this time, problem found/confirmed.

[rsutton]

Here are the permissions set, I don't see what i could be missing:




Permissions are set to apply to child objects.