can't create graphs for one device only

[JimDiGriz]

Dear Cacti community,

looking for your expertise.
I've been running cacti 1.2.21

Cacti Version 1.2.21
Cacti OS unix
NET-SNMP Version NET-SNMP version: 5.7.2
RRDtool Version Configured 1.4.0+
RRDtool Version Found 1.4.8
Devices 730
Graphs 33,886
Data Sources Script/Command: 5
SNMP Get: 360
SNMP Query: 33,276
Total: 33,641

and can't figure out of creating graphs for one Cisco 4500 series switch. I have all of the devices running well - all graphs and statistic but this particular C4500 switch is something strange.

When I add this device to cacti it can read the snmp info and provide me the list of interfaces

cacti_1.png (21.88 KiB) Viewed 844 times

When I create the graphs for some of interfaces it goes well, graphs are not created:

cacti_2.png (57.37 KiB) Viewed 844 times

debug graph mode says:
RRDtool Says:

ERROR: opening '/usr/share/cacti/rra/xxxx_traffic_in_43836.rrd': No such file or directory

and indeed there are no files for this device in rra directory.
It looks really strange but old cacti 0.8.8c can monitor this Cisco 4500 very well

What I tried so far:
- changed the graph template and device template (found old C4500) but no luck;
- tried snmpwalk from cacti and I get a lot of data, ping and snmp work well;
- changed the C4500 ip interface
- I use spine as poller, tried to switch to cmd.php but no luck


If anyone has an idea how to fix it, I'd appreciate your help!

[cigamit]

On the console, click on Troubleshooting, and enable debugging for that data source. That should help see where things are failing.

[TheWitness]

Those nasty old 4500's are notoriously buggy with their SNMP agent support. You may have to reduce the MAX OID's to 1 to make it work well.

[TheWitness]

One interesting way to debug this is by running spine in read only mode:

Code: Select all

./spine -R -V 5 -S -f device_id -l device_id

[JimDiGriz]

it does not tell me any hints

cacti_3.png (60.7 KiB) Viewed 840 times

[cigamit]

it does not tell me any hints

cacti_3.png

Thats not the Troubleshooter. Look again.

[JimDiGriz]

it does not tell me any hints

cacti_3.png

Thats not the Troubleshooter. Look again.

found the right console tab

cacti_4.png (35.23 KiB) Viewed 836 times

so, rrd files are not created, any clue what may be wrong ? buggy switch 4500 firmware ?

[JimDiGriz]

Those nasty old 4500's are notoriously buggy with their SNMP agent support. You may have to reduce the MAX OID's to 1 to make it work well.

Thanks, I set MAX OID's to 1 and no luck so far.

[TheWitness]

Do the spine thingy then.

[TheWitness]

What's worrisome is that the traffic_in and traffic_out data sources are not in the RRDfile. Can you post an rrdtool info filename for one of those RRDfiles?

[cigamit]

I think its not in the file, as the file doesn't exist (it doesn't create it until its actually polled)

[TheWitness]

Okay, got it Jimmy. Then, the next steps are:

1) Check the poller cache to make sure the entries exist. Rebuild it for the host (upper right corner of device edit page)
2) Test spine side band to ensure that the device is responding properly to pings and getting good data.
3) Be warry of 'Reindex Checks' that will insert unknown data into the RRDfiles. Though they don't exist, so that would not be a good cause.

[JimDiGriz]

What's worrisome is that the traffic_in and traffic_out data sources are not in the RRDfile. Can you post an rrdtool info filename for one of those RRDfiles?

I replaced the device id with xxx

Code: Select all

RRDtool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start='-86400' \
--end='-60' \
--pango-markup  \
--title='xxx-xx-VSS-xxx - Traffic - Te1/1/9' \
--vertical-label='bits per second' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--rigid \
--alt-autoscale-max \
--lower-limit='0' \
COMMENT:"From 09/Aug/2022 16\:56\:47 To 10/Aug/2022 16\:55\:47\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border $rrdborder --font TITLE:12: \
--font AXIS:10: \
--font LEGEND:12: \
--font UNIT:10: \
--font WATERMARK:8: \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/usr/share/cacti/rra/xxx-xx-vss-xxx_traffic_in_43854.rrd':'traffic_in':AVERAGE \
DEF:b='/usr/share/cacti/rra/xxx-xx-vss-xxx_traffic_in_43854.rrd':'traffic_out':AVERAGE \
CDEF:cdefa='a,8,*' \
CDEF:cdeff='b,8,*' \
LINE1:cdefa#00CF00FF:  \
AREA:cdefa#00CF007F:'Inbound '  \
GPRINT:cdefa:LAST:'Current\:%8.2lf%s'  \
GPRINT:cdefa:AVERAGE:'Average\:%8.2lf%s'  \
GPRINT:cdefa:MAX:'Maximum\:%8.2lf%s\n'  \
LINE1:cdeff#002A977F:'Outbound'  \
LINE1:cdeff#002A97FF:  \
GPRINT:cdeff:LAST:'Current\:%8.2lf%s'  \
GPRINT:cdeff:AVERAGE:'Average\:%8.2lf%s'  \
GPRINT:cdeff:MAX:'Maximum\:%8.2lf%s' 

RRDtool Command lengths = 1271 charaters.

RRDtool Says:

ERROR: opening '/usr/share/cacti/rra/xxx-ab-xxx-xxx_traffic_in_43854.rrd': No such file or directory

[JimDiGriz]

Okay, got it Jimmy. Then, the next steps are:

1) Check the poller cache to make sure the entries exist. Rebuild it for the host (upper right corner of device edit page)
2) Test spine side band to ensure that the device is responding properly to pings and getting good data.
3) Be warry of 'Reindex Checks' that will insert unknown data into the RRDfiles. Though they don't exist, so that would not be a good cause.

1. looks good to me

cacti_5.png (46.99 KiB) Viewed 825 times

2. snmp and ping responded well

cacti_6.png (27.89 KiB) Viewed 826 times

3. I tried to remove and re-added this device, re-indexed it without any luck so far

[TheWitness]

You still have not sent the rrdtool info filename output, or run spine the way I asked you to.