Gaps in SNMP graph

[kribbas]

Hi,

We have a set of nodes fetching from snmp, all are working fine except one. Its working and then some times during the day it becomes gaps in the graph for shorter periods. In the log files I can't see any specific error except that the spine poller is timing out, the poller is also very high when this node is enabled. When I disabled the systems stats time is reduced with almost 80 seconds.

Any ideas about what could be the cause? Is there a way to see if this node is taking longer time for the poller to process and for what reason?

I have tried increasing the SNMP timeout, maximum OIDs per get request and the threads.

[Osiris]

You should not increase those settings willie nillie unless you know what you are doing first.

For example, increasing max OID's can actually have a negative impact, as well as threads.

What you should ask yourself is is the graphs data derives from a script or SNMP? If a script, then some templates out there don't respect the timeouts (It's a community thing). Then, is the data stored as a Counter or a Guage? Lastly, is there data discarding based upon a max value that is being exceeded in the rrdfile.

These are all basic diagnostics you have to do from time to time.

[kribbas]

Hi,

Answer:

- fetching from SNMP
- Stored as Gauge
- How do I check for the data discarding? not sure about that.

[Osiris]

The rrdtool info filename command, or edit the data source and turn on info mode.

[kribbas]

Then I get this:

--step 300 \
DS:CardMemResMemoryUse:GAUGE:600:0:U \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

When checking the graph we have values at: Current/average/max: 54.84M

[Osiris]

Those look like timeouts to me. If you simply want to mask the gaps, consider increasing the heartbeat to something like 2400 seconds. You can do this from the command line using the rrdtool tune filename with options. There is also a new script in GitHub called update_heartbeat.php in the 1.2.x branch you can use for a more large scale change.

[kribbas]

Ok Would this be OK:

rrdtool tune filename.rrd -h traffic_in:2400 -h traffic_out:2400

I could not find any "through" in the rrdtool info listing.

[Osiris]

Yea, that's for another rrdfile though right?

[kribbas]

Yes, i just meant "filename" as example. The old heartbeat was 600, monitoring now to see how the change to 2400 will impact.

Will this increase in heartbeat help my system stats? Currently its around 200-250, when I disable this particular node it reduces down to 100-150, so wondering if this heartbeat change will improve that aswell?

[TheWitness]

I suggest you run spine in debug mode to identify the source of the "slowdown"

Code: Select all

./spine -R -V 3 -S -f device_id -l device_id

Post the results and we can see why that host is in bad shape.

[Osiris]

TheWitness is correct v

[kribbas]

Here is the output:

[appadmin]$ ./spine -R -V 3 -S -f 1083 -l 1083
SPINE: Using spine config file [../etc/spine.conf]
SPINE: Version 0.8.8h starting
NOTE: Spine will support multithread device polling.
Host[0] TH[1] Total Time: 0.001 Seconds
Host[1083] TH[1] Recache DataQuery[1] OID: .1.3.6.1.2.1.1.3.0, output: 216969311 9
Host[1083] TH[1] Recache DataQuery[38] OID: .1.3.6.1.2.1.1.3.0, output: 21696931 19
Host[1083] TH[1] Recache DataQuery[40] OID: .1.3.6.1.2.1.1.3.0, output: 21696931 19
Host[1083] TH[1] NOTE: There are '164' Polling Items for this Host
PHP Warning: session_start(): open(/var/lib/php/session/sess_qhvdl8mavfmrlascqr g7tkrgn3, O_RDWR) failed: Permission denied (13) in /usr/share/cacti-0.8.8h/incl ude/global.php on line 157
PHP Warning: Unknown: open(/var/lib/php/session/sess_qhvdl8mavfmrlascqrg7tkrgn3 , O_RDWR) failed: Permission denied (13) in Unknown on line 0
PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session) in Un known on line 0
Host[1083] TH[1] DS[70884] SCRIPT: /usr/bin/php -q /usr/share/cacti-0.8.8h/scrip ts/ss_nokia_host_apn.php Host IP SNMP-community 2 161 10000 10 "" "" "" "" "" "" g et tmnxMobPdnApnStatHomers 16.111.109.110.105.116.101.108.117.110.108.105.109.10 5.116.101.100, output: 14
PHP Warning: session_start(): open(/var/lib/php/session/sess_km123dcntek6226eff 341v6273, O_RDWR) failed: Permission denied (13) in /usr/share/cacti-0.8.8h/incl ude/global.php on line 157
PHP Warning: Unknown: open(/var/lib/php/session/sess_km123dcntek6226eff341v6273, O_RDWR) failed: Permission denied (13) in Unknown on line 0
PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session) in Unknown on line 0
Host[1083] TH[1] DS[70883] SCRIPT: /usr/bin/php -q /usr/share/cacti-0.8.8h/scripts/ss_nokia_host_apn.php Host IP SNMP-community 2 161 10000 10 "" "" "" "" "" "" get tmnxMobPdnApnStatHomers 10.111.116.116.46.116.118.101.46.108.116, output: 791
PHP Warning: session_start(): open(/var/lib/php/session/sess_29ifgg76ktb738ncnv6i84fd14, O_RDWR) failed: Permission denied (13) in /usr/share/cacti-0.8.8h/include/global.php on line 157
PHP Warning: Unknown: open(/var/lib/php/session/sess_29ifgg76ktb738ncnv6i84fd14, O_RDWR) failed: Permission denied (13) in Unknown on line 0
PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session) in Unknown on line 0
Host[1083] TH[1] DS[70882] SCRIPT: /usr/bin/php -q /usr/share/cacti-0.8.8h/scripts/ss_nokia_host_apn.php Host IP SNMP-community 2 161 10000 10 "" "" "" "" "" "" get tmnxMobPdnApnStatHomers 7.111.109.110.105.116.101.108, output: 81474
Host[1083] TH[1] DS[70788] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1610891272, value: 33631927191478
Host[1083] TH[1] DS[70787] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611972632, value: 5810114017
Host[1083] TH[1] DS[70788] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1610891272, value: 2879728517
Host[1083] TH[1] DS[70880] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613807664, value: 836758450
Host[1083] TH[1] DS[70879] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613807656, value: 154127495902542
Host[1083] TH[1] DS[70880] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613807664, value: 841513789
Host[1083] TH[1] DS[70879] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613807656, value: 610632323542313
Host[1083] TH[1] DS[70878] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613807648, value: 820226125
Host[1083] TH[1] DS[70878] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613807648, value: 829555101
Host[1083] TH[1] DS[70877] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613807640, value: 103777450467670
Host[1083] TH[1] DS[70876] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613807632, value: 811588392
Host[1083] TH[1] DS[70877] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613807640, value: 106627362038271
Host[1083] TH[1] DS[70876] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613807632, value: 836226308
Host[1083] TH[1] DS[70875] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613807624, value: 102245801531812
Host[1083] TH[1] DS[70875] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613807624, value: 563664240492375
Host[1083] TH[1] DS[70874] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613545520, value: 7266070982
Host[1083] TH[1] DS[70873] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613545512, value: 644870673577595
Host[1083] TH[1] DS[70874] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613545520, value: 7916626351
Host[1083] TH[1] DS[70873] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613545512, value: 4006703355407122
Host[1083] TH[1] DS[70872] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613545504, value: 6557871639
Host[1083] TH[1] DS[70872] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613545504, value: 6379933811
Host[1083] TH[1] DS[70871] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613545496, value: 316781093178025
Host[1083] TH[1] DS[70870] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613545488, value: 5947331092
Host[1083] TH[1] DS[70871] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613545496, value: 428451899731177
Host[1083] TH[1] DS[70870] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613545488, value: 5806675790
Host[1083] TH[1] DS[70869] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613545480, value: 569079427133328
Host[1083] TH[1] DS[70869] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613545480, value: 4076481467277205
Host[1083] TH[1] DS[70868] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613283376, value: 5906108088
Host[1083] TH[1] DS[70867] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613283368, value: 674883490076978
Host[1083] TH[1] DS[70868] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613283376, value: 19498619704
Host[1083] TH[1] DS[70867] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613283368, value: 4075679139068112
Host[1083] TH[1] DS[70866] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613283360, value: 6507305568
Host[1083] TH[1] DS[70866] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613283360, value: 6630320488
Host[1083] TH[1] DS[70865] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613283352, value: 303293999098647
Host[1083] TH[1] DS[70864] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613021232, value: 645028955659896
Host[1083] TH[1] DS[70865] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613283352, value: 464229381822115
Host[1083] TH[1] DS[70864] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613021232, value: 3903641741116772
Host[1083] TH[1] DS[70863] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613021224, value: 10268917518
Host[1083] TH[1] DS[70863] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613021224, value: 5876766150
Host[1083] TH[1] DS[70862] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613021216, value: 302446000158418
Host[1083] TH[1] DS[70861] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613021208, value: 5689101639
Host[1083] TH[1] DS[70862] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613021216, value: 451343318587659
Host[1083] TH[1] DS[70861] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613021208, value: 5806252670
Host[1083] TH[1] DS[70860] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612759088, value: 27072
Host[1083] TH[1] DS[70860] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612759088, value: 0
Host[1083] TH[1] DS[70859] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612759080, value: 0
Host[1083] TH[1] DS[70858] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612759072, value: 0
Host[1083] TH[1] DS[70859] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612759080, value: 26752
Host[1083] TH[1] DS[70858] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612759072, value: 5632
Host[1083] TH[1] DS[70857] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612759064, value: 5504
Host[1083] TH[1] DS[70857] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612759064, value: 0
Host[1083] TH[1] DS[70856] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612496944, value: 6337403335
Host[1083] TH[1] DS[70855] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612496936, value: 666814243929307
Host[1083] TH[1] DS[70856] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612496944, value: 9587107409
Host[1083] TH[1] DS[70855] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612496936, value: 4236485520952678
Host[1083] TH[1] DS[70854] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612496928, value: 6613436691
Host[1083] TH[1] DS[70854] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612496928, value: 6623016022
Host[1083] TH[1] DS[70853] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612496920, value: 309767447175910
Host[1083] TH[1] DS[70852] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613283344, value: 5767062019
Host[1083] TH[1] DS[70853] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612496920, value: 439399323287151
Host[1083] TH[1] DS[70852] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613283344, value: 5812420900
Host[1083] TH[1] DS[70851] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613283336, value: 561014314856198
Host[1083] TH[1] DS[70851] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613283336, value: 4144689243566979
Host[1083] TH[1] DS[70850] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613021200, value: 3978990919598435
Host[1083] TH[1] DS[70849] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1613021192, value: 5723713509
Host[1083] TH[1] DS[70850] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613021200, value: 544563146588875
Host[1083] TH[1] DS[70849] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1613021192, value: 5805752377
Host[1083] TH[1] DS[70848] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612759056, value: 5632
Host[1083] TH[1] DS[70848] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612759056, value: 0
Host[1083] TH[1] DS[70847] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612759048, value: 0
Host[1083] TH[1] DS[70846] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612496912, value: 5809307128
Host[1083] TH[1] DS[70847] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612759048, value: 5632
Host[1083] TH[1] DS[70846] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612496912, value: 5806901633
Host[1083] TH[1] DS[70845] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612496904, value: 600538215051066
Host[1083] TH[1] DS[70845] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612496904, value: 4323064879717164
Host[1083] TH[1] DS[70844] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612234800, value: 626576674265902
Host[1083] TH[1] DS[70843] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612234792, value: 5852170454
Host[1083] TH[1] DS[70844] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612234800, value: 4019841624695340
Host[1083] TH[1] DS[70843] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612234792, value: 9679516424
Host[1083] TH[1] DS[70842] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612234784, value: 437929610566457
Host[1083] TH[1] DS[70842] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612234784, value: 297725698798950
Host[1083] TH[1] DS[70841] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612234776, value: 5694465209
Host[1083] TH[1] DS[70840] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612234768, value: 4098447195736204
Host[1083] TH[1] DS[70841] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612234776, value: 5805237671
Host[1083] TH[1] DS[70840] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612234768, value: 535834213234075
Host[1083] TH[1] DS[70839] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1612234760, value: 6406848226
Host[1083] TH[1] DS[70839] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1612234760, value: 5806461307
Host[1083] TH[1] DS[70838] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611972656, value: 671333396094625
Host[1083] TH[1] DS[70838] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611972656, value: 4137091633172694
Host[1083] TH[1] DS[70837] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611972648, value: 8920422967
Host[1083] TH[1] DS[70837] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611972648, value: 31297601251
Host[1083] TH[1] DS[70836] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611972640, value: 308558540299114
Host[1083] TH[1] DS[70836] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611972640, value: 433348796761344
Host[1083] TH[1] DS[70835] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611972624, value: 4206737176362323
Host[1083] TH[1] DS[70835] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611972624, value: 581582981539124
Host[1083] TH[1] DS[70834] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611972616, value: 6293516357
Host[1083] TH[1] DS[70834] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611972616, value: 5812447056
Host[1083] TH[1] DS[70833] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611677712, value: 51130328680023
Host[1083] TH[1] DS[70833] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611677712, value: 24797865826
Host[1083] TH[1] DS[70832] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611677704, value: 71057025126837
Host[1083] TH[1] DS[70832] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611677704, value: 25554229426
Host[1083] TH[1] DS[70831] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611415568, value: 50495016357311
Host[1083] TH[1] DS[70831] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611415568, value: 733929759401
Host[1083] TH[1] DS[70830] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611415560, value: 94278040928425
Host[1083] TH[1] DS[70830] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611415560, value: 97487421119370
Host[1083] TH[1] DS[70829] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611153440, value: 98505617093454
Host[1083] TH[1] DS[70829] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611153440, value: 2733239929208
Host[1083] TH[1] DS[70828] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611153432, value: 2650348281
Host[1083] TH[1] DS[70828] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611153432, value: 12588046025
Host[1083] TH[1] DS[70827] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611153424, value: 4732987559466
Host[1083] TH[1] DS[70827] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611153424, value: 10170491918649
Host[1083] TH[1] DS[70826] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611153416, value: 33664629596803
Host[1083] TH[1] DS[70826] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1611153416, value: 2846148659
Host[1083] TH[1] DS[70825] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1610891296, value: 96960
Host[1083] TH[1] DS[70825] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1610891296, value: 128442
Host[1083] TH[1] DS[70824] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1610891288, value: 3903080378750
Host[1083] TH[1] DS[70824] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1610891288, value: 200838710236
Host[1083] TH[1] DS[70823] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1610891280, value: 2674082151
Host[1083] TH[1] DS[70823] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1610891280, value: 2990390932
Host[1083] TH[1] DS[70822] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.7, value: 25523147398
Host[1083] TH[1] DS[70816] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardMemResMemoryUsed 9, output: 54833086
Host[1083] TH[1] DS[70815] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardMemResMemoryUsed 8, output: 53491759
Host[1083] TH[1] DS[70814] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardMemResMemoryUsed 11, output: 54821459
Host[1083] TH[1] DS[70813] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardMemResMemoryUsed 10, output: 54832542
Host[1083] TH[1] DS[70812] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardMemResMemoryUsed 7, output: 54814020
Host[1083] TH[1] DS[70811] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardMemResMemoryUsed 6, output: 54808435
Host[1083] TH[1] DS[70810] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardMemResMemoryUsed 5, output: 54836990
Host[1083] TH[1] DS[70809] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_60sec 9, output: 13.18
Host[1083] TH[1] DS[70808] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_300sec 9, output: 12
Host[1083] TH[1] DS[70807] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_1sec 9, output: 10.81
Host[1083] TH[1] DS[70806] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_60sec 8, output: 3.19
Host[1083] TH[1] DS[70805] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_300sec 8, output: 2.45
Host[1083] TH[1] DS[70804] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_1sec 8, output: 1.91
Host[1083] TH[1] DS[70803] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_60sec 11, output: 12.27
Host[1083] TH[1] DS[70802] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_300sec 11, output: 12.31
Host[1083] TH[1] DS[70801] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_1sec 11, output: 11.27
Host[1083] TH[1] DS[70800] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_60sec 10, output: 13.2
Host[1083] TH[1] DS[70799] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_300sec 10, output: 12.16
Host[1083] TH[1] DS[70798] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_1sec 10, output: 11
Host[1083] TH[1] DS[70797] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_60sec 7, output: 12.19
Host[1083] TH[1] DS[70796] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_300sec 7, output: 12.37
Host[1083] TH[1] DS[70795] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_1sec 7, output: 13.19
Host[1083] TH[1] DS[70794] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_60sec 6, output: 12.16
Host[1083] TH[1] DS[70793] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_300sec 6, output: 12.2
Host[1083] TH[1] DS[70792] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_1sec 6, output: 11.75
Host[1083] TH[1] DS[70791] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_60sec 5, output: 17.13
Host[1083] TH[1] DS[70790] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_300sec 5, output: 15.9
Host[1083] TH[1] DS[70789] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_host.php ss_nokia_host_usage Host IP:2:SNMP-community:::::::161:10000 get tmnxCardCpuResMonBusyCoreUtil_1sec 5, output: 15.77
PHP Warning: session_start(): open(/var/lib/php/session/sess_pb23ujpm1vkfa7krs8lrps2rv7, O_RDWR) failed: Permission denied (13) in /usr/share/cacti-0.8.8h/include/global.php on line 157
PHP Warning: Unknown: open(/var/lib/php/session/sess_pb23ujpm1vkfa7krs8lrps2rv7, O_RDWR) failed: Permission denied (13) in Unknown on line 0
PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session) in Unknown on line 0
Host[1083] TH[1] DS[70881] SCRIPT: /usr/bin/php -q /usr/share/cacti-0.8.8h/scripts/ss_nokia_host_apn.php Host IP SNMP-community 2 161 10000 10 "" "" "" "" "" "" get tmnxMobPdnApnStatHomers 7.98.111.110.100.105.110.103, output: 7185
Host[1083] TH[1] DS[70822] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.7, value: 1333778
Host[1083] TH[1] DS[70821] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.6, value: 25700814033
Host[1083] TH[1] DS[70821] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.6, value: 1334212
Host[1083] TH[1] DS[70820] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.5, value: 25089142605
Host[1083] TH[1] DS[70820] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.5, value: 1333778
Host[1083] TH[1] DS[70819] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.4, value: 13435386297
Host[1083] TH[1] DS[70819] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.4, value: 1459554
Host[1083] TH[1] DS[70818] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.3, value: 33664663495862
Host[1083] TH[1] DS[70818] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.3, value: 17676034
Host[1083] TH[1] DS[70817] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, value: 33631962158609
Host[1083] TH[1] DS[70817] SNMP: v2: Host IP, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.2, value: 50599930
Host[1083] TH[1] DS[70787] SNMP: v2: Host IP, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1611972632, value: 5688893396
Host[1083] TH[1] Total Time: 80 Seconds
Time: 79.7865 s, Threads: 12, Hosts: 2

[TheWitness]

That looks like 1) Old version of Cacti 2) Old version of spine.

Do that from Cacti 1.2.x with the latest 1.2.x branch of spine.

[TheWitness]

Also, one of your scripts is opening global.php and not cli_check.php. You need to change that.

Code: Select all

PHP Warning: session_start(): open(/var/lib/php/session/sess_pb23ujpm1vkfa7krs8lrps2rv7, O_RDWR) failed: Permission denied (13) in /usr/share/cacti-0.8.8h/include/global.php on line 157
PHP Warning: Unknown: open(/var/lib/php/session/sess_pb23ujpm1vkfa7krs8lrps2rv7, O_RDWR) failed: Permission denied (13) in Unknown on line 0
PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session) in Unknown on line 0

Or, instead of fixing all that, run the spine debug run as root. But remember, use the latest spine with 1.2.x. But if that was your 0.8.x release, of course the latest spine is not compatible.

[TheWitness]

When using the lastest spine, you should receive output like the following:

Code: Select all

[root@vmhost3 bin]# ./spine -R -S -V 3 -f 149 -l 149
SPINE: Using spine config file [../etc/spine.conf]
Version 1.2.20 starting
Total[0.3547] Spine will support multithread device polling.
Total[0.3549] DEBUG: Initial Value of Available Threads is 4 (0 outstanding)
Total[0.3551] DEBUG: Available Threads is 3 (1 outstanding)
Total[0.3553] DEBUG: Available Threads is 2 (2 outstanding)
Total[0.3555] Device[0] HT[1] Total Time: 0.00033 Seconds
Total[0.3556] Device[149] INFO: SNMP Device '192.168.11.3:161' has a timeout of 500000 (500), with 0 retries
Total[0.3559] Device[149] Checking for System Information Update
Total[0.3560] Device[149] Updating Full System Information Table
Total[0.3588] Device[149] HT[1] DQ[4] Extended Uptime Result: No Such Object available on this agent at this OID, Is Numeric: 0
Total[0.3591] Device[149] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 127878592 < output: 127881352)
Total[0.3595] Device[149] HT[1] NOTE: There are '14' Polling Items for this Device
Total[0.3596] Device[149] INFO: SNMP Device '192.168.11.3:161' has a timeout of 500000 (500), with 0 retries
Total[0.3619] Device[149] HT[1] DS[2141] TT[2.31] SNMP: v2: 192.168.11.3, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.1, value: 0
Total[0.3619] Device[149] HT[1] DS[2141] TT[2.33] SNMP: v2: 192.168.11.3, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.1, value: 0
Total[0.3619] Device[149] HT[1] DS[2141] TT[2.33] SNMP: v2: 192.168.11.3, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.1, value: 0
Total[0.3619] Device[149] HT[1] DS[2141] TT[2.34] SNMP: v2: 192.168.11.3, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.1, value: 0
Total[0.3619] Device[149] HT[1] DS[2142] TT[2.34] SNMP: v2: 192.168.11.3, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.2, value: 0
Total[0.3619] Device[149] HT[1] DS[2142] TT[2.34] SNMP: v2: 192.168.11.3, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.2, value: 0
Total[0.3619] Device[149] HT[1] DS[2142] TT[2.35] SNMP: v2: 192.168.11.3, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.2, value: 0
Total[0.3619] Device[149] HT[1] DS[2142] TT[2.35] SNMP: v2: 192.168.11.3, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.2, value: 0
Total[0.3619] Device[149] HT[1] DS[2143] TT[2.36] SNMP: v2: 192.168.11.3, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.1, value: 195987825670
Total[0.3619] Device[149] HT[1] DS[2143] TT[2.37] SNMP: v2: 192.168.11.3, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.1, value: 71199501259
Total[0.3624] Device[149] HT[1] DS[2145] TT[0.49] SS[0] SERVER: /var/www/html/cacti/scripts/ss_hstats.php ss_hstats '149' polling_time, output: 0.109
Total[0.3628] Device[149] HT[1] DS[2146] TT[0.38] SS[1] SERVER: /var/www/html/cacti/scripts/ss_hstats.php ss_hstats '149' uptime, output: 127878592
Total[0.3639] Device[149] HT[1] DS[2144] TT[1.49] SNMP: v2: 192.168.11.3, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.2, value: 67810681272
Total[0.3640] Device[149] HT[1] DS[2144] TT[1.50] SNMP: v2: 192.168.11.3, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, value: 194518442381
Total[0.3641] Device[149] HT[1] Total Time: 0.0088 Seconds
Total[1.0559] Time: 1.0558 s, Threads: 4, Devices: 2

The TT[x.xx] is the time in milli/micro seconds (I can not remember which) to get a response. Very helpful for finding bad OID's. Keep in mind that if your MAX OID's is high, and in those MAX OID's there are bad OID's, that's going to increase the time it takes to poll a host. Testing with MAX OID's at 1 will provide a better view of the time to query each OID.