Graphs stop updating

[fpiorek]

Hello,

I've been running into a situation with my devices that were defined years ago that the graphs stop graphing.

I'm running CACTI version 1.2.17

I've noticed on the devices list that the ones in question show In State status of approximately 1051 Days

See attached screen shot.

cacti_device.PNG (7.1 KiB) Viewed 3161 times

My work around has been to delete the device, data sources and graphs. Then I recreate them.

It is not an issue with SNMP on the device because after recreating them those graphs populate normally.

Any ideas what the issue is? Is there some limit as to the length of time a device can be In State?

Any help will be appreciated.

[TheWitness]

Not without more info.

[fpiorek]

What additional information would you need to see?

[TheWitness]

Edit the device and capture the top 1/3 of the screen. Paste that image here. Then, there should be a link to "Repopulate Poller Cache" to the upper right. Click that. Lastly check the devices RRDfiles and make sure they are properly owned by the Web Server and are writable by the web server.

[fpiorek]

Hello,

See screen shot for the top 1/3 of the device details.

Cacti.PNG (33.16 KiB) Viewed 3141 times

I did click the Repopulate Poller Cache and it did not make any difference.

I checked the ownership of the RRDs and it shows SYSTEM as the owner. I'm running CACTI on a WINDOWS 2016 server.

All RRDs are showing the same ownership. Even the working graphs and devices.

I've noticed some additional graphs for devices showing instate for over 1000+ days are now blank like this one

Cacti2.PNG (30.61 KiB) Viewed 3141 times

Let me know what else you need.

Thanks for your assistance.

[TheWitness]

Turn on device debug for a few of these. What graphs are blank? Are the poller items populated?

[TheWitness]

Lastly check for Orphans.

[fpiorek]

Hi,

Did turn on device debug, Where should I see the output?

It is all graphs for that device.

The poller items under View poller cache in System Utilities do show the mib value and the RRD file that will be updated.

To check for orphans I'm guessing you mean RRD files? There are none but I do see that the RRD files have stopped being updated.

Thanks

[TheWitness]

Device Debug will log entries into the Cacti log of everything polled, why it's considered up/down, etc.

Orphans shows if the graphs are down due to errors.

[fpiorek]

Thank you for the information.

With Device Debug enabled for these devices I do not see any information in my log for them.

I also changed my log level to Debug yesterday for a couple of poll cycles.

Got lots of data but did not see any information for these devices.

FYI, Had a couple of additional devices stop updating their graph information yesterday. The In State column shows over 1000+ days

Cacti3.PNG (2.19 KiB) Viewed 3126 times

Sorry to keep harping on that field but it is the only thing that seems to be a common to this issue.

What do we look at next?

Thank you or your assistance.

[TheWitness]

Note the device id, and then goto the command line and run:

Code: Select all

./spine -R -V 5 -S -f device_id -d device_id

Post the output. Run as the same account in cron or systemd. Oh wait Windoz. Wrong OS. But whatever.

[fpiorek]

Hi,

Thanks for the quick response.

I'm not running spine still using CMD

What would be the command format for that?

[TheWitness]

The example below is with the special device id 0.

Code: Select all

[root@vmhost3 cacti]# php -q cmd.php --help
Cacti Legacy Host Data Collector, Version 1.2.22, Copyright (C) 2004-2022 The Cacti Group

usage: cmd.php --first=ID --last=ID [--poller=ID] [--mibs] [--debug]

Cacti's legacy data collector.  This data collector is called by poller.php
every poller interval to gather information from devices.  It is recommended
that every system deploy spine instead of cmd.php in production due to the built
in scalability limits of cmd.php.

Required
    --first  - First host id in the range to collect from.
    --last   - Last host id in the range to collect from.

Optional:
    --poller - The poller to run as.  Defaults to the system poller.
    --mode   - The poller mode, either online, offline, or recovery.
    --mibs   - Refresh all system mibs from hosts supporting snmp.
    --debug  - Display verbose output during execution.

[root@vmhost3 cacti]# php -q cmd.php --first=0 --last=0 --debug
Total[0.0294] NOTE: There are no items in your poller for this polling cycle!
[root@vmhost3 cacti]#

[TheWitness]

Example here:

Code: Select all

[root@vmhost3 cacti]# php -q -d display_errors=On cmd.php --first=0 --last=58 --debug
Total[0.2047] Device[58] STATUS: Device '192.168.11.200' is UP.
Total[0.2050] Device[58] RECACHE: Processing 4 items in the auto reindex cache for '192.168.11.200'.
Total[0.2685] Device[58] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert:8954600 < output:8956600)
Total[0.3337] Device[58] DQ[7] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert:8954600 < output:8956600)
Total[0.3816] Device[58] DQ[8] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert:8954600 < output:8956600)
Total[0.4243] Device[58] DQ[9] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert:8954600 < output:8956600)
Total[0.4252] Device[58] DS[126] TT[0.52] SERVER: /var/www/html/cacti/scripts/ss_hstats.php ss_hstats '58' polling_time, output: 0.494
Total[0.4257] Device[58] DS[168] TT[0.17] SERVER: /var/www/html/cacti/scripts/ss_hstats.php ss_hstats '58' uptime, output: 8954600
Total[0.4631] Device[58] DS[255] TT[37.32] SNMP: v3: 192.168.11.200, dsname: load_1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, output: 0.16
Total[0.4994] Device[58] DS[274] TT[36.16] SNMP: v3: 192.168.11.200, dsname: load_15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, output: 0.31
Total[0.5365] Device[58] DS[288] TT[37.03] SNMP: v3: 192.168.11.200, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, output: 0.27
Total[0.5891] Device[58] DS[406] TT[52.47] SNMP: v3: 192.168.11.200, dsname: mem_buffers, oid: .1.3.6.1.4.1.2021.4.14.0, output: 2260
Total[0.6368] Device[58] DS[426] TT[47.53] SNMP: v3: 192.168.11.200, dsname: mem_cache, oid: .1.3.6.1.4.1.2021.4.15.0, output: 3241128
Total[0.6814] Device[58] DS[443] TT[44.41] SNMP: v3: 192.168.11.200, dsname: mem_free, oid: .1.3.6.1.4.1.2021.4.6.0, output: 4010816
Total[0.7511] Device[58] DS[458] TT[69.42] SNMP: v3: 192.168.11.200, dsname: mem_total, oid: .1.3.6.1.4.1.2021.4.5.0, output: 16209248
Total[0.8073] Device[58] DS[556] TT[55.92] SNMP: v3: 192.168.11.200, dsname: users, oid: .1.3.6.1.2.1.25.1.5.0, output: 0
Total[0.8781] Device[58] DS[599] TT[70.56] SNMP: v3: 192.168.11.200, dsname: proc, oid: .1.3.6.1.2.1.25.1.6.0, output: 234
Total[0.9269] Device[58] DS[690] TT[48.59] SNMP: v3: 192.168.11.200, dsname: ssCpuIdle, oid: .1.3.6.1.4.1.2021.11.11.0, output: 92
Total[0.9989] Device[58] DS[702] TT[71.84] SNMP: v3: 192.168.11.200, dsname: ssCpuSystem, oid: .1.3.6.1.4.1.2021.11.10.0, output: 1
Total[1.0554] Device[58] DS[713] TT[56.18] SNMP: v3: 192.168.11.200, dsname: ssCpuUser, oid: .1.3.6.1.4.1.2021.11.9.0, output: 4
Total[1.1038] Device[58] DS[782] TT[48.16] SNMP: v3: 192.168.11.200, dsname: ssSysInterrupts, oid: .1.3.6.1.4.1.2021.11.7.0, output: 2414
Total[1.1471] Device[58] DS[820] TT[42.99] SNMP: v3: 192.168.11.200, dsname: ssSysContext, oid: .1.3.6.1.4.1.2021.11.8.0, output: 3248
Total[1.3977] Device[58] DS[858] TT[250.38] SERVER: /var/www/html/cacti/scripts/ss_net_snmp_disk_io.php ss_net_snmp_disk_io '192.168.11.200', output: reads:0 writes:371
Total[1.6632] Device[58] DS[888] TT[265.36] SERVER: /var/www/html/cacti/scripts/ss_net_snmp_disk_bytes.php ss_net_snmp_disk_bytes '192.168.11.200', output: bytesread:0 byteswritten:1601536
Total[1.7313] Device[58] DS[2005] TT[67.86] SNMP: v3: 192.168.11.200, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.3, output: 37008347345
Total[1.7855] Device[58] DS[2005] TT[53.95] SNMP: v3: 192.168.11.200, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.3, output: 19104608835
Total[1.8323] Device[58] DS[2083] TT[46.54] SNMP: v3: 192.168.11.200, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.3, output: 0
Total[1.8748] Device[58] DS[2083] TT[42.22] SNMP: v3: 192.168.11.200, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.3, output: 0
Total[1.9142] Device[58] DS[2083] TT[39.24] SNMP: v3: 192.168.11.200, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.3, output: 0
Total[1.9510] Device[58] DS[2083] TT[36.56] SNMP: v3: 192.168.11.200, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.3, output: 0
Total[1.9511] Device[58] Writing 24 items to Poller Output Table
Total[1.9537] Device[58] Time[1.79] Items[24] Errors[0]
Total[1.9599] Time: 1.9311 s, Poller: 1, Threads: N/A, Devices: 1, Items: 24, Errors: 0

[fpiorek]

Got these results from that command.

C:\inetpub\wwwroot\cacti>php -q -d display_errors=On cmd.php --first=17 --last=17 --debug
2022/07/22 09:36:51 - POLLER: Poller[1] PID[16520] Device[17] STATUS: Device 'cobldntp01-mgt.xxx.com' is UP.
2022/07/22 09:36:51 - POLLER: Poller[1] PID[16520] Device[17] DS[857] TT[126.42] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.4, output: 1925784214
2022/07/22 09:36:51 - POLLER: Poller[1] PID[16520] Device[17] DS[857] TT[88.89] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.4, output: 1144834284
2022/07/22 09:36:51 - POLLER: Poller[1] PID[16520] Device[17] DS[858] TT[97.48] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.5, output: 3784031731
2022/07/22 09:36:51 - POLLER: Poller[1] PID[16520] Device[17] DS[858] TT[105.77] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.5, output: 3400567917
2022/07/22 09:36:51 - POLLER: Poller[1] PID[16520] Device[17] DS[859] TT[97.83] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.4, output: 0
2022/07/22 09:36:51 - POLLER: Poller[1] PID[16520] Device[17] DS[859] TT[88.54] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.4, output: 0
2022/07/22 09:36:52 - POLLER: Poller[1] PID[16520] Device[17] DS[859] TT[105.09] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.4, output: 0
2022/07/22 09:36:52 - POLLER: Poller[1] PID[16520] Device[17] DS[859] TT[106.42] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.4, output: 0
2022/07/22 09:36:52 - POLLER: Poller[1] PID[16520] Device[17] DS[860] TT[106.34] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: discards_in, oid: .1.3.6.1.2.1.2.2.1.13.5, output: 0
2022/07/22 09:36:52 - POLLER: Poller[1] PID[16520] Device[17] DS[860] TT[97.03] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: discards_out, oid: .1.3.6.1.2.1.2.2.1.19.5, output: 0
2022/07/22 09:36:52 - POLLER: Poller[1] PID[16520] Device[17] DS[860] TT[90.25] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: errors_in, oid: .1.3.6.1.2.1.2.2.1.14.5, output: 0
2022/07/22 09:36:52 - POLLER: Poller[1] PID[16520] Device[17] DS[860] TT[97.65] SNMP: v3: cobldntp01-mgt.xxx.com, dsname: errors_out, oid: .1.3.6.1.2.1.2.2.1.20.5, output: 0
2022/07/22 09:36:52 - POLLER: Poller[1] PID[16520] Device[17] Time[1.36] Items[12] Errors[0]
2022/07/22 09:36:52 - POLLER: Poller[1] PID[16520] Time: 1.4006 s, Poller: 1, Threads: N/A, Devices: 1, Items: 12, Errors: 0

Yet this is what the graph shows:

Cacti4.PNG (11.94 KiB) Viewed 3107 times

What do we look at next?

Thanks again for chasing this down.