Graph Not Working for SNMPv3 Devices

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Developers, Moderators

Post Reply
hakiki
Posts: 16
Joined: Sun Dec 12, 2021 8:44 pm

Graph Not Working for SNMPv3 Devices

Post by hakiki »

Hi, I installed cacti 1.2.24 then imported a database dump from another system with older cacti version (0.8.8h) to retain some configuration and devices.

Overall, this system worked with existing graphs and devices (all are using SNMPv2). But then I added a new device with SNMPv3, the graphs are not working as follow:

Image

I tried snmpwalk the device and it returned the results. So I think its not an SNMP issue with the device.

Also I run spine command on this specific device and the results are as follows:

Code: Select all

[root@Tcacti bin]# spine -R -S -V 3 -H 4211 -L 5
SPINE: Using spine config file [../etc/spine.conf]
Version 1.2.24 starting
Total[0.0118] Spine will support multithread device polling.
Total[0.0120] DEBUG: Initial Value of Available Threads is 30 (0 outstanding)
Total[0.0122] DEBUG: Available Threads is 29 (1 outstanding)
Total[0.0123] DEBUG: Available Threads is 28 (2 outstanding)
Total[0.0125] Device[0] HT[1] Total Time: 0.00028 Seconds
Total[0.0127] Device[4211] INFO: SNMP Device '172.30.193.138:161' has a timeout of 2000000 (2000), with 3 retries
Total[0.0423] Device[4211] HT[1] DQ[14] Legacy Uptime Result: U, Is Numeric: 0
Total[0.0424] Device[4211] HT[1] DQ[14] Extended Uptime Result: , Is Numeric: 0
Total[0.0424] Device[4211] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 3710972550 < output: )
Total[0.0424] Device[4211] HT[1] DQ[14] RECACHE ASSERT FAILED: '3710972550<'
Total[0.0424] Device[4211] HT[1] NOTICE: Spike Kill in Effect for '172.30.193.138'
Total[0.0428] Device[4211] HT[1] NOTE: There are '4' Polling Items for this Device
Total[0.0429] Device[4211] HT[1] Total Time: 0.03 Seconds
Total[0.5303] Time: 0.5302 s, Threads: 30, Devices: 2
Below are details of my cacti system:

Code: Select all

Date	Tue, 01 Aug 2023 20:00:38 +0700
Cacti Version	1.2.24
Cacti OS	unix
RSA Fingerprint	0c:cf:6e:11:ff:54:63:0f:2e:b2:d4:f3:d7:df:4c:16
NET-SNMP Version	NET-SNMP version: 5.8
RRDtool Version Configured	1.7.0+
RRDtool Version Found	1.7.0
Devices	1,551
Graphs	73
Data Sources	Script/Command: 2
SNMP Query: 53
Script Server: 4
Total: 59

Interval	300
Type	SPINE 1.2.24 Copyright 2004-2023 by The Cacti Group
Items	Action[0]: 116
Action[1]: 2
Action[2]: 4
Total: 122
Concurrent Processes	Name: Main Poller, Procs: 40
Max Threads	Name: Main Poller, Threads: 30
PHP Servers	7
Minimum Connections:	Main Server: Current: 2096, Min Required: 1580
Assumes 100 spare connections for Web page users and other various connections.
The minimum required can vary greatly if there is heavy user Graph viewing activity.
Each browser tab can use upto 10 connections depending on the browser.
Script Timeout	60
Max OID	50
Last Run Statistics	Time:21.5347 Method:spine Processes:40 Threads:30 Hosts:1548 HostsPerProcess:39 DataSources:122 RRDsProcessed:19

PHP Version	7.2.24
PHP OS	Linux
PHP uname	Linux Tcacti 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Fri Jun 2 08:27:19 EDT 2023 x86_64
PHP SNMP	Not Installed
max_execution_time	1200
memory_limit	8192M
Attachments
20230801_195619_001.png
20230801_195619_001.png (148.64 KiB) Viewed 1770 times
Top
User avatar
macan
Cacti Guru User
Posts: 1106
Joined: Tue Mar 18, 2008 2:30 am
Location: Czech

Re: Graph Not Working for SNMPv3 Devices

Post by macan »

right upgrade process is described here:
https://docs.cacti.net/Upgrading-Cacti.md

Is your poller running periodically?
Let the Cacti grow!
Top
hakiki
Posts: 16
Joined: Sun Dec 12, 2021 8:44 pm

Re: Graph Not Working for SNMPv3 Devices

Post by hakiki »

I have the poller running set for every 5 minutes in /etc/cron.d/cacti

Code: Select all

[root@Tcacti ~]# cat /etc/cron.d/cacti 
*/5 * * * * apache apache /usr/share/cacti/poller.php > /dev/null 2>&1
Also cactid.service is running

Code: Select all

[root@Tcacti ~]# systemctl status cactid.service 
● cactid.service - Cacti Daemon Main Poller Service
   Loaded: loaded (/etc/systemd/system/cactid.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-08-02 06:17:09 WIB; 2s ago
  Process: 1161127 ExecStart=/usr/share/cacti/cactid.php (code=exited, status=0/SUCCESS)
 Main PID: 1161131 (php)
    Tasks: 1 (limit: 152772)
   Memory: 14.1M
   CGroup: /system.slice/cactid.service
           └─1161131 php /usr/share/cacti/cactid.php

Aug 02 06:17:09 Tcacti systemd[1]: Starting Cacti Daemon Main Poller Service...
Aug 02 06:17:09 Tcacti cactid.php[1161127]: Starting Cacti Daemon ... [OK]
Aug 02 06:17:09 Tcacti systemd[1]: Started Cacti Daemon Main Poller Service.
Other devices that are in SNMPv2 are still updating their graphs

Below are debug log for this device
Attachments
Screenshot 2023-08-02 064655.png
Screenshot 2023-08-02 064655.png (168.75 KiB) Viewed 1731 times
Top
User avatar
macan
Cacti Guru User
Posts: 1106
Joined: Tue Mar 18, 2008 2:30 am
Location: Czech

Re: Graph Not Working for SNMPv3 Devices

Post by macan »

chmod +s /path/to/bin/spine - it allowes spine do icmp ping

Try to deinstall php-snmp module
Let the Cacti grow!
Top
hakiki
Posts: 16
Joined: Sun Dec 12, 2021 8:44 pm

Re: Graph Not Working for SNMPv3 Devices

Post by hakiki »

macan wrote: ↑Wed Aug 02, 2023 7:14 am chmod +s /path/to/bin/spine - it allowes spine do icmp ping

Try to deinstall php-snmp module
Hi, macan. Do you mean to deinstall net-snmp module?
Because currently I don't have php-snmp installed on my machine (previously the php-snmp was installed but then I already removed it prior posting on this forum)


Anyway, I noticed that there are 2 versions of rrdtool installed on my machine (1.7.0 and 1.5.3) but cacti recognized 1.5.3 one. I removed it and now cacti uses 1.7.0 version. I run spine command once again and it give better result than previous on my 1st post:

Code: Select all

SPINE: Using spine config file [../etc/spine.conf]
Version 1.2.24 starting
Total[0.0133] Spine will support multithread device polling.
Total[0.0136] DEBUG: Initial Value of Available Threads is 1 (0 outstanding)
Total[0.0140] DEBUG: Available Threads is 0 (1 outstanding)
Total[0.0144] Device[0] HT[1] Total Time: 0.00039 Seconds
Total[0.0243] DEBUG: Available Threads is 0 (1 outstanding)
Total[0.0244] DEBUG: Device[3] HT[1] In Poller, About to Start Polling
Total[0.0251] Device[3] INFO: SNMP Device '172.30.193.138:161' has a timeout of 500000 (500), with 3 retries
Total[0.0412] Device[3] DEBUG: Entering ICMP Ping
Total[0.0414] Device[3] DEBUG: Attempting to ping 172.30.193.138, seq 0 (Retry 0 of 1)
Total[0.0474] Device[3] DEBUG: ICMP Device Alive, Try Count:1, Time:6.0620 ms
Total[0.0478] DEBUG: Device[3] HT[1] RECACHE: Processing 1 items in the auto reindex cache for '172.30.193.138'
Total[0.0608] Device[3] HT[1] DQ[4] Legacy Uptime Result: 3757936797, Is Numeric: 1
Total[0.0669] Device[3] HT[1] DQ[4] Extended Uptime Result: 3757936797, Is Numeric: 1
Total[0.0669] Device[3] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 3753514880 < output: 3757936797)
Total[0.0680] Device[3] HT[1] NOTE: There are '32' Polling Items for this Device
Total[0.0681] Device[3] INFO: SNMP Device '172.30.193.138:161' has a timeout of 500000 (500), with 3 retries
Total[0.1060] Device[3] HT[1] DS[28] TT[17.64] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.149, value: 8561830
Total[0.1061] Device[3] HT[1] DS[28] TT[17.68] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.149, value: 1828019994668515
Total[0.1061] Device[3] HT[1] DS[29] TT[17.69] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.150, value: 7990191
Total[0.1061] Device[3] HT[1] DS[29] TT[17.70] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.150, value: 1585258297373498
Total[0.1061] Device[3] HT[1] DS[30] TT[17.72] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.151, value: 0
Total[0.1061] Device[3] HT[1] DS[30] TT[17.73] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.151, value: 0
Total[0.1061] Device[3] HT[1] DS[31] TT[17.74] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.152, value: 9329210
Total[0.1062] Device[3] HT[1] DS[31] TT[17.75] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.152, value: 1497483157723371
Total[0.1062] Device[3] HT[1] DS[32] TT[17.76] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.153, value: 0
Total[0.1062] Device[3] HT[1] DS[32] TT[17.79] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.153, value: 0
Total[0.1131] Device[3] HT[1] DS[33] TT[6.84] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.154, value: 10287691
Total[0.1131] Device[3] HT[1] DS[33] TT[6.87] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.154, value: 1438014175467856
Total[0.1131] Device[3] HT[1] DS[34] TT[6.88] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.155, value: 11104349
Total[0.1131] Device[3] HT[1] DS[34] TT[6.89] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.155, value: 1248861720705585
Total[0.1131] Device[3] HT[1] DS[35] TT[6.91] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.156, value: 7025793
Total[0.1131] Device[3] HT[1] DS[35] TT[6.93] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.156, value: 1015367599551437
Total[0.1131] Device[3] HT[1] DS[36] TT[6.94] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.157, value: 7968563
Total[0.1132] Device[3] HT[1] DS[36] TT[6.95] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.157, value: 1203572135167920
Total[0.1132] Device[3] HT[1] DS[37] TT[6.97] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.158, value: 8426331
Total[0.1132] Device[3] HT[1] DS[37] TT[6.99] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.158, value: 921090940599731
Total[0.1194] Device[3] HT[1] DS[38] TT[6.14] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.159, value: 7913274
Total[0.1194] Device[3] HT[1] DS[38] TT[6.17] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.159, value: 1261866229373223
Total[0.1194] Device[3] HT[1] DS[39] TT[6.18] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.160, value: 6510535
Total[0.1194] Device[3] HT[1] DS[39] TT[6.20] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.160, value: 807161166062817
Total[0.1194] Device[3] HT[1] DS[40] TT[6.22] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.161, value: 7589001
Total[0.1195] Device[3] HT[1] DS[40] TT[6.23] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.161, value: 807552929798175
Total[0.1195] Device[3] HT[1] DS[41] TT[6.24] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.162, value: 7255966
Total[0.1195] Device[3] HT[1] DS[41] TT[6.25] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.162, value: 708193669068160
Total[0.1195] Device[3] HT[1] DS[42] TT[6.27] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.163, value: 6333664
Total[0.1195] Device[3] HT[1] DS[42] TT[6.30] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.163, value: 468130721656929
Total[0.1252] Device[3] HT[1] DS[43] TT[5.66] SNMP: v3: 172.30.193.138, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.164, value: 6988621
Total[0.1252] Device[3] HT[1] DS[43] TT[5.69] SNMP: v3: 172.30.193.138, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.164, value: 1016838750956796
Total[0.1254] Device[3] HT[1] Total Time: 0.1 Seconds
Total[0.1260] DEBUG: Device[3] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
Total[0.5480] Time: 0.5478 s, Threads: 1, Devices: 2

But graph still not updating..
Attachments
20230803_004511_001.png
20230803_004511_001.png (36.47 KiB) Viewed 1571 times
20230803_004530_001.png
20230803_004530_001.png (63.16 KiB) Viewed 1571 times
Top
User avatar
macan
Cacti Guru User
Posts: 1106
Joined: Tue Mar 18, 2008 2:30 am
Location: Czech

Re: Graph Not Working for SNMPv3 Devices

Post by macan »

console -> Troubleshooting -> datasources - try debug here
Let the Cacti grow!
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Re: Graph Not Working for SNMPv3 Devices

Post by TheWitness »

Also, check permissions on the RRA folder. This is something that people oftentimes miss.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
hakiki
Posts: 16
Joined: Sun Dec 12, 2021 8:44 pm

Re: Graph Not Working for SNMPv3 Devices

Post by hakiki »

macan wrote: ↑Mon Aug 07, 2023 3:43 am console -> Troubleshooting -> datasources - try debug here
This is the result for data source debug (the second one). Seems data source was not polled, but I check the data source already exist in poller items. I tried rebuild poller cache from cli but it didn't help as well.
Screenshot 2023-08-10 114900.png
Screenshot 2023-08-10 114900.png (118.85 KiB) Viewed 1506 times
TheWitness wrote: ↑Mon Aug 07, 2023 8:59 pm Also, check permissions on the RRA folder. This is something that people oftentimes miss.
I tried to give chmod 777 on the rra directory and issue still persist.
Top
User avatar
macan
Cacti Guru User
Posts: 1106
Joined: Tue Mar 18, 2008 2:30 am
Location: Czech

Re: Graph Not Working for SNMPv3 Devices

Post by macan »

try to delete problematic device, rebuild poller cache and try to add these again.
You can try snmpv2. I'm courious if the result will be the same
Let the Cacti grow!
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Re: Graph Not Working for SNMPv3 Devices

Post by TheWitness »

If you go to Console > Management > Data Sources and Check the Orphans option, is the Data Source marked as an orphan? If so, you should be able to edit it and fix the index, etc.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Re: Graph Not Working for SNMPv3 Devices

Post by TheWitness »

Note the Check box.
Attachments
OrphanedDataSource.png
OrphanedDataSource.png (19.2 KiB) Viewed 1477 times
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
Post Reply

Return to “Help: Linux/Unix Specific”

Who is online

Users browsing this forum: No registered users and 1 guest