Gaps in SNMP graph

[TheWitness]

One other observation, you only have one script server configured, I'd do like 5-10 of them. Would speed up that Nokia device query. Console > Configuration > Settings > Poller.

[TheWitness]

Upload that Nokia script. I want to take a look at it.

[kribbas]

Ok updated the script servers to 5.

When I run the debug on another node I don't get those php errors and it starts giving outputs like the output you showed. So I'm wondering why that specific node is giving that php error, they are both using the same spine.conf.

For this other similar node it seems to be using another script, perhaps that is the cause?

SPINE: Version 0.8.8h starting
NOTE: Spine will support multithread device polling.
Host[0] TH[1] Total Time: 0.0017 Seconds
Host[815] TH[1] Recache DataQuery[1] OID: .1.3.6.1.2.1.1.3.0, output: 210970268 2
Host[815] TH[1] Recache DataQuery[38] OID: .1.3.6.1.2.1.1.3.0, output: 21097026 82
Host[815] TH[1] Recache DataQuery[40] OID: .1.3.6.1.2.1.1.3.0, output: 21097026 82
Host[815] TH[1] NOTE: There are '253' Polling Items for this Host
Host[815] TH[1] DS[52767] SS[0] SERVER: /usr/share/cacti-0.8.8h/scripts/ss_nokia_cmg.php ss_nokia_cmg_usage ::::::161:500 get tmnxCar dCpuResMonBusyCoreUtil_1sec 3, output: 1.24

[TheWitness]

The old Cacti would open a PHP session for the script server. It was not smart enough to determine that it was a CLI script and not create a session. By default, root and apache are the only ones that have access to the PHP session directory. So, that's the reason.

[kribbas]

Now the poller seems to have become off, getting a lot of snmp get timeout for several nodes

and
07/06/2022 04:55:33 PM - SPINE: Poller[0] ERROR: SS[0] PHP Script Server communications lost. Restarting PHP Script Server
07/06/2022 04:55:33 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
07/06/2022 04:55:33 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal

Is this a result from the ./spine -R -V 3 -S -f device_id -l device_id?

I also reverted back to 1 php script server but nothing helps, how can I restore it?

[TheWitness]

How many cores/threads is the poller? It's basically overloaded the database. If you run the following from the command line, you can see what is blocking:

Code: Select all

sudo mysql -e "show processlist" | grep -v Sleep

[kribbas]

12 threads
1 php script server
25 scripter server timeout
15 max oid per snmp get requests

This was working fine before, except for that one node, now a lot is getting timeout.

[TheWitness]

I meant the "Cacti server" not Cacti.

[kribbas]

[appadmin /]$ sudo mysql -e "show processlist" | grep -v Sleep
Id User Host db Command Time State Info
69245848 root localhost NULL Query 0 NULL show processlist

[kribbas]

should I run service mysqld restart? I have reverted all changes but stilll getting major error.

[TheWitness]

Nope, but answer the last question.

[kribbas]

Not sure I understand, where do I show the processlist? I printed the output in previous reply form command line.

[TheWitness]

This:

Code: Select all

cat /proc/cpuinfo | grep -i cpuid | wc -l

[kribbas]

I disabled some hosts and its looking better now. However several hosts are status "down" but they are in fact up and responding to snmpwalk from command line, any way to re-compile the spine to fetch again?

Can I do a service snmpd restart? to see if the snmp polling will start working again?

[kribbas]

output: 4