SPINE Poller problem

[Eliran]

Hi all,
I having a big problem when I using SPINE,
I have a big environment that now our Cacti only looks at ~50% of it.
the poller have a lot of problems and it takes a lot of time to complete his cycle, about 600 sec.
What I am doing wrong?

My cacti info
---------------
Technical Support
General Information
Date Sun, 28 Aug 2011 12:50:33 +0300
Cacti Version 0.8.7g
Cacti OS win32
SNMP Version NET-SNMP version: 5.5
RRDTool Version RRDTool 1.2.x
Hosts 236
Graphs 4090

Data Sources
Script/Command: 5
SNMP: 2762
SNMP Query: 1719
Script - Script Server (PHP): 14
Script Query - Script Server: 1681
Total: 6181


Poller Information
Interval 300
Type spine
Items Action[0]: 11265
Action[2]: 2233
Total: 13498
Concurrent Processes 5
Max Threads 70
PHP Servers 10
Script Timeout 25
Max OID 10
Last Run Statistics Time:490.3842 Method:spine Processes:4 Threads:50 Hosts:199 HostsPerProcess:50 DataSources:13498 RRDsProcessed:4824

PHP Information
PHP Version 5.2.14
PHP OS WINNT
PHP uname Windows NT CACTI 6.1 build 7601
PHP SNMP Installed
max_execution_time 300
memory_limit 512M

I am using HP DL160 G5
OS: Windows 2008 R2 x64
Processor: E5405 @ 2.00GHz, 1995 Mhz, 4 Core(s), 4 Logical Processor(s)
Ram: 2 GB

Please help me...

[Eliran]

After checking more on-line and in the logs I still don't know how to fix this problem.
I am adding my log, maybe it will help you to help me.

Log.txt

(1000.67 KiB) Downloaded 200 times

[erik.costa]

Try to change the poller settings. Sometimes increasing "Maximum Threads per Process" too high do a reverse effect (more time to process)

Try to put:
Maximum Concurrent Poller Processes: 10
Maximum Threads per Process: 10
Number of PHP Script Servers: 10
and
The Maximum SNMP OID's Per SNMP Get Request: 10

For me its a good start!

#######################

In time ...

I see your log file...
Why too many timeouts ?? You can disable this time-outed hosts to see if have some difference?

[Eliran]

erik thanks for your replay.

I will try to do it and report back.

[Eliran]

As I promised I am reporting back with the results.

Now all is working good, what I did:

1. SNMP time out for windows servers 5000 ms
2. Processes 8
3. Threads 50
4. PHP Script Servers 10
5. Maximum SNMP OID's Per SNMP Get Request 30
6. Find out Servers that have problem with their SNMP, I had only 1 server and it need to be fix.

Here is my stats,

09/05/2011 07:56:52 PM - SYSTEM STATS: Time:110.8015 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:50:52 PM - SYSTEM STATS: Time:50.9215 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:46:22 PM - SYSTEM STATS: Time:80.7622 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:40:50 PM - SYSTEM STATS: Time:48.8322 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:35:56 PM - SYSTEM STATS: Time:55.0966 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:31:05 PM - SYSTEM STATS: Time:63.9222 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:25:55 PM - SYSTEM STATS: Time:54.2389 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:20:50 PM - SYSTEM STATS: Time:48.8008 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:15:49 PM - SYSTEM STATS: Time:47.9728 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:10:50 PM - SYSTEM STATS: Time:49.4578 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:05:59 PM - SYSTEM STATS: Time:58.3746 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 07:01:18 PM - SYSTEM STATS: Time:76.8205 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 06:56:42 PM - SYSTEM STATS: Time:101.0244 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 06:50:50 PM - SYSTEM STATS: Time:48.7367 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 06:45:59 PM - SYSTEM STATS: Time:58.5024 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 06:41:21 PM - SYSTEM STATS: Time:80.4342 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 06:36:03 PM - SYSTEM STATS: Time:62.2420 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 06:31:19 PM - SYSTEM STATS: Time:77.9071 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386
09/05/2011 06:25:50 PM - SYSTEM STATS: Time:49.3450 Method:spine Processes:8 Threads:50 Hosts:52 HostsPerProcess:7 DataSources:5137 RRDsProcessed:2386

Thanks for all the help,

Eliran

[erik.costa]

Good job!