System to slow for Number of Datasources

[opm]

Hello,

I am planning to Monitor arround 200 000 Datasources or more are there posibilities to tune my system?

What about the number of parallel threads? Why is it limited to 30? I have a Hyperthreading CPU which might work better with 50 or more threads. Is it a problem of Linux or of Cactid?

What makes the poller slow? Network connectivity? RAM? Kernel? CPU?

My System: IP4 3,4G HyperThreading / 2G RAM / GBit Ethernet
Suse Linux 9.3 / Kernel 2.6.11.4-21.9-smp
needs 75s for 15000 Datasources so it might be impossible to monitor 200 000 within 300s

Has anyone experience with the performance of Cacti? Can you tell numbers or show Graphs?

Thanks alot

[rony]

Threads are commonly set to 2 times the amount of Physical CPU's you have. Setting it higher can accually hurt your performance.

Are all these device snmp? How many are scripts?

[opm]

the most of the devices are snmp, scripts are under 2%. I made a performance test and a forcast and during the first 15000 Datasources this was correct an it tells an increasement to 30 threads brings a real benefit. I will post this forcast tomorrow.

[rony]

I will let TheWitness chime in on this. He knows the polling parameters better than I do, since he wrote it.

[rkramer]

I've seen benefits of using thread count of up to 10x (or more) the # of processors, especially if you are just doing snmp. most snmp equipment responds REAAAALLY slowly, at least in comparison to processor (and network) speeds. most of the time is spent waiting for a response, so having extra threads allows you to wait for more at a time.

[TheWitness]

The new version of Cactid can use 100 threads and also can uptain upto 50 OID's in a single get request. This should help you out. Remember that it's one size fits all. At some point, you will start getting errors.

The big bottleneck in Cacti is likely going to be disk performance. With lot's of memory and a good OS, with Write caching enabled, you should be able to really tourch the box.

We are looking into a new architecture in 0.9 to increase poller performance, but we've been distracted with some meaningful changes to the user interface and database structure.

TheWitness