Scalability

[olebowle]

Hi,

up until now we are using Cacti to monitor a moderate amount of devices (5k - 10k) of the same type and thus, have set-up a template for them. We are polling the OIDs externally via an asynchronous poller implemented in C and later parse and process its output using a Cacti script server script. This worked fine for us so far.

We are now trying to poll a medium 5 digit number of devices and are running into some issues even creating them in Cacti. We noticed it takes ever slightly more time to create a new device using add_device.php depending on the number of existing devices.

for i in $(seq 100000); do php -q add_device.php --description=cm-$i.example.com --external-id=$i --ip=cm-$i.example.com --template=7 --community=public --avail=none --version=2; done

As you can see in the diagram creating 1000 device takes approx. 55 minutes now (3.3 seconds / device). In the beginning it used to be (0.4 seconds / device). More troubling is that the time to create a new device increases linearly with the number of existing devices. It seems the bottleneck are some database queries during creation.

We tuned the MariaDB database parameters according to the suggestions in the wizard and nothing else is using the database right now. The cacti version is cacti-1.2.15-1.el7 (CentOS Linux release 7.9.2009).

1) Is this expected behavior?

2) How could this be resolved / is there a workaround?

3) How many devices can you practically monitor with Cacti also considering potential i/o load issues with the underlying rrdtool? (We tried playing with Cacti Boost)
https://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD
https://collectd.org/wiki/index.php/Ins ... ool_plugin

BR,
Ole

[olebowle]

Just to follow-up: We switched from RRDs to TimescaleDB (with continuous aggregates and data retention policies) and from Cacti to Grafana for most of your devices, where the graphs / metrcis to be monitored are always the same.

Devices, for which we wan't to monitor specific user-defined interfaces we remain using Cacti.