[SOLVED] Cacti 1.2.7 performance issues - VERY slow while browsing graphs

[adiviso82]

Hi all,

I've been playing with cacti for a few years now, I've installed it in versions before 1.0.0 (worked flawlessly) and many times after (always slow).
My setup involves updating rrd files with external scripts every 15 mins (independent of cacti) and using cacti only to display graphs.
I am using graph templates, host templates and then I will create devices based on the host templates. After this point, in the Trees section, I will just add each of the device (not individual graphs).

The problem: With a config of 1 host template with 1000 graphs and 4 devices based on it (so 4000 graphs) it takes about 40 seconds from clicking any device before the graphs are loaded. The final config should include 20.000+ graphs.

I've tried numerous times to install everything from scratch since I use virtual machines (Ubuntu 16, 18, cacti 1.1.38, 1.2.7 more recently). I've also tried NGINX instead of Apache, or MariaDB instead of MySQL or last days CentOS 7 instead of Ubuntu 18. Nothing works. The only way to have a fast cacti is to use version 0.8.8f and then everything is really fast. I remember when cacti 1.0.0 came out I did an upgrade and from the very beginning I started having this problem. I know that somehow, all these years, I must have some bad config that I keep making because I have seen another cacti 1.2.7 that worked fast. The only hint I've got is from the logs (1 minute waiting in that case), probably something related to "priming cache". The piece of code that generates this is in /var/www/html/cacti/lib/auth.php

The virtual machines were installed on different desktops/laptops with at least 2GB of RAM and SSD. The log output is from cacti 1.2.7. I don't know what else to mention, but any help is really appreciated.

Many thanks,
Adrian

################
#### CACTI LOG ####
################
2019/12/04 23:01:32 - WEBUI Obtaining 'Graph Template' cache
2019/12/04 23:01:32 - WEBUI Fetch InitRows:-1, Hash:1597d7f033a83a19f5b1a2356ce08ea0
2019/12/04 23:01:32 - WEBUI Found Valid 'Graph Template' priming cache
2019/12/04 23:01:33 - WEBUI Obtaining 'Graph Template' cache
2019/12/04 23:01:33 - WEBUI Fetch InitRows:-1, Hash:1597d7f033a83a19f5b1a2356ce08ea0
2019/12/04 23:01:33 - WEBUI Found Valid 'Graph Template' priming cache
2019/12/04 23:02:23 - WEBLOG CACTI2RRD: /bin/rrdtool info /var/www/html/cacti/rra/server1_capacitybusy_1829.rrd
2019/12/04 23:02:23 - WEBLOG CACTI2RRD: /bin/rrdtool graphv - --imgformat=PNG --start='1575410492' --end='1575496892' --pango-markup --title='MGW - server1 - capacityBusy' --slope-mode --base=1000 --height=250 --width=650 --alt-autoscale-max --lower-limit='0' COMMENT:"From 2019/12/03 23\:01\:32 To 2019/12/04 23\:01\:32\c" COMMENT:" \n" --border 1 --slope-mode --watermark 'Generated by Cacti®' DEF:a='/var/www/html/cacti/rra/server1_capacitybusy_1829.rrd':'capacityBusy':AVERAGE LINE1:a#008080FF:'capacityBusy' GPRINT:a:MIN:'MIN\:%8.2lf %s' GPRINT:a:MAX:'MAX\:%8.2lf %s' GPRINT:a:AVERAGE:'AVG\:%8.2lf %s' GPRINT:a:LAST:'LAST\:%8.2lf %s\n'
2019/12/04 23:02:23 - WEBLOG CACTI2RRD: /bin/rrdtool graph - --imgformat=PNG --start='1575410492' --end='1575496892' --pango-markup --title='MGW - server1 - capacityBusy' --slope-mode --base=1000 --height=250 --width=650 --alt-autoscale-max --lower-limit='0' COMMENT:"From 2019/12/03 23\:01\:32 To 2019/12/04 23\:01\:32\c" COMMENT:" \n" --border 1 --slope-mode --watermark 'Generated by Cacti®' DEF:a='/var/www/html/cacti/rra/server1_capacitybusy_1829.rrd':'capacityBusy':AVERAGE LINE1:a#008080FF:'capacityBusy' GPRINT:a:MIN:'MIN\:%8.2lf %s' GPRINT:a:MAX:'MAX\:%8.2lf %s' GPRINT:a:AVERAGE:'AVG\:%8.2lf %s' GPRINT:a:LAST:'LAST\:%8.2lf %s\n'
2019/12/04 23:02:24 - WEBLOG CACTI2RRD: /bin/rrdtool info /var/www/html/cacti/rra/server1_capacitydependencyf_2038.rrd


#############
#### MYSQL #### - as advised by cacti install
#############
max_heap_table_size = 1073741824
max_allowed_packet = 16777216
tmp_table_size = 134217728
join_buffer_size = 134217728
innodb_buffer_pool_size = 4294967296
innodb_doublewrite = ON
innodb_flush_log_at_timeout = 10
innodb_read_io_threads = 32
innodb_write_io_threads = 16
innodb_buffer_pool_instances=33
innodb_io_capacity=200
innodb_io_capacity_max=2000
collation-server=utf8mb4_unicode_ci
character-set-server=utf8mb4

[Osiris]

Well, you better learn how to root cause performance issues. My guess is that you have an I/O wait problem. Is the server on SSD, NVMe or Flash? If not, are you using boost? You should run 'vmstat 1' for a while and see where the blocking is coming. Also, there is a database command called 'show processlist' that will show you if it's the database. However, based upon experience, I would call this a classic, "I should use boost or get SSD/Flash" issue.

[adiviso82]

I also believed I have an I/O wait problem, but it cannot be, let me explain why. I am using 2 VM setups: a production server (Ubuntu 18, enterprise level VM with quad CPU x Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, 16GB RAM, SSD drives) and a test server (another VM on my desktop, Samsung SSD, 16GB RAM, i7 CPU).

The same cacti configuration (number of graphs, devices) was working perfectly in version 0.8.8h, but not with any of the 1+ versions, in this case 1.2.7.
In cacti 0.8.8h there was no boost function, so I don't find that relevant, as I want to compare apples to apples. As far as I know, the boost function controls how the RRDs are updated. In my case the RRDs are updated externally in less than 30 seconds then there is nothing happening until the next update in 5 minutes. The VMs don't have other functions to keep the CPU/HDD busy in any way. Moreover, in the test server sometimes the RRDs are not even created, so I am just checking purely the web servers' speed.

During tests I was running top command to see the cpu load and just before the graphs are displayed the mysql service goes to 70-80% for few seconds. But in between the click on the device and the time when the graphs appear the cacti logs don't show anything happening. I will post some printouts from vmstat (haven't tried it before but looks fine to me) and another printout from processlist in mysql (again load is almost 0).

I think there might be a bug in the source code for cacti, but I don't know how to approach this, my php skills are at best beginner level.

[adiviso82]

:~$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 3596 7195156 736568 7222856 0 0 26 335 1 8 10 8 82 0 0
0 0 3596 7195156 736568 7222856 0 0 0 0 109 166 0 0 100 0 0
0 0 3596 7195156 736568 7222856 0 0 0 0 114 175 0 0 100 0 0
0 0 3596 7195156 736568 7222856 0 0 0 0 89 142 0 0 100 0 0
0 0 3596 7195156 736568 7222856 0 0 0 0 99 152 0 0 100 0 0
0 0 3596 7195156 736568 7222856 0 0 0 0 122 183 0 0 100 0 0
0 0 3596 7195156 736568 7222856 0 0 0 0 105 160 0 0 100 0 0
0 0 3596 7190628 736568 7222856 0 0 0 0 12226 24143 8 5 86 0 0 CLICKED ON A DEVICE
0 0 3596 7190692 736568 7222856 0 0 0 0 185 223 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 122 214 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 158 1090 1 0 99 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 107 171 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 636 129 194 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 93 155 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 118 174 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 105 168 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 109 175 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 118 185 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 121 195 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 119 187 0 0 100 0 0
0 0 3596 7190692 736568 7222856 0 0 0 0 110 173 0 0 100 0 0
0 0 3596 7190468 736568 7222856 0 0 0 308 178 205 0 0 100 0 0
0 0 3596 7190660 736568 7222856 0 0 0 8 144 325 0 0 100 0 0
0 0 3596 7190660 736568 7222856 0 0 0 0 107 172 0 0 100 0 0
0 0 3596 7190660 736568 7222856 0 0 0 0 112 169 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 0 112 182 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 0 108 163 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 36 102 168 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 0 105 173 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 0 114 177 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 12 102 163 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 0 98 172 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 0 128 185 0 0 100 0 0
0 0 3596 7190660 736568 7222860 0 0 0 0 97 157 0 0 100 0 0
1 0 3596 7186648 736568 7222864 0 0 0 0 2357 6937 29 12 59 0 0 ABOUT HERE THE GRAPHS APPEARED (AFTER ~40 SECONDS)
0 0 3596 7194396 736568 7222868 0 0 0 0 809 1172 3 2 95 0 0
0 0 3596 7194396 736568 7222868 0 0 0 0 99 154 0 0 100 0 0
0 0 3596 7194396 736568 7222868 0 0 0 0 110 174 0 0 100 0 0
0 0 3596 7194396 736568 7222868 0 0 0 4 120 183 0 0 100 0 0
0 0 3596 7194396 736568 7222868 0 0 0 0 106 171 0 0 100 0 0
0 0 3596 7194396 736572 7222868 0 0 0 76 179 252 0 0 100 0 0
0 0 3596 7194396 736572 7222868 0 0 0 0 111 177 0 0 100 0 0
22 0 3596 7125896 736572 7224864 0 0 0 12 813 2947 7 10 83 0 0
0 0 3596 7160200 736572 7222004 0 0 0 216 1537 3295 10 10 80 0 0
1 0 3596 7160200 736572 7222004 0 0 0 20 332 271 9 0 91 0 0
^C

[adiviso82]

+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| 3 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 1 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 4 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 2 | system user | | | Daemon | | InnoDB purge coordinator | | 0.000 |
| 5 | system user | | | Daemon | | InnoDB shutdown handler | | 0.000 |
| 135242 | cacti | localhost | cacti | Sleep | 18 | | | 0.000 |
| 135245 | cacti | localhost | cacti | Sleep | 18 | | | 0.000 |
| 135246 | cacti | localhost | cacti | Sleep | 18 | | | 0.000 |
| 135247 | root | localhost | | Query | 0 | Init | show processlist | 0.000 |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+

+--------+-------------+-----------+-------+---------+------+--------------------------+---------------------------------------------------------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-------------+-----------+-------+---------+------+--------------------------+---------------------------------------------------------------------+----------+
| 3 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 1 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 4 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 2 | system user | | | Daemon | | InnoDB purge coordinator | | 0.000 |
| 5 | system user | | | Daemon | | InnoDB shutdown handler | | 0.000 |
| 135242 | cacti | localhost | cacti | Query | 0 | Writing to net | SELECT id FROM graph_local WHERE graph_template_id = '5317' LIMIT 1 | 0.000 |
| 135245 | cacti | localhost | cacti | Sleep | 0 | | | 0.000 |
| 135246 | cacti | localhost | cacti | Sleep | 19 | | | 0.000 |
| 135247 | root | localhost | | Query | 0 | Init | show processlist | 0.000 |
+--------+-------------+-----------+-------+---------+------+--------------------------+---------------------------------------------------------------------+----------+

+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| 3 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 1 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 4 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 2 | system user | | | Daemon | | InnoDB purge coordinator | | 0.000 |
| 5 | system user | | | Daemon | | InnoDB shutdown handler | | 0.000 |
| 135242 | cacti | localhost | cacti | Sleep | 0 | | | 0.000 |
| 135245 | cacti | localhost | cacti | Sleep | 0 | | | 0.000 |
| 135246 | cacti | localhost | cacti | Sleep | 20 | | | 0.000 |
| 135247 | root | localhost | | Query | 0 | Init | show processlist | 0.000 |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
.
.
.
.
.
.
.
.
.
.

+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| 3 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 1 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 4 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 2 | system user | | | Daemon | | InnoDB purge coordinator | | 0.000 |
| 5 | system user | | | Daemon | | InnoDB shutdown handler | | 0.000 |
| 135245 | cacti | localhost | cacti | Sleep | 8 | | | 0.000 |
| 135246 | cacti | localhost | cacti | Sleep | 8 | | | 0.000 |
| 135247 | root | localhost | | Query | 0 | Init | show processlist | 0.000 |
| 135248 | cacti | localhost | cacti | Sleep | 8 | | | 0.000 |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+

+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| 3 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 1 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 4 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 2 | system user | | | Daemon | | InnoDB purge coordinator | | 0.000 |
| 5 | system user | | | Daemon | | InnoDB shutdown handler | | 0.000 |
| 135245 | cacti | localhost | cacti | Sleep | 9 | | | 0.000 |
| 135246 | cacti | localhost | cacti | Sleep | 9 | | | 0.000 |
| 135247 | root | localhost | | Query | 0 | Init | show processlist | 0.000 |
| 135248 | cacti | localhost | cacti | Sleep | 9 | | | 0.000 |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+

+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
| 3 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 1 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 4 | system user | | | Daemon | | InnoDB purge worker | | 0.000 |
| 2 | system user | | | Daemon | | InnoDB purge coordinator | | 0.000 |
| 5 | system user | | | Daemon | | InnoDB shutdown handler | | 0.000 |
| 135245 | cacti | localhost | cacti | Sleep | 10 | | | 0.000 |
| 135246 | cacti | localhost | cacti | Sleep | 10 | | | 0.000 |
| 135247 | root | localhost | | Query | 0 | Init | show processlist | 0.000 |
| 135248 | cacti | localhost | cacti | Sleep | 10 | | | 0.000 |
+--------+-------------+-----------+-------+---------+------+--------------------------+------------------+----------+
^C

[adiviso82]

UPDATE: i did tests last night with a fresh CentOS, cacti 1.2.7 and I did not import any of my templates, but I created a host template with a lot of (dummy) graphs, about 400. Then I created 10 devices to get to the ~4000 graphs in total. Again there were no RRDs present. But the response times were very very good, almost as fast as an old 0.8.8h cacti.

Maybe importing templates from older cacti versions is somehow messing with the cacti code, I don't know how. My plan is to replicate this on Ubuntu VM. I will keep you updated, but please if any ideas do share. I can also give you some templates to test with, If anyone has the time for it.

[Osiris]

Re-importing should not impact rendering per-se. You should always preview templates before re-importing them so that you don't do something you don't want to have happen, happen.

[adiviso82]

Re-importing should not impact rendering per-se. You should always preview templates before re-importing them so that you don't do something you don't want to have happen, happen.

I will follow up with that, too. I didn't pay much attention to this step but i've tried it once or twice and I saw that all items were imported with success. The only problem I imagine is using templates from 0.8.8f in 1.2.7 (now I have mix of them). I will choose a template created in 1.2.7 and preview it before importing, thank you for the tip.

[adiviso82]

Re-importing should not impact rendering per-se. You should always preview templates before re-importing them so that you don't do something you don't want to have happen, happen.

I will follow up with that, too. I didn't pay much attention to this step but i've tried it once or twice and I saw that all items were imported with success. The only problem I imagine is using templates from 0.8.8f in 1.2.7 (now I have mix of them). I will choose a template created in 1.2.7 and preview it before importing, thank you for the tip.

No problem/error while importing the templates. I'm doing a test now, comparing speed between one host template with one old graph template (still from a 1.2.7 cacti) and the same graph created from scratch on the current install (1.2.7). To be able to test this with 1000 graphs per host_template I plan to generate the XML host template via a script.

Does anyone know how to use the cacti function to generate a hash value? Are there any rules about these values? Thanks!

[adiviso82]

Problem fixed guys! I've upgraded to 1.2.8 and I went through the MySQL settings one more time, set them according to cacti installation and it worked like a charm.

[camerabob]

What solved this? (for others)

[adiviso82]

To be honest the easy answer would be that I upgraded to 1.2.8, i've seen some bugs that might be related to this. But a very possible option would be that also the MYSQL settings (in /etc/mysql/my.cnf) were not according to the cacti installation. I might do another test from scratch by the end of the week with Ubuntu 18.04 + cacti 1.2.7 and pay attention to the mysql settings. Then upgrade and identify what was the exact problem. To be more explicit about mysql, i was under the wrong impression that I can paste the parameters anywhere in the my.cnf, but they are supposed to go under the [mysqld] section. But for sure it's one of the 2 things.

[netniV]

With newer versions, you can usually place your custom values into a separate configuration file to be picked up after the defaults.

[adiviso82]

To be honest the easy answer would be that I upgraded to 1.2.8, i've seen some bugs that might be related to this. But a very possible option would be that also the MYSQL settings (in /etc/mysql/my.cnf) were not according to the cacti installation. I might do another test from scratch by the end of the week with Ubuntu 18.04 + cacti 1.2.7 and pay attention to the mysql settings. Then upgrade and identify what was the exact problem. To be more explicit about mysql, i was under the wrong impression that I can paste the parameters anywhere in the my.cnf, but they are supposed to go under the [mysqld] section. But for sure it's one of the 2 things.

Hi guys, I confirm, the issue was solved solely by the cacti 1.2.8 upgrade, probably due to one of the below:
issue#2998: Session performance issues due to excessive use for database storage
issue#3055: During install/upgrade, database tests are not performed correctly
issue#3060: When upgrading from older MySQL databases, format is not changed from compact to dynamic

Just to confirm my latest test, I used Ubuntu mariadb10.4.10 + php 7.2 + cacti 1.2.7.
1) Loaded an previous backup from cacti 1.2.7 => graphs were slow
2) used optimized mysql settings, still slow
3) upgraded to cacti 1.2.8, during setup the format of mysql databases was changed to dynamic, now everything is blazing fast.

Please ask if any questions remain unanswered.