Performance Issue on version 1.2.20

[jsoares]

Hi,
I upgraded Cacti recently to version 1.2.20 and since then every request i make to any page takes at a bare minimum of 10 seconds.
If i want to login it takes 10 seconds for the login prompt to show , every time i click any option in the menu it takes at least 10 seconds to show.
This didn't happen in the previous version and i can't find anything wrong in the logs , cacti , apache , php or the system logs.

I have other web applications running in the same server under the same apache instalation that have no issue, also using the same database engine and only cacti has this issue.

Example,

Loading a page from the tree with 6 graphs take almost 2 minutes.

cacti.PNG (339.79 KiB) Viewed 828 times

Loading the login screen

cacti2.PNG (140.52 KiB) Viewed 828 times

How can i diagnose this?

Thanks in advance.

[cigamit]

Your not really giving us anything to go by. Check the performance of the server. High CPU? High Disk I/O? What is the STAT line from the log?

[jsoares]

I know, thats the issue, theres nothing i can see in the server side that explains that behavior.

Heres the vmstat during a request ( note that this is not for the complete request because it takes too long)

Code: Select all

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 1885824 2347692   1360 3300636    0    2     7    79   23   36  5  2 93  0  0
 0  0 1885824 2347944   1360 3300624    0    0     0   183 1440 1761  4  1 94  0  0
 0  0 1885824 2347916   1360 3300628    0    0     0    14  918 1307  1  1 98  0  0
 0  0 1885824 2347800   1360 3300628    0    0     0    15  897 1356  1  1 99  0  0
 2  0 1885824 2347968   1360 3300628    0    0     0     0 1125 1504  3  1 96  0  0
 0  0 1885824 2347888   1360 3300692    0    0     0   115 1251 1390  7  1 91  0  0
 0  0 1885824 2347888   1360 3300696    0    0     0    66 1575 2103  3  2 95  0  0
 2  0 1885824 2347864   1360 3300696    0    0     0  1257 1207 1464  4  1 95  0  0
 1  0 1885824 2346292   1360 3302136    0    0     0   136 1635 1586 13  2 85  0  0
 0  0 1885824 2347976   1360 3300484    0    0     0    93 1156 1428  5  1 94  0  0
 2  0 1885824 2348200   1360 3300484    0    0     0     0  993 1252  3  1 96  0  0
 0  0 1885824 2348144   1360 3300580    0    0     0    71 1636 1800  9  2 89  0  0
 0  0 1885824 2348148   1360 3300584    0    0     0    63  986 1385  1  1 98  0  0
 0  0 1885824 2347808   1360 3300584    0    0     0    42 1259 1560  4  1 95  0  0
 0  0 1885824 2347752   1360 3300584    0    0     0    17  948 1306  2  1 98  0  0
 2  0 1885824 2347688   1360 3300584    0    0     0     0  779 1159  1  1 99  0  0
 5  0 1885824 2347892   1360 3300592    0    0     0    42 1362 1617  4  2 94  0  0
 0  0 1885824 2347976   1360 3300660    0    0     0    81 1207 1408  7  2 92  0  0
 0  0 1885824 2348116   1360 3300660    0    0     0    29 1076 1321  5  1 93  0  0
 1  0 1885824 2347584   1360 3300660    0    0     0    56 1159 1445  4  1 95  0  0
 1  0 1885824 2347584   1360 3300684    0    0     0    46 1156 1278  7  2 91  0  0
 0  0 1885824 2347856   1360 3300760    0    0     0   158 2107 2575  4  3 93  0  0
 1  0 1885824 2347636   1360 3300760    0    0     0     0  917 1197  4  1 96  0  0
 1  0 1885824 2347996   1360 3300780    0    0     0    85 1184 1382  5  2 93  0  0
 0  0 1885824 2347912   1360 3300828    0    0     0    45 1028 1478  1  1 98  0  0
 0  0 1885824 2347500   1360 3300844    0    0     0    42 2259 2022 14  5 81  0  0
 0  0 1885824 2347440   1360 3300848    0    0     0    12 1259 1617  1  1 98  0  0
 0  0 1885824 2347504   1360 3300848    0    0     0    81 1070 1454  1  1 97  0  0
 1  0 1885824 2348260   1360 3300856    0    0     0    74 1372 1658  5  1 94  0  0
 0  0 1885824 2345236   1360 3303428    0    0     0    75 1596 1326 17  2 82  0  0
 1  0 1885824 2348032   1360 3300664    0    0     0    71  895 1208  2  1 97  0  0
 1  0 1885824 2347804   1360 3300664    0    0     0   147 1382 1651  4  1 94  0  0
 0  0 1885824 2347804   1360 3300688    0    0     0    43 1042 1314  4  1 95  0  0
 0  0 1885824 2348308   1360 3300736    0    0     0    43  997 1367  2  1 97  0  0
 1  0 1885824 2348448   1360 3300736    0    0     0     0 1002 1279  4  1 95  0  0
 0  0 1885824 2347944   1360 3300756    0    0     0   101 1270 1471  6  1 92  0  0
 0  0 1885824 2347828   1360 3300808    0    0     0  5702 1542 1811  3  2 95  0  0
 0  0 1885824 2348220   1360 3300808    0    0     0   778 1085 1360  3  1 96  0  0
 0  0 1885824 2348388   1360 3300808    0    0     0    14  916 1321  1  1 98  0  0
 0  0 1885824 2348224   1360 3300816    0    0     0    46 1078 1467  1  1 97  0  0
 1  0 1885824 2348224   1360 3300816    0    0     0     0  889 1144  4  1 96  0  0
 0  0 1885824 2347692   1360 3300816    0    0     0   115 1599 1820  7  2 92  0  0
 0  0 1885824 2347628   1360 3300896    0    0     0    77 1033 1427  2  1 97  0  0
 2  0 1885824 2347964   1360 3300896    0    0     0    52 1153 1480  4  1 95  0  0
 0  0 1885824 2345472   1360 3303644    0    0     0    64 1569 1378 15  2 83  0  0
 0  0 1885824 2348300   1360 3300680    0    0     0     0  763 1153  1  1 98  0  0
 2  0 1885824 2348268   1360 3300680    0    0     0   102 1210 1460  5  1 94  0  0
 2  0 1885824 2347876   1360 3300776    0    0     0    70 1175 1352  6  2 92  0  0
 1  0 1885824 2347464   1360 3300776    0    0     0    98 1389 1646  5  1 93  0  0
 0  0 1885824 2347880   1360 3300784    0    0     0    18 1044 1323  4  1 95  0  0
 0  0 1885824 2347884   1360 3300784    0    0     0    14  822 1215  1  1 98  0  0
 0  0 1885824 2347864   1360 3300784    0    0     0    29  974 1350  1  1 98  0  0
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0 1885824 2348000   1360 3300784    0    0     0     0  975 1258  4  1 95  0  0
 0  0 1885824 2347608   1360 3300868    0    0     0   127 1463 1592  8  2 90  0  0
 0  0 1885824 2347552   1360 3300868    0    0     0    46 1013 1453  1  1 98  0  0
 1  0 1885824 2348028   1360 3300868    0    0     0    39 1068 1302  4  1 95  0  0
 0  0 1885824 2347748   1360 3300948    0    0     0    46 1158 1299  7  1 92  0  0
 0  0 1885824 2347660   1360 3300948    0    0     0    59 1056 1471  2  1 98  0  0
 1  0 1885824 2347768   1360 3300956    0    0     0    98 1364 1579  6  1 93  0  0
 0  0 1885824 2347796   1360 3301024    0    0     0   115 1247 1459  6  1 92  0  0
 0  0 1885824 2347796   1360 3301028    0    0     0    14  779 1175  1  1 99  0  0
 0  0 1885824 2347844   1360 3301032    0    0     0    36 1133 1417  4  1 95  0  0
 0  0 1885824 2347788   1360 3301036    0    0     0    33  880 1305  1  1 98  0  0
 0  0 1885824 2347756   1360 3301036    0    0     0    28  906 1355  1  1 99  0  0
 1  0 1885824 2347532   1360 3301036    0    0     0     0  941 1265  5  1 95  0  0
 1  0 1885824 2345236   1360 3303724    0    0     0   112 1586 1362 16  2 82  0  0
 0  0 1885824 2348004   1360 3300828    0    0     0    28  808 1199  1  1 98  0  0
 1  0 1885824 2347948   1360 3300832    0    0     0   608 1060 1304  4  1 95  0  0
10  0 1885824 2347984   1360 3300860    0    0     0    74 1849 1359 23  2 75  0  0
 0  0 1885824 2348404   1360 3300912    0    0     0    32 1396 1321  5  1 95  0  0

Code: Select all

MariaDB [(none)]> show processlist;
+------+-------------+-----------+---------+---------+------+--------------------------+------------------+----------+
| Id   | User        | Host      | db      | Command | Time | State                    | Info             | Progress |
+------+-------------+-----------+---------+---------+------+--------------------------+------------------+----------+
|    1 | system user |           | NULL    | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|    2 | system user |           | NULL    | Daemon  | NULL | InnoDB purge coordinator | NULL             |    0.000 |
|    3 | system user |           | NULL    | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|    4 | system user |           | NULL    | Daemon  | NULL | InnoDB purge worker      | NULL             |    0.000 |
|    5 | system user |           | NULL    | Daemon  | NULL | InnoDB shutdown handler  | NULL             |    0.000 |
| 2148 | root        | localhost | NULL    | Query   |    0 | Init                     | show processlist |    0.000 |
| 2149 | cacti_user  | localhost | cactidb | Sleep   |    7 |                          | NULL             |    0.000 |
+------+-------------+-----------+---------+---------+------+--------------------------+------------------+----------+
7 rows in set (0.000 sec)

These are the latest stats in cacti.log

Code: Select all

2022/05/09 17:35:30 - SYSTEM STATS: Time:18.6515 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 17:40:30 - SYSTEM STATS: Time:18.6558 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 17:45:31 - SYSTEM STATS: Time:19.6642 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 17:50:30 - SYSTEM STATS: Time:18.6977 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 17:55:30 - SYSTEM STATS: Time:18.6599 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:00:29 - SYSTEM STATS: Time:17.6648 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:05:37 - SYSTEM STATS: Time:23.6710 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:10:30 - SYSTEM STATS: Time:18.6662 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:15:29 - SYSTEM STATS: Time:17.6218 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:20:29 - SYSTEM STATS: Time:17.6703 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:25:29 - SYSTEM STATS: Time:17.6442 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1033
2022/05/09 18:30:31 - SYSTEM STATS: Time:18.6317 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:30:45 - RECACHE STATS: Poller:1 RecacheTime:3.3164 DevicesRecached:1
2022/05/09 18:35:30 - SYSTEM STATS: Time:18.6678 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:40:29 - SYSTEM STATS: Time:17.6812 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:45:31 - SYSTEM STATS: Time:19.6752 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:50:29 - SYSTEM STATS: Time:17.6783 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 18:55:31 - SYSTEM STATS: Time:18.6662 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:00:31 - SYSTEM STATS: Time:19.6549 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:05:31 - SYSTEM STATS: Time:19.6825 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:10:37 - SYSTEM STATS: Time:22.6218 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:15:28 - SYSTEM STATS: Time:16.5562 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:20:29 - SYSTEM STATS: Time:17.6286 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:25:29 - SYSTEM STATS: Time:17.6192 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:30:32 - SYSTEM STATS: Time:20.6487 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:35:28 - SYSTEM STATS: Time:16.5896 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:40:29 - SYSTEM STATS: Time:17.6843 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:45:30 - SYSTEM STATS: Time:18.6592 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:50:28 - SYSTEM STATS: Time:16.5590 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 19:55:29 - SYSTEM STATS: Time:17.6387 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:00:29 - SYSTEM STATS: Time:17.6370 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:05:27 - SYSTEM STATS: Time:16.5744 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:10:30 - SYSTEM STATS: Time:18.6567 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:15:31 - SYSTEM STATS: Time:19.6039 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:20:26 - SYSTEM STATS: Time:14.6570 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:25:29 - SYSTEM STATS: Time:17.6196 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:30:31 - SYSTEM STATS: Time:19.6392 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:35:30 - SYSTEM STATS: Time:18.7060 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:40:29 - SYSTEM STATS: Time:17.6496 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:45:35 - SYSTEM STATS: Time:22.6525 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:50:31 - SYSTEM STATS: Time:18.6782 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 20:55:36 - SYSTEM STATS: Time:23.6948 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 21:00:31 - SYSTEM STATS: Time:19.6602 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 21:05:31 - SYSTEM STATS: Time:19.6698 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 21:10:35 - SYSTEM STATS: Time:24.6159 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 21:15:29 - SYSTEM STATS: Time:17.6556 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 21:20:29 - SYSTEM STATS: Time:17.6552 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:1035
2022/05/09 21:25:29 - SYSTEM STATS: Time:17.6405 Method:cmd.php Processes:4 Threads:1 Hosts:84 HostsPerProcess:21 DataSources:2245 RRDsProcessed:103

What else can i check ?
As for hardware its a VM with 4 cores and 16 GB of RAM although it had 8 GB, just added more memory after the issues started in a desperate attempt to fix it.

[macan]

Try clear cache, another browser or incognito.

[jsoares]

I tried on a new machine that never acessed cacti before and it takes 2 minutes just to list 80 devices , not the graphs, just the list of devices.
It doesnt seem to be a client issue.

Best regards,

[macan]

I expecting something in web server or system log.
You can try update php. I once had similar problem and upgrade php from x.y4 to x.y5 solved it.

[jsoares]

Hi,

The server is currently running with PHP 7.4.29 and apache Apache 2.4.37. Those are the latest versions i can get.
I could try and go for PHP8 but i don't know if the rest of the web applications in the server can handle that.

I'll try setting up a server with the same versions i have on this one and see if cacti works without any data sources or configurations. If it has the same symptoms i'll try different versions and see if i can find a more stable combination.

Thanks for the help.

[jsoares]

So i've tried reinstalling cacti on the same system and before i import the database it works fine, its fast to navigate the site and everything seems to work. After i import the database and before importing the rdds files ( with the poller still off) the symptom comes back , everything starts to drag their feet. This leads me to thing that the problem is not in the app , not even in the database itself but the data. The site is fast with the default cacti data and it only slows down when i import the old data.

The database is 8 Mb in size so it shouldn't be an issue.

Going to try and find some way to debug the interactions between cacti and the database to see if i find the culprit.

[macan]

Please show: select * from version;

[jsoares]

Hi,

There you go!

MariaDB [cactidb]> select * from version;
+--------+
| cacti |
+--------+
| 1.2.20 |
+--------+
1 row in set (0.000 sec)


Thanks!

[macan]

Try to update to 1.2.21