[1.2.11] spine poller over > 300 seconds happens on specific time

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Developers, Moderators

Post Reply
bongs
Posts: 8
Joined: Tue Nov 06, 2018 6:53 am

[1.2.11] spine poller over > 300 seconds happens on specific time

Post by bongs »

hi, i have problem with cacti that happen on 1.1.38 and upgrade to 1.2.11.

Code: Select all

 Technical Support [Summary]
Date	Mon, 15 Jun 2020 12:30:51 +0700
Cacti Version	1.2.11
Cacti OS	unix
NET-SNMP Version	NET-SNMP version: 5.7.2
RRDtool Version Configured	1.4.0+
RRDtool Version Found	1.4.8
Devices	1,083
Graphs	54,757
Data Sources	Script/Command: 5
SNMP Get: 2
SNMP Query: 54,752
Script Query: 4
Script Server: 1
Total: 54,764
Interval	300
Type	SPINE 1.2.11 Copyright 2004-2020 by The Cacti Group
Items	Action[0]: 106,102
Action[1]: 13
Action[2]: 1
Total: 106,116
Concurrent Processes	10
Max Threads	30
PHP Servers	1
Script Timeout	25
Max OID	1
Last Run Statistics	Time:44.8511 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106110 RRDsProcessed:52213
MemTotal	32,958,308,000
MemFree	2,084,272,000
MemAvailable	25,361,796,000
Buffers	0
Cached	22,591,716,000
Active	20,122,952,000
Inactive	8,747,828,000
SwapTotal	2,097,148,000
SwapFree	2,092,532,000
PHP Version	7.3.17
PHP OS	Linux
PHP uname	Linux cacti 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64
PHP SNMP	Installed. Note: If you are planning on using SNMPv3, you must remove php-snmp and use the Net-SNMP toolset.
max_execution_time	60
memory_limit	1024M
version	10.2.31-MariaDB	>=	5.6	MySQL 5.6+ and MariaDB 10.0+ are great releases, and are very good versions to choose. Make sure you run the very latest release though which fixes a long standing low level networking issue that was causing spine many issues with reliability.
collation_server	utf8mb4_unicode_ci	=	utf8mb4_unicode_ci	When using Cacti with languages other than English, it is important to use the utf8mb4_unicode_ci collation type as some characters take more than a single byte.
character_set_client	utf8mb4	=	utf8mb4	When using Cacti with languages other than English, it is important to use the utf8mb4 character set as some characters take more than a single byte.
max_connections	750	>=	100	Depending on the number of logins and use of spine data collector, MariaDB will need many connections. The calculation for spine is: total_connections = total_processes * (total_threads + script_servers + 1), then you must leave headroom for user connections, which will change depending on the number of concurrent login accounts.
max_allowed_packet	33554432	>=	16777216	With Remote polling capabilities, large amounts of data will be synced from the main server to the remote pollers. Therefore, keep this value at or above 16M.
max_heap_table_size	512M	>=	503M	If using the Cacti Performance Booster and choosing a memory storage engine, you have to be careful to flush your Performance Booster buffer before the system runs out of memory table space. This is done two ways, first reducing the size of your output column to just the right size. This column is in the tables poller_output, and poller_output_boost. The second thing you can do is allocate more memory to memory tables. We have arbitrarily chosen a recommended value of 10% of system memory, but if you are using SSD disk drives, or have a smaller system, you may ignore this recommendation or choose a different storage engine. You may see the expected consumption of the Performance Booster tables under Console -> System Utilities -> View Boost Status.
tmp_table_size	64M	>=	503M	When executing subqueries, having a larger temporary table size, keep those temporary tables in memory.
join_buffer_size	64M	>=	1006M	When performing joins, if they are below this size, they will be kept in memory and never written to a temporary file.
innodb_file_per_table	ON	=	ON	When using InnoDB storage it is important to keep your table spaces separate. This makes managing the tables simpler for long time users of MariaDB. If you are running with this currently off, you can migrate to the per file storage by enabling the feature, and then running an alter statement on all InnoDB tables.
innodb_file_format	Barracuda	=	Barracuda	When using innodb_file_per_table, it is important to set the innodb_file_format to Barracuda. This setting will allow longer indexes important for certain Cacti tables.
innodb_large_prefix	ON	=	1	If your tables have very large indexes, you must operate with the Barracuda innodb_file_format and the innodb_large_prefix equal to 1. Failure to do this may result in plugins that can not properly create tables.
innodb_buffer_pool_size	5120M	>=	7858M	InnoDB will hold as much tables and indexes in system memory as is possible. Therefore, you should make the innodb_buffer_pool large enough to hold as much of the tables and index in memory. Checking the size of the /var/lib/mysql/cacti directory will help in determining this value. We are recommending 25% of your systems total memory, but your requirements will vary depending on your systems size.
innodb_doublewrite	ON	=	ON	This settings should remain ON unless your Cacti instances is running on either ZFS or FusionI/O which both have internal journaling to accomodate abrupt system crashes. However, if you have very good power, and your systems rarely go down and you have backups, turning this setting to OFF can net you almost a 50% increase in database performance.
innodb_lock_wait_timeout	100	>=	50	Rogue queries should not for the database to go offline to others. Kill these queries before they kill your system.
innodb_flush_method		eq	O_DIRECT	Maximum I/O performance happens when you use the O_DIRECT method to flush pages.
innodb_flush_log_at_timeout	3	>=	3	As of MariaDB 10.2.31, the you can control how often MariaDB flushes transactions to disk. The default is 1 second, but in high I/O systems setting to a value greater than 1 can allow disk I/O to be more sequential
innodb_read_io_threads	32	>=	32	With modern SSD type storage, having multiple read io threads is advantageous for applications with high io characteristics.
innodb_write_io_threads	16	>=	16	With modern SSD type storage, having multiple write io threads is advantageous for applications with high io characteristics.
innodb_buffer_pool_instances	5	>=	41	MariaDB will divide the innodb_buffer_pool into memory regions to improve performance. The max value is 64. When your innodb_buffer_pool is less than 1GB, you should use the pool size divided by 128MB. Continue to use this equation upto the max of 64.
innodb_io_capacity	200	>=	5000	If you have SSD disks, use this suggestion. If you have physical hard drives, use 200 * the number of active drives in the array. If using NVMe or PCIe Flash, much larger numbers as high as 100000 can be used.
innodb_io_capacity_max	4000	>=	10000	If you have SSD disks, use this suggestion. If you have physical hard drives, use 2000 * the number of active drives in the array. If using NVMe or PCIe Flash, much larger numbers as high as 200000 can be used.
below is spine status:

Code: Select all

2020/06/15 12:30:54 - SYSTEM STATS: Time:52.2057 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106114 RRDsProcessed:52240
2020/06/15 12:25:47 - SYSTEM STATS: Time:44.8511 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106110 RRDsProcessed:52213
2020/06/15 12:20:53 - RECACHE STATS: Poller:1 RecacheTime:8.4068 DevicesRecached:1
2020/06/15 12:20:44 - SYSTEM STATS: Time:42.2282 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106034 RRDsProcessed:52184
2020/06/15 12:15:45 - SYSTEM STATS: Time:42.6589 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52164
2020/06/15 12:10:51 - SYSTEM STATS: Time:49.3718 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52191
2020/06/15 12:05:56 - SYSTEM STATS: Time:54.4588 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52253
2020/06/15 12:00:49 - SYSTEM STATS: Time:47.0824 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52221
2020/06/15 11:55:46 - SYSTEM STATS: Time:44.4528 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106098 RRDsProcessed:52203
2020/06/15 11:50:44 - SYSTEM STATS: Time:41.5115 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106098 RRDsProcessed:52232
2020/06/15 11:45:44 - SYSTEM STATS: Time:42.0854 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52193
2020/06/15 11:41:01 - SYSTEM STATS: Time:58.0195 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52157
2020/06/15 11:35:51 - SYSTEM STATS: Time:48.9865 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52228
2020/06/15 11:30:49 - SYSTEM STATS: Time:47.0204 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52253
2020/06/15 11:25:48 - SYSTEM STATS: Time:45.7976 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106000 RRDsProcessed:52197
2020/06/15 11:20:52 - SYSTEM STATS: Time:50.2404 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106000 RRDsProcessed:52197
2020/06/15 11:15:45 - SYSTEM STATS: Time:42.3884 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106000 RRDsProcessed:52114
2020/06/15 11:10:46 - SYSTEM STATS: Time:44.6994 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106056 RRDsProcessed:52199
2020/06/15 11:10:00 - SYSTEM STATS: Time:285.3742 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106056 RRDsProcessed:18111
2020/06/15 11:00:50 - SYSTEM STATS: Time:48.1747 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106056 RRDsProcessed:52229
2020/06/15 10:55:59 - SYSTEM STATS: Time:57.2101 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52257
2020/06/15 10:50:51 - SYSTEM STATS: Time:49.2905 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106112 RRDsProcessed:52257
2020/06/15 10:45:44 - SYSTEM STATS: Time:42.1638 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106110 RRDsProcessed:52176
2020/06/15 10:40:46 - SYSTEM STATS: Time:43.3096 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106110 RRDsProcessed:52212
2020/06/15 10:36:02 - SYSTEM STATS: Time:49.7584 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52234
2020/06/15 10:35:00 - SYSTEM STATS: Time:298.3795 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52241
2020/06/15 10:25:44 - SYSTEM STATS: Time:41.8252 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106104 RRDsProcessed:52253
2020/06/15 10:20:44 - SYSTEM STATS: Time:42.2941 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106088 RRDsProcessed:52245
2020/06/15 10:15:45 - SYSTEM STATS: Time:42.7917 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52164
2020/06/15 10:10:44 - SYSTEM STATS: Time:42.5140 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52255
2020/06/15 10:06:02 - SYSTEM STATS: Time:53.2470 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52255
2020/06/15 10:00:53 - SYSTEM STATS: Time:50.9073 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52255
2020/06/15 09:55:49 - SYSTEM STATS: Time:46.7165 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52230
2020/06/15 09:50:57 - SYSTEM STATS: Time:55.1649 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52227
2020/06/15 09:45:44 - SYSTEM STATS: Time:41.8903 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52176
2020/06/15 09:40:56 - SYSTEM STATS: Time:54.1434 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52212
2020/06/15 09:35:46 - SYSTEM STATS: Time:44.1075 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52227
2020/06/15 09:30:48 - SYSTEM STATS: Time:46.4508 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52213
2020/06/15 09:25:52 - SYSTEM STATS: Time:49.6047 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52227
2020/06/15 09:25:01 - SYSTEM STATS: Time:298.7144 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52227
2020/06/15 09:15:44 - SYSTEM STATS: Time:41.9626 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52188
2020/06/15 09:11:06 - SYSTEM STATS: Time:64.1281 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52225
2020/06/15 09:10:00 - SYSTEM STATS: Time:294.7454 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:19364
2020/06/15 09:01:34 - RECACHE STATS: Poller:1 RecacheTime:10.1842 DevicesRecached:1
2020/06/15 09:00:53 - SYSTEM STATS: Time:51.0126 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52255
2020/06/15 08:55:45 - RECACHE STATS: Poller:1 RecacheTime:0.3369 DevicesRecached:1
2020/06/15 08:55:44 - SYSTEM STATS: Time:41.9192 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52255
2020/06/15 08:50:55 - SYSTEM STATS: Time:52.6085 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:51975
2020/06/15 08:46:47 - RECACHE STATS: Poller:1 RecacheTime:40.5171 DevicesRecached:1
2020/06/15 08:45:59 - SYSTEM STATS: Time:57.3440 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52201
2020/06/15 08:40:46 - SYSTEM STATS: Time:43.4004 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52174
2020/06/15 08:35:55 - SYSTEM STATS: Time:53.5417 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52212
2020/06/15 08:30:48 - SYSTEM STATS: Time:45.7951 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52255
2020/06/15 08:25:44 - SYSTEM STATS: Time:42.5641 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52229
2020/06/15 08:20:48 - SYSTEM STATS: Time:45.7837 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52241
2020/06/15 08:15:47 - SYSTEM STATS: Time:44.5301 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52195
2020/06/15 08:10:43 - SYSTEM STATS: Time:41.6657 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52227
2020/06/15 08:05:55 - SYSTEM STATS: Time:53.6365 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106028 RRDsProcessed:52144
2020/06/15 08:00:52 - SYSTEM STATS: Time:49.5787 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106028 RRDsProcessed:52201
2020/06/15 07:55:45 - SYSTEM STATS: Time:42.5862 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106028 RRDsProcessed:52176
2020/06/15 07:50:48 - SYSTEM STATS: Time:46.4479 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106084 RRDsProcessed:52243
2020/06/15 07:45:44 - SYSTEM STATS: Time:42.0804 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52228
2020/06/15 07:40:56 - SYSTEM STATS: Time:53.8624 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52231
2020/06/15 07:35:59 - SYSTEM STATS: Time:51.9039 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52255
2020/06/15 07:30:58 - SYSTEM STATS: Time:55.5806 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52241 
i'm doing filter cacti log and found that worst rrdprocessed only happens sometimes on every hour at minutes 10.

Code: Select all

 [root@cacti]#  cat /usr/share/cacti/log/cacti.log | awk -F ':' '$5<300 && $5>280'
2020/06/15 01:10:00 - SYSTEM STATS: Time:297.7581 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:22222
2020/06/15 01:35:00 - SYSTEM STATS: Time:297.7467 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:53917
2020/06/15 01:40:00 - SYSTEM STATS: Time:290.4004 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:26640
2020/06/15 02:10:02 - SYSTEM STATS: Time:293.0757 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:5796
2020/06/15 02:40:01 - SYSTEM STATS: Time:296.5866 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106080 RRDsProcessed:25857
2020/06/15 03:10:00 - SYSTEM STATS: Time:295.6252 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106104 RRDsProcessed:21506
2020/06/15 05:10:01 - SYSTEM STATS: Time:295.2717 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:36003
2020/06/15 07:10:00 - SYSTEM STATS: Time:293.1604 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:34785
2020/06/15 09:10:00 - SYSTEM STATS: Time:294.7454 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:19364
2020/06/15 09:25:01 - SYSTEM STATS: Time:298.7144 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106052 RRDsProcessed:52227
2020/06/15 10:35:00 - SYSTEM STATS: Time:298.3795 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106108 RRDsProcessed:52241
2020/06/15 11:10:00 - SYSTEM STATS: Time:285.3742 Method:spine Processes:10 Threads:10 Hosts:1082 HostsPerProcess:109 DataSources:106056 RRDsProcessed:18111
 

any ideas will be appreciated.

BongS
bmfmancini
Cacti User
Posts: 250
Joined: Wed Mar 13, 2019 3:37 pm
Location: toronto
Contact:

Re: [1.2.11] spine poller over > 300 seconds happens on specific time

Post by bmfmancini »

Looks like some sort of network issue

Is there a backup or something happening at that time that could be slowing down the network ?
Telecom Lifer, Monitoring and performance enthusiast, Father, Husband

Cacti Bug Hunter and Member

www.seanmancini.com
sean(at)seanmancini.com
github.com/bmfmancini

My Cacti scripts and templates ( Non official)
Cacti install wizard https://github.com/bmfmancini/cacti-install-wizard
Cacti templates https://github.com/bmfmancini/Cacti-templates

Always willing to help!
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests