Performance Probelms with Spine

[Schluchti40]

Hello,

i have a Cacti installation running on a virtual machine (Proxmox) 2 Sockets 4 COres, 64 GB Ram. often i get this message in the cacti.log:

2020/04/29 07:56:17 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 67.7 seconds have passed since the last poll!
2020/04/29 07:56:17 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2020/04/29 07:56:42 - SYSTEM STATS: Time:21.2194 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3747
2020/04/29 07:56:44 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.2604 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 07:57:09 - SYSTEM STATS: Time:7.4885 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3747
2020/04/29 07:57:19 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:2.8584 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 07:58:09 - SYSTEM STATS: Time:7.3245 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3747
2020/04/29 07:58:11 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.2071 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 07:59:09 - SYSTEM STATS: Time:7.3175 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3742
2020/04/29 07:59:11 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.2109 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 08:00:09 - SYSTEM STATS: Time:7.6101 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3747
2020/04/29 08:00:11 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.1944 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 08:01:16 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 75.7 seconds have passed since the last poll!
2020/04/29 08:01:16 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2020/04/29 08:01:39 - SYSTEM STATS: Time:15.2663 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3747
2020/04/29 08:01:41 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.2856 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 08:02:33 - SYSTEM STATS: Time:31.3978 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3747
2020/04/29 08:02:35 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.3811 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 08:03:27 - SYSTEM STATS: Time:25.8198 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3745
2020/04/29 08:03:29 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.1871 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 08:04:16 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 75 seconds have passed since the last poll!
2020/04/29 08:04:16 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2020/04/29 08:04:23 - SYSTEM STATS: Time:7.5076 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3747
2020/04/29 08:04:25 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.1690 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 08:05:22 - SYSTEM STATS: Time:11.6896 Method:spine Processes:5 Threads:20 Hosts:138 HostsPerProcess:28 DataSources:7424 RRDsProcessed:3747
2020/04/29 08:05:24 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:1.2254 RRDUser:0.0000 RRDSystem:0.0000 RRDReal:0.0000
2020/04/29 08:06:21 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 73 seconds have passed since the last poll!

now i have tested spine with -R and i don't always get the same runtime back sometimes it takes longer sometimes it runs shorter:

[cacti@cactitemplate etc]$ /usr/local/spine/bin/spine -R -C /usr/local/spine/etc/spine.conf
SPINE: Using spine config file [/usr/local/spine/etc/spine.conf]
SPINE: Version 1.2.10 starting
SPINE: Time: 51.2877 s, Threads: 20, Devices: 139
[cacti@cactitemplate etc]$ /usr/local/spine/bin/spine -R -C /usr/local/spine/etc/spine.conf
SPINE: Using spine config file [/usr/local/spine/etc/spine.conf]
SPINE: Version 1.2.10 starting
SPINE: Time: 26.5041 s, Threads: 20, Devices: 139
[cacti@cactitemplate etc]$ /usr/local/spine/bin/spine -R -C /usr/local/spine/etc/spine.conf
SPINE: Using spine config file [/usr/local/spine/etc/spine.conf]
SPINE: Version 1.2.10 starting
SPINE: Time: 3.4617 s, Threads: 20, Devices: 139
[cacti@cactitemplate etc]$ /usr/local/spine/bin/spine -R -C /usr/local/spine/etc/spine.conf
SPINE: Using spine config file [/usr/local/spine/etc/spine.conf]
SPINE: Version 1.2.10 starting
SPINE: Time: 5.3101 s, Threads: 20, Devices: 139
[cacti@cactitemplate etc]$ /usr/local/spine/bin/spine -R -C /usr/local/spine/etc/spine.conf
SPINE: Using spine config file [/usr/local/spine/etc/spine.conf]
SPINE: Version 1.2.10 starting
SPINE: Time: 21.5908 s, Threads: 20, Devices: 139

can anyone help me or give me a hint as to what might be causing this!
Thanks

[TheWitness]

For so few hosts, I would set 1 processes, ad 20 threads max. Also, if you have crap storage, you could us boost.

[TheWitness]

Sound's like you have maybe network issues too. You should no be seeing that variation in collection timing.

[cigamit]

Seems your VM is having timing issues too (probably because its running in Proxmox, lots of google results of it having this issue). Notice the time stamp. Your cron isn't firing at the correct time (top of the minute).

Code: Select all

2020/04/29 07:56:17 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 67.7 seconds have passed since the last poll!
2020/04/29 08:01:16 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 75.7 seconds have passed since the last poll!
2020/04/29 08:04:16 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 75 seconds have passed since the last poll!
2020/04/29 08:06:21 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 73 seconds have passed since the last poll!