Poller database problems after upgrading to 0.8.7

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Developers, Moderators

Post Reply
denny
Posts: 14
Joined: Wed Oct 24, 2007 11:04 pm

Poller database problems after upgrading to 0.8.7

Post by denny »

After upgrading to 0.8.7, I am experiencing serious poller problems with 1 minute polling.

Php memory began at 128M. Increasing to 256M, and subsequently to 512M has had no effect. Both cmd.php and spine have the same result. Increasing/decreasing concurrent poller processes or threads seem to have no effect.

Any help appreciated.

Thanks,

Denny
  • 10/25/2007 10:08:01 PM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1062', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (239,'traffic_in','2007-10-25 22:08:01','3971067733'),(239,'traffic_out','2007-10-25 22:08:01','3873725980'),(240,'traffic_in','2007-10-25 22:08:01','1124103642'),(240,'traffic_out','2007-10-25 22:08:01','458596355'),(241,'traffic_in','2007-10-25 22:08:01','1879346167'),(241,'traffic_out','2007-10-25 22:08:01','2001957028'),(242,'traffic_in','2007-10-25 22:08:01','3200746519'),(242,'traffic_out','2007-10-25 22:08:01','1076563751'),(243,'traffic_in','2007-10-25 22:08:01','0'),(243,'traffic_out','2007-10-25 22:08:01','84'),(244,'traffic_in','2007-10-25 22:08:01','0'),(244,'traffic_out','2007-10-25 22:08:01','0'),(245,'traffic_in','2007-10-25 22:08:01','21870009'),(245,'traffic_out','2007-10-25 22:08:01','277565436'),(246,'errors_out','2007-10-25 22:08:01','0'),(246,'errors_in','2007-10-25 22:08:01','151'),(246,'discards_in','2007-10-25 22:08:01','592'),(246,'discards_out','2007-10-25 22:08:01','0'),(247,'errors_out','2007-10-25 22:08:01','
    10/25/2007 10:08:01 PM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1062', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (192,'traffic_in','2007-10-25 22:08:01','2066883184'),(192,'traffic_out','2007-10-25 22:08:01','3992116150'),(193,'traffic_out','2007-10-25 22:08:01','0'),(193,'traffic_in','2007-10-25 22:08:01','0'),(194,'errors_out','2007-10-25 22:08:01','0'),(194,'discards_out','2007-10-25 22:08:01','0'),(194,'discards_in','2007-10-25 22:08:01','1995'),(194,'errors_in','2007-10-25 22:08:01','0'),(195,'errors_out','2007-10-25 22:08:01','0'),(195,'discards_out','2007-10-25 22:08:01','0'),(195,'discards_in','2007-10-25 22:08:01','0'),(195,'errors_in','2007-10-25 22:08:01','0'),(196,'hdd_used','2007-10-25 22:08:01','94760'),(196,'hdd_free','2007-10-25 22:08:01','467456'),(197,'hdd_free','2007-10-25 22:08:01','124592'),(197,'hdd_used','2007-10-25 22:08:01','108292'),(198,'hdd_free','2007-10-25 22:08:01','4326492'),(198,'hdd_used','2007-10-25 22:08:01','10934732'),(199,'hdd_free','2007-10-25 22:08:01','4185488'),(199,'hdd_used','2007-10-25 22:08:01','38547'
    10/25/2007 10:07:04 PM - SYSTEM STATS: Time:2.2521 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/25/2007 10:06:04 PM - SYSTEM STATS: Time:2.2522 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/25/2007 10:05:07 PM - SYSTEM STATS: Time:5.3387 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/25/2007 10:05:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: traffic_in(DS[202]), traffic_out(DS[202]), traffic_in(DS[203]), traffic_out(DS[203]), traffic_in(DS[204]), traffic_out(DS[204]), traffic_in(DS[205]), traffic_out(DS[205]), traffic_in(DS[206]), traffic_out(DS[206]), traffic_in(DS[207]), traffic_out(DS[207]), traffic_in(DS[208]), traffic_out(DS[208]), traffic_in(DS[209]), traffic_out(DS[209]), traffic_in(DS[210]), traffic_out(DS[210]), traffic_in(DS[211]), traffic_out(DS[211]), traffic_in(DS[212]), traffic_out(DS[212]), traffic_in(DS[213]), traffic_out(DS[213]), traffic_in(DS[214]), traffic_out(DS[214]), traffic_in(DS[215]), traffic_out(DS[215]), traffic_in(DS[216]), traffic_out(DS[216]), traffic_in(DS[217]), traffic_out(DS[217]), traffic_in(DS[218]), traffic_out(DS[218]), traffic_in(DS[219]), traffic_out(DS[219]), traffic_in(DS[220]), traffic_out(DS[220]), traffic_in(DS[221]), traffic_out(DS[221]), traffic_in(DS[222]), traffic_out(DS[222]), traffic_in(DS[223]), traffic_out(DS[223]), traffic_in(DS[224]), traffic_out(DS[224]), discards_in(DS[225]), discards_out(DS[225]), errors_in(DS[225]), errors_out(DS[225]), discards_in(DS[226]), discards_out(DS[226]), errors_in(DS[226]), errors_out(DS[226]), traffic_in(DS[227]), traffic_out(DS[227]), traffic_in(DS[228]), traffic_out(DS[228]), traffic_in(DS[229]), traffic_out(DS[229]), traffic_in(DS[230]), traffic_out(DS[230]), traffic_in(DS[231]), traffic_out(DS[231]), traffic_in(DS[232]), traffic_out(DS[232]), traffic_in(DS[233]), traffic_out(DS[233]), traffic_in(DS[234]), traffic_out(DS[234]), traffic_in(DS[235]), traffic_out(DS[235]), discards_in(DS[236]), discards_out(DS[236]), errors_in(DS[236]), errors_out(DS[236]), discards_in(DS[237]), discards_out(DS[237]), errors_in(DS[237]), errors_out(DS[237]), discards_in(DS[238]), discards_out(DS[238]), errors_in(DS[238]), errors_out(DS[238])
    10/25/2007 10:04:04 PM - SYSTEM STATS: Time:3.1884 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:19
    10/25/2007 10:04:04 PM - SYSTEM STATS: Time:3.3013 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/25/2007 10:04:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: traffic_in(DS[239]), traffic_out(DS[239]), traffic_in(DS[240]), traffic_out(DS[240]), traffic_in(DS[241]), traffic_out(DS[241]), traffic_in(DS[242]), traffic_out(DS[242]), traffic_in(DS[243]), traffic_out(DS[243]), traffic_in(DS[244]), traffic_out(DS[244]), traffic_in(DS[245]), traffic_out(DS[245]), discards_in(DS[246]), discards_out(DS[246]), errors_in(DS[246]), errors_out(DS[246]), discards_in(DS[247]), discards_out(DS[247]), errors_in(DS[247]), errors_out(DS[247]), discards_in(DS[248]), discards_out(DS[248]), errors_in(DS[248]), errors_out(DS[248]), discards_in(DS[249]), discards_out(DS[249]), errors_in(DS[249]), errors_out(DS[249]), discards_in(DS[250]), discards_out(DS[250]), errors_in(DS[250]), errors_out(DS[250]), traffic_in(DS[251]), traffic_out(DS[251]), discards_in(DS[252]), discards_out(DS[252]), errors_in(DS[252]), errors_out(DS[252]), cpu_nice(DS[253]), cpu_system(DS[254]), cpu_user(DS[255]), mem_buffers(DS[259]), mem_cache(DS[260]), mem_free(DS[261]), hdd_free(DS[262]), hdd_used(DS[262]), hdd_free(DS[263]), hdd_used(DS[263]), hdd_free(DS[264]), hdd_used(DS[264]), hdd_free(DS[265]), hdd_used(DS[265]), hdd_free(DS[266]), hdd_used(DS[266]), hdd_free(DS[267]), hdd_used(DS[267]), Bytes_Read(DS[268]), Bytes_Written(DS[268]), cpu_nice(DS[269]), cpu_system(DS[270]), cpu_user(DS[271]), load_1min(DS[272]), load_15min(DS[273]), load_5min(DS[274]), mem_buffers(DS[275]), mem_cache(DS[276]), mem_free(DS[277]), hdd_free(DS[278]), hdd_used(DS[278]), hdd_free(DS[279]), hdd_used(DS[279]), hdd_free(DS[280]), hdd_used(DS[280]), hdd_free(DS[281]), hdd_used(DS[281]), hdd_free(DS[282]), hdd_used(DS[282]), hdd_free(DS[283]), hdd_used(DS[283]), Bytes_Read(DS[284]), Bytes_Written(DS[284]), load_1min(DS[297]), load_15min(DS[298]), load_5min(DS[299])
    10/25/2007 10:03:03 PM - SYSTEM STATS: Time:2.3113 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/25/2007 10:02:03 PM - SYSTEM STATS: Time:2.2551 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Can you please apply this patch.

http://forums.cacti.net/viewtopic.php?t=23925

The warning is that something has affected your poller output to RRDfile process. I discovered this bug via the attached post.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
denny
Posts: 14
Joined: Wed Oct 24, 2007 11:04 pm

Post by denny »

Thanks! I inserted the new versions of query_host_cpu.php and query_host_partitions.php. The number of errors is reduced.

However, I am still receiving DB insert errors, and poller output table warnings:
  • 10/26/2007 10:46:03 AM - SYSTEM STATS: Time:2.4737 Method:spine Processes:5 Threads:5 Hosts:6 HostsPerProcess:2 DataSources:174 RRDsProcessed:84
    10/26/2007 10:46:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: traffic_in(DS[202]), traffic_out(DS[202]), traffic_in(DS[203]), traffic_out(DS[203]), traffic_in(DS[204]), traffic_out(DS[204]), traffic_in(DS[205]), traffic_out(DS[205]), traffic_in(DS[206]), traffic_out(DS[206]), traffic_in(DS[207]), traffic_out(DS[207]), traffic_in(DS[208]), traffic_out(DS[208]), traffic_in(DS[209]), traffic_out(DS[209]), traffic_in(DS[210]), traffic_out(DS[210]), traffic_in(DS[211]), traffic_out(DS[211]), traffic_in(DS[212]), traffic_out(DS[212]), traffic_in(DS[213]), traffic_out(DS[213]), traffic_in(DS[214]), traffic_out(DS[214]), traffic_in(DS[215]), traffic_out(DS[215]), traffic_in(DS[216]), traffic_out(DS[216]), traffic_in(DS[217]), traffic_out(DS[217]), traffic_in(DS[218]), traffic_out(DS[218]), traffic_in(DS[219]), traffic_out(DS[219]), traffic_in(DS[220]), traffic_out(DS[220]), traffic_in(DS[221]), traffic_out(DS[221])
    10/26/2007 10:45:04 AM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1062', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (202,'traffic_out','2007-10-26 10:45:01','160990850'),(203,'traffic_out','2007-10-26 10:45:01','906948408'),(202,'traffic_in','2007-10-26 10:45:01','215915532'),(203,'traffic_in','2007-10-26 10:45:01','776748535'),(204,'traffic_out','2007-10-26 10:45:01','0'),(205,'traffic_out','2007-10-26 10:45:01','0'),(204,'traffic_in','2007-10-26 10:45:01','0'),(205,'traffic_in','2007-10-26 10:45:01','0'),(206,'traffic_out','2007-10-26 10:45:01','720555'),(219,'traffic_out','2007-10-26 10:45:01','70168306'),(206,'traffic_in','2007-10-26 10:45:01','0'),(219,'traffic_in','2007-10-26 10:45:01','15131640'),(218,'traffic_out','2007-10-26 10:45:01','0'),(217,'traffic_out','2007-10-26 10:45:01','0'),(218,'traffic_in','2007-10-26 10:45:01','0'),(214,'traffic_in','2007-10-26 10:45:01','0'),(214,'traffic_out','2007-10-26 10:45:01','0'),(215,'traffic_in','2007-10-26 10:45:01','0'),(215,'traffic_out','2007-10-26 10:45:01','0'),(216,'traffic_in','2007-10-26 10:4'
    10/26/2007 10:45:02 AM - SYSTEM STATS: Time:1.4771 Method:spine Processes:5 Threads:5 Hosts:6 HostsPerProcess:2 DataSources:174 RRDsProcessed:64
    10/26/2007 10:45:02 AM - SYSTEM STATS: Time:1.3612 Method:spine Processes:5 Threads:5 Hosts:6 HostsPerProcess:2 DataSources:174 RRDsProcessed:5
    10/26/2007 10:45:01 AM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1062', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (227,'traffic_in','2007-10-26 10:45:01','1873903971'),(227,'traffic_out','2007-10-26 10:45:01','2069028525'),(228,'traffic_in','2007-10-26 10:45:01','3093583269'),(228,'traffic_out','2007-10-26 10:45:01','1668048328'),(229,'traffic_in','2007-10-26 10:45:01','2875220589'),(229,'traffic_out','2007-10-26 10:45:01','724739033'),(230,'traffic_in','2007-10-26 10:45:01','1844747801'),(230,'traffic_out','2007-10-26 10:45:01','2010805263'),(231,'traffic_in','2007-10-26 10:45:01','541659'),(231,'traffic_out','2007-10-26 10:45:01','1446915'),(232,'traffic_in','2007-10-26 10:45:01','3083713348'),(232,'traffic_out','2007-10-26 10:45:01','837663440'),(233,'traffic_in','2007-10-26 10:45:01','0'),(233,'traffic_out','2007-10-26 10:45:01','25668'),(234,'traffic_in','2007-10-26 10:45:01','2873768712'),(234,'traffic_out','2007-10-26 10:45:01','629559704'),(235,'traffic_in','2007-10-26 10:45:01','1469139'),(235,'traffic_out','2007-10-26 10:45:01','496649'),'
    10/26/2007 10:45:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: traffic_in(DS[192]), traffic_out(DS[192]), traffic_in(DS[193]), traffic_out(DS[193]), discards_in(DS[194]), discards_out(DS[194]), errors_in(DS[194]), errors_out(DS[194]), discards_in(DS[195]), discards_out(DS[195]), errors_in(DS[195]), errors_out(DS[195]), hdd_free(DS[196]), hdd_used(DS[196]), hdd_free(DS[197]), hdd_used(DS[197]), hdd_free(DS[198]), hdd_used(DS[198]), hdd_free(DS[199]), hdd_used(DS[199]), hdd_free(DS[200]), hdd_used(DS[200]), Bytes_Read(DS[201]), Bytes_Written(DS[201]), traffic_in(DS[239]), traffic_out(DS[239]), traffic_in(DS[240]), traffic_out(DS[240]), traffic_in(DS[241]), traffic_out(DS[241]), traffic_in(DS[242]), traffic_out(DS[242]), traffic_in(DS[243]), traffic_out(DS[243]), traffic_in(DS[244]), traffic_out(DS[244]), traffic_in(DS[245]), traffic_out(DS[245]), discards_in(DS[246]), discards_out(DS[246]), errors_in(DS[246]), errors_out(DS[246]), discards_in(DS[247]), discards_out(DS[247]), errors_in(DS[247]), errors_out(DS[247]), discards_in(DS[248]), discards_out(DS[248]), errors_in(DS[248]), errors_out(DS[248]), discards_in(DS[249]), discards_out(DS[249]), errors_in(DS[249]), errors_out(DS[249]), discards_in(DS[250]), discards_out(DS[250]), errors_in(DS[250]), errors_out(DS[250]), cpu_nice(DS[269]), cpu_system(DS[270]), cpu_user(DS[271]), load_1min(DS[272]), load_15min(DS[273]), load_5min(DS[274]), mem_buffers(DS[275]), mem_cache(DS[276]), mem_free(DS[277]), hdd_free(DS[278]), hdd_used(DS[278]), hdd_free(DS[279]), hdd_used(DS[279]), hdd_free(DS[280]), hdd_used(DS[280]), hdd_free(DS[281]), hdd_used(DS[281]), hdd_free(DS[282]), hdd_used(DS[282]), hdd_free(DS[283]), hdd_used(DS[283]), Bytes_Read(DS[284]), Bytes_Written(DS[284]), cpu_nice(DS[285]), cpu_system(DS[286]), cpu_user(DS[287]), load_1min(DS[288]), load_15min(DS[289]), load_5min(DS[290]), mem_buffers(DS[291]), mem_cache(DS[292]), mem_free(DS[293])
    10/26/2007 10:44:03 AM - SYSTEM STATS: Time:2.5076 Method:spine Processes:5 Threads:5 Hosts:6 HostsPerProcess:2 DataSources:174 RRDsProcessed:84
    10/26/2007 10:43:04 AM - SYSTEM STATS: Time:2.5255 Method:spine Processes:5 Threads:5 Hosts:6 HostsPerProcess:2 DataSources:174 RRDsProcessed:84
    10/26/2007 10:42:03 AM - SYSTEM STATS: Time:2.4613 Method:spine Processes:5 Threads:5 Hosts:6 HostsPerProcess:2 DataSources:174 RRDsProcessed:84
    10/26/2007 10:41:03 AM - SYSTEM STATS: Time:2.4691 Method:spine Processes:5 Threads:5 Hosts:6 HostsPerProcess:2 DataSources:174 RRDsProcessed:84
    10/26/2007 10:38:03 AM - SYSTEM STATS: Time:2.4619 Method:spine Processes:5 Threads:5 Hosts:6 HostsPerProcess:2 DataSources:174 RRDsProcessed:84
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Do you have a double crontab issue?

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Crap, this sounds like you have two data sources with the same primary key. That is pretty bad. I will have to get to this later. Have to go out for a bit.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
denny
Posts: 14
Joined: Wed Oct 24, 2007 11:04 pm

Post by denny »

TheWitness wrote:Do you have a double crontab issue?

TheWitness
No, just one crontab entry. However, I have noticed that poller.php will sometimes hang around for several (5) minutes. But not always.

A bit of checking shows that this appears related to the poller output table problem. When the hung poller.php process exits, it (or the one that has just started) will report that the poller output table is not empty.

Notice in the log snippet below that there is no STATS report at 12:02, but 2 reports at 12:07. The second report is for the process that started by cron at 12:02.
  • 10/26/2007 12:07:04 PM - SYSTEM STATS: Time:3.2339 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:47
    10/26/2007 12:07:04 PM - SYSTEM STATS: Time:2.2764 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:74
    10/26/2007 12:07:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: traffic_in(DS[239]), traffic_out(DS[239]), traffic_in(DS[240]), traffic_out(DS[240]), traffic_in(DS[241]), traffic_out(DS[241]), traffic_in(DS[242]), traffic_out(DS[242]), traffic_in(DS[243]), traffic_out(DS[243]), traffic_in(DS[244]), traffic_out(DS[244]), traffic_in(DS[245]), traffic_out(DS[245]), discards_in(DS[246]), discards_out(DS[246]), errors_in(DS[246]), errors_out(DS[246]), discards_in(DS[247]), discards_out(DS[247]), errors_in(DS[247]), errors_out(DS[247]), discards_in(DS[248]), discards_out(DS[248]), errors_in(DS[248]), errors_out(DS[248]), discards_in(DS[249]), discards_out(DS[249]), errors_in(DS[249]), errors_out(DS[249]), discards_in(DS[250]), discards_out(DS[250]), errors_in(DS[250]), errors_out(DS[250]), cpu_nice(DS[269]), cpu_system(DS[270]), cpu_user(DS[271]), load_1min(DS[272]), load_15min(DS[273]), load_5min(DS[274]), mem_buffers(DS[275]), mem_cache(DS[276]), mem_free(DS[277]), hdd_free(DS[278]), hdd_used(DS[278]), hdd_free(DS[279]), hdd_used(DS[279]), hdd_free(DS[280]), hdd_used(DS[280]), hdd_free(DS[281]), hdd_used(DS[281]), hdd_free(DS[282]), hdd_used(DS[282]), hdd_free(DS[283]), hdd_used(DS[283]), Bytes_Read(DS[284]), Bytes_Written(DS[284])
    10/26/2007 12:06:04 PM - SYSTEM STATS: Time:2.2466 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/26/2007 12:05:04 PM - SYSTEM STATS: Time:2.3042 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/26/2007 12:04:04 PM - SYSTEM STATS: Time:2.2562 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/26/2007 12:03:04 PM - SYSTEM STATS: Time:2.2773 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203 RRDsProcessed:102
    10/26/2007 12:01:04 PM - SYSTEM STATS: Time:2.2428 Method:spine Processes:1 Threads:1 Hosts:7 HostsPerProcess:7 DataSources:203
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

That is not a hung poller problem. In this version, if you set the cron to 5 minutes, it will stay active for all 5 when you have a 1 minute poller interval. If you are using a 1 minute interval, it may be better to set cron to 1 minute. Don't kill the poller processes.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
denny
Posts: 14
Joined: Wed Oct 24, 2007 11:04 pm

Post by denny »

Might this be the a source of the multiple poller issue?

I currently have cron set up for every minute, which I assumed was required for the 1 minute polling. Is this not the case?

On the data source keying issue, is there something that I can to do help diagnose this?

Thanks,

Denny
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

look for multiple poller.php's running
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
denny
Posts: 14
Joined: Wed Oct 24, 2007 11:04 pm

Post by denny »

TheWitness wrote:look for multiple poller.php's running
Well, given that poller.php hangs for 5 minutes, and that cron is starting a poller.php every minute, by definition there will be multiple pollers running at the same time.

So, should I change cron to run only every 5 minutes?

Denny
Top
denny
Posts: 14
Joined: Wed Oct 24, 2007 11:04 pm

Post by denny »

denny wrote:So, should I change cron to run only every 5 minutes?
Well... there we have it. I changed cron to run every 5 minutes, and now everything looks good. No poller output table warnings, and no database errors. :)

I had a quick look at the code in poller.php, and it appears to be very focused on either a 1 minute or a 5 minute routine. If it detects a cron interval of less than 5 minutes, it tries to be smart and calculate how long to run for. With a cron interval of 5 minutes, everything is hard coded.

I'm guessing that there is a problem with the calculation of MAX_POLLER_RUNTIME. When cron is running at 1 minute, the calculation is sometimes coming out as <= 60, and sometimes as 300, which explains why the poller process would sometimes linger for 5 minutes, and sometimes it wouldn't.

As there appears to be no protection for multiple poller processes running simultaneously, it looks pretty much guaranteed to cause a problem if cron is running every minute. With cron running at every 5 minutes everything works because of the hard coded run limit of 298 seconds.

However, even if cron is set to run every 5 minutes, running the poller by hand (such as to diagnose a polling problem) could result in the same problem.

Denny
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Thanks for the feedback Denny. The process must be changed.

Larry
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
davide
Posts: 9
Joined: Fri Mar 25, 2005 9:14 am

Post by davide »

denny wrote: Well... there we have it. I changed cron to run every 5 minutes, and now everything looks good. No poller output table warnings, and no database errors.
Denny
Hello,
I just update to 0.8.7 and spine and I had same problem.

I setup cron to 5 minutes and poller interval to 1 minute. All seem works without problem.

Is this setting right?

Regards,

Davide
Davide
Top
jnelson2000
Posts: 6
Joined: Fri Oct 26, 2007 2:30 pm

Post by jnelson2000 »

how did you change cron?
Top
lukethor
Posts: 3
Joined: Thu Jul 21, 2005 8:28 am

Setting up the number of threads and process

Post by lukethor »

Is there any way to sett up the number of prcesses and threads in cacti ?

thanks
Eric
Top
Post Reply

Return to “Help: General”

Who is online

Users browsing this forum: No registered users and 2 guests