Poller tries to add duplicate rows to poller_output table

amr · Post by **amr** » Thu Feb 12, 2009 4:10 am

Hi!

I'm experiencing a problem with poller. Cacti version 0.8.7c. Periodically i see in Cacti log next messages

02/12/2009 09:55:06 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (73, '', '2009-02-12 09:55:02', 'rate:100 min:12 avg:12 max:12')'
02/12/2009 09:55:01 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (7, 'proc', '2009-02-12 09:55:01', '184')'
02/12/2009 09:55:01 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (6, 'users', '2009-02-12 09:55:01', '0')'
02/12/2009 09:55:01 AM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (5, '', '2009-02-12 09:55:01', '1min:0.15 5min:0.18 10min:0.24')'

Error 1062 means that cacti tries to add row to table poller_output but it discards cos such primary key already exists. Could you tell me why this problem occurs?

Thank you

amr

amr · Post by **amr** » Thu Feb 12, 2009 5:02 am

Excuse me

Problem was found here http://forums.cacti.net/viewtopic.php?t ... light=1062
and in my configuration.

With best regards
amr

kop · Post by **kop** » Thu Feb 12, 2009 2:59 pm

I am having the same issue, it has recently appeared since my upgrading to 0.8.7c. I also read the posts on bad drive, but it happens on 2 machines. I checked my crontab and only have one entry. Here's a portion of my log.

02/12/2009 02:55:03 PM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '496-gps_status-2009-02-12 14:55:02' for key 'PRIMARY'', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (496,'gps_status','2009-02-12 14:55:02','1'),(497,'reg','2009-02-12 14:55:02','35'),(498,'temp','2009-02-12 14:55:02','81')'
02/12/2009 02:55:03 PM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '496-gps_status-2009-02-12 14:55:02' for key 'PRIMARY'', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (496,'gps_status','2009-02-12 14:55:02','1'),(497,'reg','2009-02-12 14:55:02','35'),(498,'temp','2009-02-12 14:55:02','81')'
02/12/2009 02:53:10 PM - RECACHE STATS: RecacheTime:1.4990 HostsRecached:2
02/12/2009 02:53:10 PM - PCOMMAND: Poller[0] Host[1283] WARNING: Recache Event Detected for Host
02/12/2009 02:53:09 PM - PCOMMAND: Poller[0] Host[1268] WARNING: Recache Event Detected for Host
02/12/2009 02:53:09 PM - SYSTEM STATS: Time:187.0891 Method:spine Processes:4 Threads:1 Hosts:1277 HostsPerProcess:320 DataSources:6114 RRDsProcessed:4402
02/12/2009 02:52:41 PM - SPINE: Poller[0] Host[1460] Hostnanme[192.168.212.144] NOTICE: HOST EVENT: Host Returned from DOWN State
02/12/2009 02:51:35 PM - SPINE: Poller[0] Host[620] DS[5711] WARNING: SNMP timeout detected [500 ms], ignoring host '192.168.212.25'
02/12/2009 02:51:35 PM - SPINE: Poller[0] Host[620] DS[5711] WARNING: SNMP timeout detected [500 ms], ignoring host '192.168.212.25'
02/12/2009 02:51:35 PM - SPINE: Poller[0] Host[620] DS[2849] WARNING: SNMP timeout detected [500 ms], ignoring host '192.168.212.25'
02/12/2009 02:51:02 PM - SPINE: Poller[0] Host[168] Hostnanme[192.168.207.101] NOTICE: HOST EVENT: Host Returned from DOWN State
02/12/2009 02:50:10 PM - SPINE: Poller[0] Host[775] Hostnanme[192.168.200.85] NOTICE: HOST EVENT: Host Returned from DOWN State
02/12/2009 02:50:03 PM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '496-gps_status-2009-02-12 14:50:02' for key 'PRIMARY'', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (496,'gps_status','2009-02-12 14:50:02','1'),(497,'reg','2009-02-12 14:50:02','35'),(498,'temp','2009-02-12 14:50:02','81')'
02/12/2009 02:50:03 PM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '496-gps_status-2009-02-12 14:50:02' for key 'PRIMARY'', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (496,'gps_status','2009-02-12 14:50:02','1'),(497,'reg','2009-02-12 14:50:02','35'),(498,'temp','2009-02-12 14:50:02','81')'
0

I have worked on this problem for a couple days now and am out of ideas.

Post by **gandalf** » Thu Feb 12, 2009 4:25 pm

kop wrote:I am having the same issue, it has recently appeared since my upgrading to 0.8.7c. I also read the posts on bad drive, but it happens on 2 machines. I checked my crontab and only have one entry. Here's a portion of my log.

02/12/2009 02:51:35 PM - SPINE: Poller[0] Host[620] DS[5711] WARNING: SNMP timeout detected [500 ms], ignoring host '192.168.212.25'
02/12/2009 02:51:35 PM - SPINE: Poller[0] Host[620] DS[5711] WARNING: SNMP timeout detected [500 ms], ignoring host '192.168.212.25'

But this clearly shows that two pollers are running. Which crontabs did you check?
Reinhard

kop · Post by **kop** » Thu Feb 12, 2009 5:44 pm

Well, I did a crontab -l and it shows one entry for the poller.
I also use webmin and under scheduled cron jobs it also only shows
one entry for the poller.
How else could it be running? Or where else to look?
I looked in /etc/cron.d and there is no entry there.

Post by **gandalf** » Fri Feb 13, 2009 2:46 pm

Check
- /etc/crontab
- /etc/cron.d/cacti
- crontab of root and cactiuser
Reinhard

kop · Post by **kop** » Fri Feb 13, 2009 3:46 pm

I found a little shell script that reports cron jobs. I've pasted the output below. I have also upgraded to 0.8.7d. I'm still getting errors. I reduced my spine processes to 2 and increased threads to 4 as this is a single cpu box. I'm preparing
a clean install on another machine and will report on it later.

--------------------------------------------------
Username: root
--------------------------------------------------
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/bin/php /usr/share/cacti/poller.php >/dev/null
29 0 * * * /etc/webmin/cron/tempdelete.pl
@weekly /etc/webmin/fsdump/backup.pl 237181234539820
--------------------------------------------------
--------------------------------------------------
Username: bin
--------------------------------------------------
no crontab for bin
--------------------------------------------------
--------------------------------------------------
Username: daemon
--------------------------------------------------
no crontab for daemon
--------------------------------------------------
--------------------------------------------------
Username: adm
--------------------------------------------------
no crontab for adm
--------------------------------------------------
--------------------------------------------------
Username: lp
--------------------------------------------------
no crontab for lp
--------------------------------------------------
--------------------------------------------------
Username: sync
--------------------------------------------------
no crontab for sync
--------------------------------------------------
--------------------------------------------------
Username: shutdown
--------------------------------------------------
no crontab for shutdown
--------------------------------------------------
--------------------------------------------------
Username: halt
--------------------------------------------------
no crontab for halt
--------------------------------------------------
--------------------------------------------------
Username: mail
--------------------------------------------------
no crontab for mail
--------------------------------------------------
--------------------------------------------------
Username: news
--------------------------------------------------
no crontab for news
--------------------------------------------------
--------------------------------------------------
Username: uucp
--------------------------------------------------
no crontab for uucp
--------------------------------------------------
--------------------------------------------------
Username: operator
--------------------------------------------------
no crontab for operator
--------------------------------------------------
--------------------------------------------------
Username: games
--------------------------------------------------
no crontab for games
--------------------------------------------------
--------------------------------------------------
Username: gopher
--------------------------------------------------
no crontab for gopher
--------------------------------------------------
--------------------------------------------------
Username: ftp
--------------------------------------------------
no crontab for ftp
--------------------------------------------------
--------------------------------------------------
Username: nobody
--------------------------------------------------
no crontab for nobody
--------------------------------------------------
--------------------------------------------------
Username: dbus
--------------------------------------------------
no crontab for dbus
--------------------------------------------------
--------------------------------------------------
Username: rpm
--------------------------------------------------
no crontab for rpm
--------------------------------------------------
--------------------------------------------------
Username: ntp
--------------------------------------------------
no crontab for ntp
--------------------------------------------------
--------------------------------------------------
Username: apache
--------------------------------------------------
no crontab for apache
--------------------------------------------------
--------------------------------------------------
Username: nscd
--------------------------------------------------
no crontab for nscd
--------------------------------------------------
--------------------------------------------------
Username: vcsa
--------------------------------------------------
no crontab for vcsa
--------------------------------------------------
--------------------------------------------------
Username: netdump
--------------------------------------------------
no crontab for netdump
--------------------------------------------------
--------------------------------------------------
Username: pcap
--------------------------------------------------
no crontab for pcap
--------------------------------------------------
--------------------------------------------------
Username: avahi
--------------------------------------------------
no crontab for avahi
--------------------------------------------------
--------------------------------------------------
Username: named
--------------------------------------------------
no crontab for named
--------------------------------------------------
--------------------------------------------------
Username: mailnull
--------------------------------------------------
no crontab for mailnull
--------------------------------------------------
--------------------------------------------------
Username: smmsp
--------------------------------------------------
no crontab for smmsp
--------------------------------------------------
--------------------------------------------------
Username: haldaemon
--------------------------------------------------
no crontab for haldaemon
--------------------------------------------------
--------------------------------------------------
Username: rpc
--------------------------------------------------
no crontab for rpc
--------------------------------------------------
--------------------------------------------------
Username: rpcuser
--------------------------------------------------
no crontab for rpcuser
--------------------------------------------------
--------------------------------------------------
Username: nfsnobody
--------------------------------------------------
no crontab for nfsnobody
--------------------------------------------------
--------------------------------------------------
Username: sshd
--------------------------------------------------
no crontab for sshd
--------------------------------------------------
--------------------------------------------------
Username: xfs
--------------------------------------------------
no crontab for xfs
--------------------------------------------------
--------------------------------------------------
Username: beagleindex
--------------------------------------------------
no crontab for beagleindex
--------------------------------------------------
--------------------------------------------------
Username: distcache
--------------------------------------------------
no crontab for distcache
--------------------------------------------------
--------------------------------------------------
Username: mysql
--------------------------------------------------
no crontab for mysql
--------------------------------------------------
--------------------------------------------------
Username: webalizer
--------------------------------------------------
no crontab for webalizer
--------------------------------------------------
--------------------------------------------------
Username: tomcat
--------------------------------------------------
no crontab for tomcat
--------------------------------------------------
--------------------------------------------------
Username: gdm
--------------------------------------------------
no crontab for gdm
--------------------------------------------------
--------------------------------------------------
Username: kpenland
--------------------------------------------------
no crontab for kpenland
--------------------------------------------------
--------------------------------------------------
Username: cacti
--------------------------------------------------
no crontab for cacti
--------------------------------------------------
--------------------------------------------------
Username: cactiuser
--------------------------------------------------
no crontab for cactiuser
--------------------------------------------------
--------------------------------------------------
Username: ntop
--------------------------------------------------
no crontab for ntop
--------------------------------------------------
--------------------------------------------------
Username: mailman
--------------------------------------------------
# At 8AM every day, mail reminders to admins as to pending requests.
# They are less likely to ignore these reminders if they're mailed
# early in the morning, but of course, this is local time...

0 8 * * * /usr/bin/python -S /usr/local/mailman/cron/checkdbs
#
# At 9AM, send notifications to disabled members that are due to be
# reminded to re-enable their accounts.
0 9 * * * /usr/bin/python -S /usr/local/mailman/cron/disabled
#
# Noon, mail digests for lists that do periodic as well as threshhold delivery.
0 12 * * * /usr/bin/python -S /usr/local/mailman/cron/senddigests
#
# 5 AM on the first of each month, mail out password reminders.
0 5 1 * * /usr/bin/python -S /usr/local/mailman/cron/mailpasswds
#
# Every 5 mins, try to gate news to mail. You can comment this one out
# if you don't want to allow gating, or don't have any going on right now,
# or want to exclusively use a callback strategy instead of polling.
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/bin/python -S /usr/local/mailman/cron/gate_news
#
# At 3:27am every night, regenerate the gzip'd archive file. Only
# turn this on if the internal archiver is used and
# GZIP_ARCHIVE_TXT_FILES is false in mm_cfg.py
27 3 * * * /usr/bin/python -S /usr/local/mailman/cron/nightly_gzip
#
# At 4:30AM daily, cull old entries from the 'bad' and 'shunt' queues.
30 4 * * * /usr/bin/python -S /usr/local/mailman/cron/cull_bad_shunt
--------------------------------------------------
--------------------------------------------------
Username: postfix
--------------------------------------------------
no crontab for postfix
--------------------------------------------------

Post by **gandalf** » Fri Feb 13, 2009 4:34 pm

That seems to check for user related crontabs only, not for system related. You may find crontab log entries at /var/log/cron
Reinhard

kop · Post by **kop** » Mon Feb 16, 2009 5:35 pm

I seem to have it fixed. I had to reduce my spine processes to 1 with 4 threads. That eliminated my sql duplicate errors. With one spine process and four threads here are my system stats. Not bad.

02/16/2009 05:26:58 PM - SYSTEM STATS: Time:116.0114 Method:spine Processes:1 Threads:4 Hosts:1252 HostsPerProcess:1252 DataSources:5926 RRDsProcessed:4445
02/16/2009 05:22:09 PM - RECACHE STATS: RecacheTime:2.0938 HostsRecached:1
02/16/2009 05:22:06 PM - SYSTEM STATS: Time:125.1505 Method:spine Processes:1 Threads:4 Hosts:1252 HostsPerProcess:1252 DataSources:5926 RRDsProcessed:4445
02/16/2009 05:16:56 PM - SYSTEM STATS: Time:114.8545 Method:spine Processes:1 Threads:4 Hosts:1252 HostsPerProcess:1252 DataSources:5926 RRDsProcessed:4447
02/16/2009 05:12:00 PM - RECACHE STATS: RecacheTime:2.1264 HostsRecached:2
02/16/2009 05:11:58 PM - SYSTEM STATS: Time:116.5003 Method:spine Processes:1 Threads:4 Hosts:1252 HostsPerProcess:1252 DataSources:5926 RRDsProcessed:4451
02/16/2009 05:07:07 PM - RECACHE STATS: RecacheTime:2.2678 HostsRecached:2
02/16/2009 05:07:05 PM - SYSTEM STATS: Time:123.9908 Method:spine Processes:1 Threads:4 Hosts:1253 HostsPerProcess:1253 DataSources:5930 RRDsProcessed:4452
02/16/2009 05:02:05 PM - RECACHE STATS: RecacheTime:2.0837 HostsRecached:1
02/16/2009 05:02:03 PM - SYSTEM STATS: Time:120.9175 Method:spine Processes:1 Threads:4 Hosts:1255 HostsPerProcess:1255 DataSources:5939 RRDsProcessed:4460
02/16/2009 04:57:18 PM - RECACHE STATS: RecacheTime:2.4283 HostsRecached:2
02/16/2009 04:57:15 PM - SYSTEM STATS: Time:133.9267 Method:spine Processes:1 Threads:4 Hosts:1256 HostsPerProcess:1256 DataSources:5943 RRDsProcessed:4463
02/16/2009 04:52:09 PM - SYSTEM STATS: Time:128.0917 Method:spine Processes:1 Threads:4 Hosts:1259 HostsPerProcess:1259 DataSources:5955 RRDsProcessed:4438
02/16/2009 04:47:09 PM - SYSTEM STATS: Time:127.1098 Method:spine Processes:1 Threads:4 Hosts:1262 HostsPerProcess:1262 DataSources:5967 RRDsProcessed:4466
02/16/2009 04:42:09 PM - RECACHE STATS: RecacheTime:2.1165 HostsRecached:1
02/16/2009 04:42:06 PM - SYSTEM STATS: Time:124.9694 Method:spine Processes:1 Threads:4 Hosts:1262 HostsPerProcess:1262 DataSources:5967 RRDsProcessed:4462

Cacti Version - 0.8.7d
Plugin Architecture - 2.4
Poller Type - Cactid v
Server Info - Linux 2.6.18-92.1.18.el5xen
Web Server - Apache/2.2.3 (CentOS)
PHP - 5.2.8
PHP Extensions - libxml, xml, wddx, tokenizer, sysvshm, sysvsem, sysvmsg, session, pcre, SimpleXML, sockets, SPL, shmop, standard, Reflection, posix, mime_magic, iconv, hash, gmp, gettext, ftp, filter, exif, date, curl, ctype, calendar, bz2, zlib, openssl, apache2handler, bcmath, dba, dbase, dom, gd, imap, json, mcrypt, mhash, mysqli, mysql, odbc, PDO, pdo_mysql, PDO_ODBC, pdo_sqlite, SQLite, xmlreader, xmlwriter, xsl, zip
MySQL - 5.1.31
RRDTool - 1.2.29
SNMP - 5.3.1
Plugins
Global Plugin Settings (settings - v0.5)
Cacti Log View (CLog - v1.1)
Device Monitoring (monitor - v0.8.2)
Update Checker (update - v0.4)
Host Info (hostinfo - v0.2)

Poller tries to add duplicate rows to poller_output table

Poller tries to add duplicate rows to poller_output table

Who is online