CACTID: Poller[0] ERROR: Problem with MySQL: Duplicate entry

[jagbarcelo]

I have upgraded from cactid-0.8.6g-pre-beta-win32 to cactid-0.8.6g. Now the following messages (lots of them) appear on the cmd window doing the 5 minutes-poll.

Code: Select all

...
02/16/2006 01:30:04 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-5:12' for key 1'
02/16/2006 01:30:04 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-5:12' for key 1'
02/16/2006 01:30:04 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-5:12' for key 1'
02/16/2006 01:30:04 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-5:12' for key 1'
02/16/2006 01:30:04 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-5:12' for key 1'
02/16/2006 01:30:04 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-5:12' for key 1'
02/16/2006 01:30:04 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-5:12' for key 1'
02/16/2006 01:30:04 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-5:12' for key 1'
02/16/2006 01:30:09 PM - CACTID: Poller[0] Host[14] DS[105] WARNING: Result from SCRIPT not valid. Partial Result: ...
02/16/2006 01:30:11 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-17:12' for key 1'
02/16/2006 01:30:11 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-17:12' for key 1'
02/16/2006 01:30:11 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-17:12' for key 1'
02/16/2006 01:30:11 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-17:12' for key 1'
02/16/2006 01:30:11 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-17:12' for key 1'
02/16/2006 01:30:11 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-17:12' for key 1'
02/16/2006 01:30:11 PM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '0-1-17:12' for key 1'
...

The rrd files are updated and the graphs are shown, so everything seems to run ok, but these ERROR messages are at least suspecting. What do they mean? How can I get rid of them?

My config:
* Windows 2003 SP1 + IIS 6.0
* PHP Version 5.1.1
* cacti-0.8.6h
* cactid-0.8.6g
* MySQL 5.0.17-nt

Regards.

[BSOD2600]

1) mysql -u root -p cacti
>truncate table poller_output;
>quit

2) mysqlcheck --auto-repair --databases cacti

[time]

I'm also getting this error and tried the steps above without success - still getting the error. The output didn't give any errors:

truncate table poller_output;
Query OK, 0 rows affected (0.00 sec)

and

mysqlcheck --auto-repair --databases cacti
cacti.cdef OK
cacti.cdef_items OK
cacti.colors OK
cacti.data_input OK
cacti.data_input_data OK
cacti.data_input_fields OK
cacti.data_local OK
cacti.data_template OK
cacti.data_template_data OK
cacti.data_template_data_rra OK
cacti.data_template_rrd OK
cacti.graph_local OK
cacti.graph_template_input OK
cacti.graph_template_input_defs OK
cacti.graph_templates OK
cacti.graph_templates_gprint OK
cacti.graph_templates_graph OK
cacti.graph_templates_item OK
cacti.graph_tree OK
cacti.graph_tree_items OK
cacti.host OK
cacti.host_graph OK
cacti.host_snmp_cache OK
cacti.host_snmp_query OK
cacti.host_template OK
cacti.host_template_graph OK
cacti.host_template_snmp_query OK
cacti.poller OK
cacti.poller_command OK
cacti.poller_item OK
cacti.poller_output OK
cacti.poller_reindex OK
cacti.poller_time OK
cacti.rra OK
cacti.rra_cf OK
cacti.settings OK
cacti.settings_graphs OK
cacti.settings_tree OK
cacti.snmp_query OK
cacti.snmp_query_graph OK
cacti.snmp_query_graph_rrd OK
cacti.snmp_query_graph_rrd_sv OK
cacti.snmp_query_graph_sv OK
cacti.user_auth OK
cacti.user_auth_perms OK
cacti.user_auth_realm OK
cacti.user_log OK
cacti.version OK

[BSOD2600]

Same error occur when you use cmd.php?

[time]

CMD.php times out with the number of devices / data sources I've got, but I tried it and still get similar errors and warnings:

02/17/2006 03:03:56 PM - PCOMMAND: Poller[0] Host[37] WARNING: Recache Event Detected for Host
02/17/2006 03:03:51 PM - PCOMMAND: Poller[0] Host[35] WARNING: Recache Event Detected for Host
02/17/2006 03:03:38 PM - PCOMMAND: Poller[0] Host[24] WARNING: Recache Event Detected for Host
02/17/2006 03:03:32 PM - PCOMMAND: Poller[0] Host[21] WARNING: Recache Event Detected for Host
02/17/2006 03:03:06 PM - PCOMMAND: Poller[0] Host[191] WARNING: Recache Event Detected for Host
02/17/2006 03:02:06 PM - PCOMMAND: Poller[0] Host[17] WARNING: Recache Event Detected for Host
02/17/2006 03:02:02 PM - PCOMMAND: Poller[0] Host[162] WARNING: Recache Event Detected for Host
02/17/2006 03:01:58 PM - PCOMMAND: Poller[0] Host[159] WARNING: Recache Event Detected for Host
02/17/2006 03:01:53 PM - PCOMMAND: Poller[0] Host[156] WARNING: Recache Event Detected for Host
02/17/2006 03:01:49 PM - PCOMMAND: Poller[0] Host[155] WARNING: Recache Event Detected for Host
02/17/2006 03:01:15 PM - PCOMMAND: Poller[0] Host[154] WARNING: Recache Event Detected for Host
02/17/2006 03:01:08 PM - PCOMMAND: Poller[0] Host[153] WARNING: Recache Event Detected for Host
02/17/2006 03:00:55 PM - PCOMMAND: Poller[0] Host[12] WARNING: Recache Event Detected for Host
02/17/2006 03:00:45 PM - PCOMMAND: Poller[0] Host[121] WARNING: Recache Event Detected for Host
02/17/2006 03:00:37 PM - PCOMMAND: Poller[0] Host[119] WARNING: Recache Event Detected for Host
02/17/2006 03:00:32 PM - CMDPHP: Poller[0] ASSERT: '429496700<429496700' failed. Recaching host '{hostname}', data query #1
02/17/2006 03:00:31 PM - PCOMMAND: Poller[0] Host[117] WARNING: Recache Event Detected for Host
02/17/2006 03:00:20 PM - PCOMMAND: Poller[0] Host[115] WARNING: Recache Event Detected for Host
02/17/2006 03:00:12 PM - PCOMMAND: Poller[0] Host[113] WARNING: Recache Event Detected for Host
02/17/2006 03:00:06 PM - PCOMMAND: Poller[0] Host[111] WARNING: Recache Event Detected for Host
02/17/2006 03:00:02 PM - PCOMMAND: Poller[0] Host[109] WARNING: Recache Event Detected for Host
02/17/2006 03:00:00 PM - RECACHE STATS: RecacheTime:305.7231 HostsRecached:29
02/17/2006 03:00:00 PM - PCOMMAND: Poller[0] ERROR: Poller Command processing timed out after processing 'Array'
02/17/2006 02:59:59 PM - PCOMMAND: Poller[0] Host[108] WARNING: Recache Event Detected for Host
02/17/2006 02:59:55 PM - PCOMMAND: Poller[0] Host[106] WARNING: Recache Event Detected for Host
02/17/2006 02:59:54 PM - SYSTEM STATS: Time:293.0418 Method:cmd.php Processes:6 Threads:N/A Hosts:169 HostsPerProcess:29 DataSources:5774 RRDsProcessed:2146
02/17/2006 02:59:54 PM - POLLER: Poller[0] Maximum runtime of 292 seconds exceeded. Exiting.

I did notice one difference though.. With the cactid poller I never got any lines with the recache stats like this:

02/17/2006 03:00:00 PM - RECACHE STATS: RecacheTime:305.7231 HostsRecached:29

After a couple of CMD.php runs I've changed it back to cactid and it seems a lot better:

02/17/2006 03:15:35 PM - RECACHE STATS: RecacheTime:0.1562 HostsRecached:1
02/17/2006 03:15:35 PM - PCOMMAND: Poller[0] Host[74] WARNING: Recache Event Detected for Host
02/17/2006 03:15:34 PM - SYSTEM STATS: Time:33.2910 Method:cactid Processes:1 Threads:6 Hosts:169 HostsPerProcess:169 DataSources:5774 RRDsProcessed:3115

But it still wants to recache that one host. It is the same as the one in the log above where it said:

02/17/2006 03:00:32 PM - CMDPHP: Poller[0] ASSERT: '429496700<429496700' failed. Recaching host '{hostname}', data query #1

[time]

Well it looks like swapping to cmd.php and back to cactid fixed the problem except for this host. Interestingly enough I didn't have the same problem with the previous version of cactid.

[BSOD2600]

Interesting running cmd.php fixed your problems (mostly). TheWitness will need to investigate further why cactid was not functioning as it should.

[zsnyder]

I'm seeing this same behavior. Unfortunately for me, switching to cmd.php and back to cactid doesn't resolve the issue.

In addition to seeing over 1,200 'Duplicate entry' errors per polling cycle with cactid, I'm also seeing Recache Events. I didn't see those while running cmd.php, either.

Any word on a fix for this?

[TheWitness]

These duplicate entries should be eliminated after the second poll with Cactid. For some reason, when switching between cmd.php and cactid, the duplicate entries appear, but should only appear for 1 polling cycle.

The reason that they appear is part of the recaching. In addition, prior to Cactid 0.8.6g, the recaching method was broken for some reason. Therefore, the update to 0.8.6g will show these for the first time.

In addition, if the error's don't go away, there is a bug in some hosts where the sysUptime stops updating after some period of time. This causes a recache for every polling cycle.

I have prepared a patch to ignore this situation and it is due to release shortly as Cactid 0.8.6h.

TheWitness

[zsnyder]

Well, these errors certainly don't go away after one polling cycle. In fact, If I leave Cactid running, I see nearly 1,200 of these messages EACH polling cycle. I will take a look to see if sysUptime is staying static, but I suspect that's probably not the case for such a huge number of hosts.

In the meantime, I'll continue to use cmd.php and anxiously await Cactid 0.8.6h. Any beta version that I could help you test?

[TheWitness]

Windows or Unix variant?

TheWitness

[amoeba]

Same problem here; running current version on CentOS4

Issue only on select hosts; will try to identify what's different about them.

Eagerly awaiting update...

[amoeba]

Some of our Pix'es were running 7.0(2)4 - updating them to 7.1(1) resolved the problem here.

[zsnyder]

I'm running Fedora Core 4.

[TheWitness]

This problem has been corrected in SVN. If you are interested, please check out the source files and compile. We should be releasing soon. I have asked to possibly include one additional change, but it's fairly minor.

Note: BRANCH_0_8_6

TheWitness