Poller Output Table not Empty

[Jericon]

First and foremost, the issue I am experiencing sounds exactly like what is in an old topic: http://forums.cacti.net/viewtopic.php?f ... 5&start=75 However I did not want to drag that topic up.

I have 21 entries that are "stuck" in my poller_output table. If I delete them from the table, they repopulate the next time around. These are for the same Data Sources. I have tried using both memory and myisam tables. I have tried using both spine and cmd.php. The same issue occurs no matter what.

All items are missing the "rrd_name" and have the same poller output. The DS's in question are actually all from better-cacti-templates.

Code: Select all

 
[mysql-cacti 46]>select * from poller_output order by local_data_id asc limit 5; 
+---------------+----------+---------------------+--------+
| local_data_id | rrd_name | time                | output |
+---------------+----------+---------------------+--------+
|          3539 |          | 2011-10-28 03:26:02 | MySQL: |
|          3541 |          | 2011-10-28 03:26:02 | MySQL: |
|          3543 |          | 2011-10-28 03:26:02 | MySQL: |
|          3545 |          | 2011-10-28 03:26:02 | MySQL: |
|          3547 |          | 2011-10-28 03:26:02 | MySQL: |
+---------------+----------+---------------------+--------+

Any ideas what might be causing this? I find that I am able to create and use some hosts without issues, while others have problems. There are no differences between the hosts short of their hostname and Ip address. I am using a custom cli script that calls the cacti cli's to add hosts with the same templates.

[Jericon]

I just made an interesting find. It *always* happens with 2 certain hosts. The hostnames are the same as the others, just different numbers.

Good:
db1001.dc1
db1002.dc1
db1021.dc1
db2001.dc2
db2002.dc2
db2021.dc2

Bad:
db1011.dc1
db2011.dc2

Obviously the hostnames are a little sanitized, but the numbers are the same. The xx11 hosts have issues, the others do not.

[TheWitness]

I suspect that 'maybe' you are running out of connections. The output of 'MySQL:' appears quite strange. Is this script or script server output?

TheWitness

[Jericon]

I suspect that 'maybe' you are running out of connections. The output of 'MySQL:' appears quite strange. Is this script or script server output?

TheWitness

How would I determine that?

I did, however, check my max_used_connections vs max_connections limit both on my cacti db, and the db servers that are being polled and I am no where near the limit. My monitored db servers have a limit of 5k connections, and if I hit that I'd be in bad shape (my app would be down). My cacti db has a limit of 1000 connections and is only using 186 at max.

[Jericon]

I finally figured this out. The xx11 servers are in a separate replication pod. One of my associates populated it with a very old backup that did not have the right host permissions for the cacti user in mysql.