Poller.php 1min polling - gaps in the graphs - version 0.8.7

[Jackie.Lau]

I'm having this problem where poller is running twice for a datasource on occassion. I have setup Spine to run every 60s and Cron to run every 60s. I noticed the following results for poller:

11/08/2007 04:16:06 PM - SPINE: Poller[0] Host[123] DS[3207] SCRIPT: /usr/bin/perl /var/www/html/cacti/scripts/eva_controllers.pl foobar "EVA01:XXX" "EVA01:XXY", output: evacon1:4.00 evacon2:6.00
11/08/2007 04:15:10 PM - SPINE: Poller[0] Host[123] DS[3207] SCRIPT: /usr/bin/perl /var/www/html/cacti/scripts/eva_controllers.pl foobar "EVA01:XXX" "EVA01:XXY", output: evacon1:3.00 evacon2:5.00
11/08/2007 04:15:09 PM - SPINE: Poller[0] Host[123] DS[3207] SCRIPT: /usr/bin/perl /var/www/html/cacti/scripts/eva_controllers.pl foobar "EVA01:XXX" "EVA01:XXY", output: evacon1:3.00 evacon2:5.00
11/08/2007 04:14:09 PM - SPINE: Poller[0] Host[123] DS[3207] SCRIPT: /usr/bin/perl /var/www/html/cacti/scripts/eva_controllers.pl foobar "EVA01:XXX" "EVA01:XXY", output: evacon1:4.00 evacon2:5.00
11/08/2007 04:14:08 PM - SPINE: Poller[0] Host[123] DS[3207] SCRIPT: /usr/bin/perl /var/www/html/cacti/scripts/eva_controllers.pl foobar "EVA01:XXX" "EVA01:XXY", output: evacon1:4.00 evacon2:5.00
11/08/2007 04:13:07 PM - SPINE: Poller[0] Host[123] DS[3207] SCRIPT: /usr/bin/perl /var/www/html/cacti/scripts/eva_controllers.pl foobar "EVA01:XXX" "EVA01:XXY", output: evacon1:5.00 evacon2:6.00


Notice in bold that it ran twice within a sec for the same DS.

What then happens is it then triggers this error in the logs:

11/08/2007 04:15:10 PM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1062', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (3208,'','2007-11-08 16:15:04','latency:0.00 queue:0.00'),(3209,'','2007-11-08 16:15:04','readkb:10923.00 writekb:76266.00'),(3210,'','2007-11-08 16:15:04','readlatency:9634.00 writelatency:23369.00'),(3211,'','2007-11-08 16:15:04','readreq:650.00 writereq:610.00'),(3212,'','2007-11-08 16:15:04','latency:0.00 queue:1.00'),(3213,'','2007-11-08 16:15:04','readkb:7870.00 writekb:19344.00'),(3214,'','2007-11-08 16:15:04','readlatency:9814.00 writelatency:13087.00'),(3215,'','2007-11-08 16:15:04','readreq:374.00 writereq:1691.00'),(3207,'','2007-11-08 16:15:04','evacon1:3.00 evacon2:5.00')'
11/08/2007 04:14:09 PM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1062', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (3208,'','2007-11-08 16:14:03','latency:0.00 queue:0.00'),(3209,'','2007-11-08 16:14:03','readkb:8609.00 writekb:31203.00'),(3210,'','2007-11-08 16:14:03','readlatency:6610.00 writelatency:21270.00'),(3211,'','2007-11-08 16:14:03','readreq:817.00 writereq:333.00'),(3212,'','2007-11-08 16:14:03','latency:0.00 queue:1.00'),(3213,'','2007-11-08 16:14:03','readkb:1399.00 writekb:27818.00'),(3214,'','2007-11-08 16:14:03','readlatency:9227.00 writelatency:10323.00'),(3215,'','2007-11-08 16:14:03','readreq:93.00 writereq:774.00'),(3207,'','2007-11-08 16:14:03','evacon1:4.00 evacon2:5.00')'

Which results in gaps in the 1min graphs. I setup spine for 1 process and 20 threads. The poller usually finishes within 10s. Please advise. Thanks!

[TheWitness]

Edit poller.php and set the cron interval to 300. Then setup your cron as such. This issue will be addressed in 0.8.7a to be released on Monday.

TheWitness

[koolaidrocks]

Is there an update on this issue? I've upgrade my installation to 0.8.7a but I still receive the following poller and spine errors in my logs:

11/20/2007 11:05:14 AM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1062', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (1714,'cisco_memfree','2007-11-20 11:05:14','2429996'),(1715,'cisco_memused','2007-11-20 11:05:14','2272776'),(1716,'5min_cpu','2007-11-20 11:05:14','7'),(1717,'traffic_in','2007-11-20 11:05:14','64'),(1717,'traffic_out','2007-11-20 11:05:14','64'),(1718,'traffic_in','2007-11-20 11:05:14','64'),(1718,'traffic_out','2007-11-20 11:05:14','64'),(1719,'traffic_in','2007-11-20 11:05:14','64'),(1719,'traffic_out','2007-11-20 11:05:14','64'),(1720,'traffic_in','2007-11-20 11:05:14','64'),(1720,'traffic_out','2007-11-20 11:05:14','64'),(1721,'traffic_in','2007-11-20 11:05:14','64'),(1721,'traffic_out','2007-11-20 11:05:14','64'),(1722,'traffic_in','2007-11-20 11:05:14','64'),(1722,'traffic_out','2007-11-20 11:05:14','64'),(1723,'traffic_in','2007-11-20 11:05:14','64'),(1723,'traffic_out','2007-11-20 11:05:14','64'),(1724,'traffic_in','2007-11-20 11:05:14','2588363910'),(1724,'traffic_out','2007-11-20 11:05:14','2588363910'),(1725,'traffic_in',''

And:

11/20/2007 11:06:08 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: cisco_memfree(DS[1744]), cisco_memused(DS[1745]), 5min_cpu(DS[1746]), traffic_in(DS[1747]), traffic_out(DS[1747]), traffic_in(DS[1748]), traffic_out(DS[1748]), traffic_in(DS[1749]), traffic_out(DS[1749]), traffic_in(DS[1750]), traffic_out(DS[1750]), traffic_in(DS[1751]), traffic_out(DS[1751]), traffic_in(DS[1752]), traffic_out(DS[1752]), traffic_in(DS[1753]), traffic_out(DS[1753]), traffic_in(DS[1754]), traffic_out(DS[1754]), traffic_in(DS[1755]), traffic_out(DS[1755]), traffic_in(DS[1756]), traffic_out(DS[1756]), traffic_in(DS[1757]), traffic_out(DS[1757]), traffic_in(DS[1758]), traffic_out(DS[1758]), traffic_in(DS[1759]), traffic_out(DS[1759]), traffic_in(DS[1760]), traffic_out(DS[1760]), traffic_in(DS[1761]), traffic_out(DS[1761]), traffic_in(DS[1762]), traffic_out(DS[1762]), traffic_in(DS[1763]), traffic_out(DS[1763]), traffic_in(DS[1764]), traffic_out(DS[1764]), traffic_in(DS[1765]), traffic_out(DS[1765]), traffic_in(DS[1766]), traffic_out(DS[1766]), traffic_in(DS[1767]), traffic_out(DS[1767]), traffic_in(DS[1768]), traffic_out(DS[1768]), traffic_in(DS[1769]), traffic_out(DS[1769]), traffic_in(DS[1770]), traffic_out(DS[1770]), traffic_in(DS[1771]), traffic_out(DS[1771])

I would be greatly appreciate any help!

[TheWitness]

1) Have you updated to 0.8.7a
2) Can you clear your poller cache
3) Can you check for multiple crons
4) Did you install from source or RPM?

TheWitness

[koolaidrocks]

1.) Yes, 0.8.7a w/ Plugin Arch
2.) I only have the rebuild option, so I did that.
3.) crontab -l outputs the following: */1 * * * * php /var/www/html/poller.php > /dev/null 2>&1
4.) It was installed using the source.

[TheWitness]

What is the Cron Interval in the UI set to. Make sure it matches your actual cron entry.

TheWitness

[koolaidrocks]

Yes, they match up. In reference to my first post, I've fixed the SQL error by reloading the entire install...but I still get this error every so often:

11/26/2007 05:02:03 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Potential Data Source Issues for Data Sources: cisco_memfree(DS[26]), cisco_memused(DS[27]), 5min_cpu(DS[30]), traffic_in(DS[31]), traffic_out(DS[31]), traffic_in(DS[32]), traffic_out(DS[32]), cisco_memfree(DS[33]), cisco_memused(DS[34]), cisco_tempcur(DS[35]), cisco_tempthr(DS[36]), 5min_cpu(DS[37]), traffic_in(DS[38]), traffic_out(DS[38]), traffic_in(DS[39]), traffic_out(DS[39]), traffic_in(DS[47]), traffic_out(DS[47]), traffic_in(DS[48]), traffic_out(DS[48]), traffic_in(DS[49]), traffic_out(DS[49]), traffic_in(DS[50]), traffic_out(DS[50]), traffic_in(DS[51]), traffic_out(DS[51]), 5min_cpu(DS[52]), cisco_pix_conn(DS[53]), cisco_pix_free_proc(DS[54]), cisco_pix_io_mem(DS[55]), traffic_in(DS[56]), traffic_out(DS[56]), traffic_in(DS[57]), traffic_out(DS[57]), cisco_memfree(DS[60]), cisco_memused(DS[61]), 5min_cpu(DS[62]), traffic_in(DS[63]), traffic_out(DS[63]), traffic_in(DS[64]), traffic_out(DS[64]), traffic_in(DS[65]), traffic_out(DS[65]), traffic_in(DS[66]), traffic_out(DS[66]), traffic_in(DS[67]), traffic_out(DS[67]), traffic_in(DS[68]), traffic_out(DS[68]), traffic_in(DS[69]), traffic_out(DS[69]), traffic_in(DS[70]), traffic_out(DS[70]), traffic_in(DS[71]), traffic_out(DS[71]), traffic_in(DS[72]), traffic_out(DS[72]), traffic_in(DS[73]), traffic_out(DS[73]), traffic_in(DS[74]), traffic_out(DS[74]), traffic_in(DS[75]), traffic_out(DS[75]), traffic_in(DS[76]), traffic_out(DS[76]), traffic_in(DS[77]), traffic_out(DS[77]), traffic_in(DS[78]), traffic_out(DS[78]), traffic_in(DS[79]), traffic_out(DS[79]), traffic_in(DS[80]), traffic_out(DS[80]), traffic_in(DS[81]), traffic_out(DS[81]), traffic_in(DS[82]), traffic_out(DS[82]), traffic_in(DS[83]), traffic_out(DS[83]), traffic_in(DS[84]), traffic_out(DS[84]), traffic_in(DS[85]), traffic_out(DS[85]), traffic_in(DS[86]), traffic_out(DS[86]), traffic_in(DS[87]), traffic_out(DS[87]), traffic_in(DS[88]), traffic_out(DS[88]), traffic_in(DS[89]), traffic_out(DS[89])

Any ideas?
Thanks

[TheWitness]

Spine 0.8.7a was re-released today. Please deploy it.

TheWitness

oh, btb, if you run the poller manually aka (spine or cmd.php) you will see these warnings. It's normal.

[koolaidrocks]

I applied the update and the poller wouldn't run, it would only timeout after 58 seconds. In the process of attempting to fix spine, I screwed up my NET-SNMP install...among other things.

so, I've decided to start over with a fresh install of FC7.

Thanks TheWitness, I'm sure I'll be back in a week or two for more of your excellent help.