Rebuild Poller Cache causes one graph to stop updating

[ciscobox]

G'day!

I was doing some tests on 10-sec, 30-sec and 1-min polling this afternoon. I have created a bunch of RRAs for this intervals.
All is well, until I rebuild the poller cache - System Utilities->Rebuild Poller Cache.
I don't know what really happened, I enabled high poller logging level but I cannot point out what is causing this behavior.
Another thing is, only the lowest numbered Data source ID was affected and stop updating the graph.

Here's the working sample snaphots, filtered the graph view in Last Hour:
10-sec polling voip router:


1-min polling mpls router:


Here's the data source for the 1-min polling mpls router:


Now here is the snaphot after I rebuild the poller cache:
10-sec polling voip router:


1-min polling mpls router:



You see, the one with the lowest ID Data source (the graph that shows F0/0, Data source ID 154) stops from updating the graph.
I have done it 5 times and it really gives a pattern wherein upon rebuilding the poller cache, only the lowest ID Data source in 1-min polling graphs is the one that stop from updating.

If I'm going to implement the 10-sec polling interval for our voip WAN links, together with 1-min interval on some of our data mpls links, then I have no choice but to stop using the rebuild poller cache script, for it would kill my existing 1-min graph on the mpls routers.

Using cacti/spine .8.7i.
Only the watermark plugins installed.

Let me know guys if you need more info i.e logs, snapshots..

[ciscobox]

BTW, rebuilding the poller cache again doesn't work as well as restarting the cacti box.
I also tried manually deleting the affected graph, retaining the data source and then creating the graph again using the same data source but still doesn't work.

This is the only logs that I think is relevant to the data source ID 154:
08/18/2012 09:59:10 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_in_1min(DS[154])
08/18/2012 09:59:01 PM - SPINE: Poller[0] Host[24] TH[1] DS[154] SNMP: v2: 10.110.9.1, dsname: traffic_in_1min, oid: .1.3.6.1.2.1.2.2.1.10.1, value: 363441505
08/18/2012 09:58:50 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out_1min(DS[154])
08/18/2012 09:58:41 PM - SPINE: Poller[0] Host[24] TH[1] DS[154] SNMP: v2: 10.110.9.1, dsname: traffic_out_1min, oid: .1.3.6.1.2.1.2.2.1.16.1, value: 219089909
08/18/2012 09:58:10 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_in_1min(DS[154])
08/18/2012 09:58:01 PM - SPINE: Poller[0] Host[24] TH[1] DS[154] SNMP: v2: 10.110.9.1, dsname: traffic_in_1min, oid: .1.3.6.1.2.1.2.2.1.10.1, value: 362978193
08/18/2012 09:57:50 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out_1min(DS[154])
08/18/2012 09:57:41 PM - SPINE: Poller[0] Host[24] TH[1] DS[154] SNMP: v2: 10.110.9.1, dsname: traffic_out_1min, oid: .1.3.6.1.2.1.2.2.1.16.1, value: 218897297
08/18/2012 09:57:10 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_in_1min(DS[154])
08/18/2012 09:57:01 PM - SPINE: Poller[0] Host[24] TH[1] DS[154] SNMP: v2: 10.110.9.1, dsname: traffic_in_1min, oid: .1.3.6.1.2.1.2.2.1.10.1, value: 362700757
08/18/2012 09:56:50 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out_1min(DS[154])
08/18/2012 09:56:41 PM - SPINE: Poller[0] Host[24] TH[1] DS[154] SNMP: v2: 10.110.9.1, dsname: traffic_out_1min, oid: .1.3.6.1.2.1.2.2.1.16.1, value: 218733671
08/18/2012 09:56:10 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_in_1min(DS[154])
08/18/2012 09:56:01 PM - SPINE: Poller[0] Host[24] TH[1] DS[154] SNMP: v2: 10.110.9.1, dsname: traffic_in_1min, oid: .1.3.6.1.2.1.2.2.1.10.1, value: 362420626
08/18/2012 09:55:50 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out_1min(DS[154])
08/18/2012 09:55:41 PM - SPINE: Poller[0] Host[24] TH[1] DS[154] SNMP: v2: 10.110.9.1, dsname: traffic_out_1min, oid: .1.3.6.1.2.1.2.2.1.16.1, value: 218612212
08/18/2012 09:55:10 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_in_1min(DS[154])


Now, when I run the "poller_output_empty.php" in the cli directory, here's the output:
cacti# php poller_output_empty.php
There were 0, RRD updates made this pass
cacti#

But the graph is still not updating. Tried rebooting the cacti box still no luck..

[ciscobox]

edit.

[gandalf]

Please show data source definitions for this item
R.

[ciscobox]

Please show data source definitions for this item
R.

I will try to rebuild the topology later and hopefully recreate the same issue, then post the details here.

But as far as i know, I didn't mess with the data sources. I just keep on deleting (both data source and graphs), and recreating graphs on the same interfaces - f0/0, f0/0.9, f0/0.151, on all the devices during my testing. And as we all know, the data sources will just increment the ID number everytime we create a new graph for a device (Devices->Create new graphs...).

Or buddy, if you mean the data templates:
For the 10 sec polling, I set the "step" to a value of 10 and 20 for the heartbeat.
For the 1 min polling, step = 60 and heartbeat = 120.