Data Template change breaks all data sources

JMoMo · Post by **JMoMo** » Sat Nov 05, 2005 6:25 am

I've got a horrific problem with Cacti. This is effecting 100+ hosts. Polling has stopped and I'm taking it in the pooper.

The symptoms are that ALL pre existing data sources for a particular Data Template have stopped updating. I've verified the timestamp on the RRD for each, and they have not been updated since the problem first occurred. Another symptom is that on graphs_new.php, when creating new graphs, the Data Query items are no longer grayed out like they were previously and should be.

In this particular case, the Data Template is "Interface - Traffic", which uses the "traffic_in" and "traffic_out" DS from the "SNMP - Interface Statistics" Data Query "In/Out Bits" Associated Graph/Data Template.

I believe that the cause of the symptoms was that an item on the Data Template was changed. When it was saved, everything went bad. Specifically, I noticed that the "Maximum Value" field had an extra 0 added to the existing value by another Cacti user. I've since reverted to it's previous value.

The graphs still draw, but it's obvious that polling has stopped and they are not being updated with new data. Very interesting is that I can go through the graph creation process and create a new graph, which will then start getting populated with polled data. The old graph will continue to not get any new data.

The old Data Source IS listed for the hosts. It's Selected Data Template is correct. The data_template_rrd table "looks" okay.

I've done everything that I could think of though the HTTP UI to fix the problem. After that, I went into mysql and started selecting around looking for the problem, but I don't see anything obvious. It all looks good to me, so I'm totally lost. If I knew where to look, that would help a lot. I'm not sure which tables to check or what to join.

My Cacti version when the problem happened was 0.8.6f fully patched, now upgraded to 0.8.6g and patched. Platform is Red Hat Enterprise 3.

Note that this looks somewhat similar to the following Cacti bugs;
http://bugs.cacti.net/view.php?id=498
http://bugs.cacti.net/view.php?id=357

I'll provide any information requested.

I need to be able to repair this ASAP. I'm loosing stats and that's not good at all.

I've created bug 614 for this issue;
http://bugs.cacti.net/view.php?id=614

Maybe someone in the community can provide some guidance though.

And yes, it is 4AM.

Post by **TheWitness** » Sat Nov 05, 2005 6:58 am

Hello,

Could you please edit the data template and post the graphic here.

TheWitness

JMoMo · Post by **JMoMo** » Sat Nov 05, 2005 1:46 pm

Here is that screenshot of the Data Template.

Post by **TheWitness** » Sat Nov 05, 2005 1:48 pm

Run 1 pass of the poller in DEBUG mode and post Logfile.

TheWitness

Post by **TheWitness** » Sat Nov 05, 2005 1:49 pm

Also, edit one of your data sources and post the DATA SOURCE debug info.

TheWitness

JMoMo · Post by **JMoMo** » Sat Nov 05, 2005 2:22 pm

Here is additional info about my system. I'll grab that stuff that you requested here in a minute. I ran DEBUG logs last night and I noticed that the traffic_in and traffic_out DS items were not even in the log. I'll run again today with at least one new DS that works and of couse all the rest that dont.

user@host-->rpm -qa|egrep -i "cacti|php|rrdtool|mysql"
perl-DBD-MySQL-2.1021-3
mysql-3.23.58-15.RHEL3.1
mysql-server-3.23.58-1.72.1
php-ldap-4.3.2-14.ent.ker.rhel3
cacti-docs-0.8.5-1.a.ker.rhel3
php-4.3.2-14.ent.ker.rhel3
rrdtool-1.2.8-1.1.el3.test
mysql-bench-3.23.58-15.RHEL3.1
php-mysql-4.3.2-14.ent.ker.rhel3
php-snmp-4.3.2-14.ent.ker.rhel3
perl-rrdtool-1.2.8-1.1.el3.test
cacti-0.8.6g-1

Post by **TheWitness** » Sat Nov 05, 2005 2:33 pm

EDIT a Traffic Data Source and post a graphic of your Data Source DEBUG.

TheWitness

JMoMo · Post by **JMoMo** » Sat Nov 05, 2005 3:03 pm

Attached is the log file. Note that if you grep through it, you will only find "traffic" DS items associated with a host named bubbles. This is a graph that I added this morning, since the last time that I edited the Data Template. Notice that no other traffic_in or traffic_out DS items are there.

If I was to modify that Data Template, then this one would break too.

--

Unrelated: Damn it; "The Extension bz2 is not allowed"

JMoMo · Post by **JMoMo** » Sat Nov 05, 2005 3:14 pm

Here are those Data Source debug images. I don't think they will be helpful. I didn't see any difference between the bad/broken DS and the good/working DS, other than the Title variable isn't being substituted in the broken one.

Post by **TheWitness** » Sat Nov 05, 2005 3:25 pm

last thing, run "php poller.php" from the command line and post the output. If none of the data sources are in the poller cache, of course you don't have to do this.

TheWitness

JMoMo · Post by **JMoMo** » Sat Nov 05, 2005 3:34 pm

I'll skip on your last request, because I'm almost certain that the items are not in the poller cache. Actually, I have some other info that might be useful from the database.

--

I don't want to throw you off with stuff that isn't relevant, but I thought this was interesting;

From the poller_item table, the broken DS items are not there. However, the new DS items for the new graph that I made this morning is there. The new DS local_data_id is 5618

The old broken DS local_data_id is 697 and it's on the data_template_data table.

See text file.

Post by **TheWitness** » Sat Nov 05, 2005 3:37 pm

Take one host and edit and resave it and see if the items re-appear in the poller_items table. Let me know, I will write a quick script to resave your hosts if this fixes it.

TheWitness

JMoMo · Post by **JMoMo** » Sat Nov 05, 2005 3:52 pm

I edit and saved two different hosts with no positive effect. I disabled one and flipped a few switches and things, and then restored it to it's original configuration, and saved a few times, but nothing good happened.

I noticed that the poller_cache table was cleared for that host when I disabled it. The data was then repopulated when I enabled the host. This is probably expected behavior.

Post by **TheWitness** » Sat Nov 05, 2005 3:53 pm

Any chance of remote access? Can you dump your DB and e-mail?

TheWitness

JMoMo · Post by **JMoMo** » Sat Nov 05, 2005 3:59 pm

I'll dump my DB and send it to you.

Data Template change breaks all data sources

Data Template change breaks all data sources

Who is online