Strange issue with perl and cacti

[Cougar]

Hi there,

I am not certain that this is a cacti issue but seem to have proven that it is not an issue with my perl script. Here is what is happening.

I am using a combination of perl script and bespoke application to read the indoor and outdoor temperature from a device connected to a serial port. This in turn saves the temperature to two separate text files.

I then have a perl script for each text file that is run by cacti and reads the value from this these two files.

Cacti graphs these two values on different graphs.

Here is my issue, lets see if I can explain this properly... at the following four times..(but not every occurrence) 18:55, 00:55, 06:55 and 12:55 the outdoor graph takes the value of the indoor graph creating a nasty spike. It seems to be quite random wether the graph will spike at these times or not, but it happens more often than not.

I have debugged my scripts and they work fine. At the times of the spikes the text files have the correct values in them. This I have checked by writing the values to a seperate text file which does not get cleared. I thought it might have been a issue with the two scripts running together with the same variables but changing this did not help either.

Has anyone come across this before? Or does anyone have any ideas what it cold be.

Your help is appreciated.
Thanks

[gandalf]

If this is a "low traffic" cacti install, please try running the poller in DEBUG mode at these times. If you catch a spike, you should find the calls to the perl scripts along with the results in log/cacti.log. This may help understanding the problem
Reinhard

[Cougar]

Yes thanks I will try that. All logic tells me its my script playing up but just a strange issue and all the checks I can think of doing seem to prove it is not my script.

Ill try to debug and let you know what I get.

[Cougar]

OK this is what I am getting.. read from the bottom...

at 18:50 and 18:55 all seems OK... then at 19:00 for some reason there is a rrdtool update to intemp_151.rrd (which is the rrd file for the outside temperature) with the value of localtemp.pl (which is the inside temperature)

I know this doesnt help much but any ideas as to why this would happen?


12/27/2006 07:00:03 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (500,'outemp','2006-12-27 19:00:02','5')"
12/27/2006 07:00:03 PM - CMDPHP: Poller[0] Host[0] DS[500] CMD: perl /var/lib/cacti/localtempOUT.pl, output: 5
12/27/2006 07:00:03 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_151.rrd --template outemp 1167246002:18.2
12/27/2006 07:00:03 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "delete from poller_output where local_data_id='154' and rrd_name='outemp' and time='2006-12-27 19:00:02'"
12/27/2006 07:00:03 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name)"
12/27/2006 07:00:03 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
12/27/2006 07:00:03 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('date',NOW())"
12/27/2006 07:00:03 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (154,'outemp','2006-12-27 19:00:02','18.2')"
12/27/2006 07:00:03 PM - CMDPHP: Poller[0] Host[0] DS[154] CMD: perl /var/lib/cacti/localtemp.pl, output: 18.2
...
12/27/2006 06:55:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (500,'outemp','2006-12-27 18:55:01','4.9')"
12/27/2006 06:55:01 PM - CMDPHP: Poller[0] Host[0] DS[500] CMD: perl /var/lib/cacti/localtempOUT.pl, output: 4.9
12/27/2006 06:55:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (154,'outemp','2006-12-27 18:55:01','18.1')"
12/27/2006 06:55:01 PM - CMDPHP: Poller[0] Host[0] DS[154] CMD: perl /var/lib/cacti/localtemp.pl, output: 18.1
...
12/27/2006 06:50:02 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (500,'outemp','2006-12-27 18:50:01','4.9')"
12/27/2006 06:50:02 PM - CMDPHP: Poller[0] Host[0] DS[500] CMD: perl /var/lib/cacti/localtempOUT.pl, output: 4.9
12/27/2006 06:50:02 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (154,'outemp','2006-12-27 18:50:01','18')"
12/27/2006 06:50:02 PM - CMDPHP: Poller[0] Host[0] DS[154] CMD: perl /var/lib/cacti/localtemp.pl, output: 18

[gandalf]

12/27/2006 06:50:02 PM - CMDPHP: Poller[0] Host[0] DS[154] CMD: perl /var/lib/cacti/localtemp.pl, output: 18
12/27/2006 06:50:02 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (154,'outemp','2006-12-27 18:50:01','18')"

These two always belong together. The script localtemp.pl is associated with DS[154]. From the insert statement, we learn that the data source name is "outemp". Unfortunately, you've deleted the rrdtool update statement belonging to this pair

12/27/2006 06:50:02 PM - CMDPHP: Poller[0] Host[0] DS[500] CMD: perl /var/lib/cacti/localtempOUT.pl, output: 4.9
12/27/2006 06:50:02 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id,rrd_name,time,output) values (500,'outemp','2006-12-27 18:50:01','4.9')"

And these two belong together as well. Unfortunately, the data source name is "outemp"; same as above. This does not confuse cacti, but you should not become confused about this.

Please retry but keep the rrdtool update statements for DS[154] and DS[500]

Reinhard

[Cougar]

Yes I understand that thanks. Something is fishy with the database though because the Inside Teperature internal variable should be intemp. For outside it should be outemp (misstyped outtemp)

Looking at the updates something quite strange happens at 19:00. The file intemp_151.rrd which is for the outside temperature (i should really recreate my rrd's to better names) gets updated 3 times. The 1st time with the WRONG value which is inside temperature and the next two time with the correct temperature, but the wrong value is shown in the graph.

Rather than try and figure out how and why it is doing that? can you tell me if there is an easy way to flush my database to remove all data and then start again. I will then rename my rrds correctly and see if the issue persists.. I only have about 8 rrds running so this will not be a big task to recreate. But I dont want to start a cacti install from scratch. (the reason i dont want to simply delete it all from within cacti is that my debug log shows many old rrd's still trying to update even though they are no longer in cacti or even exist in the rra folder... seems to be only in the database)

12/27/2006 07:00:10 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_150.rrd --template intemp 1167246002:18.2
12/27/2006 07:00:10 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_151.rrd --template outemp 1167246002:5
12/27/2006 07:00:04 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_151.rrd --template outemp 1167246002:5
12/27/2006 07:00:03 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_151.rrd --template outemp 1167246002:18.2
...
12/27/2006 06:55:07 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_150.rrd --template intemp 1167245701:18.1
12/27/2006 06:55:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_151.rrd --template outemp 1167245701:4.9
...
12/27/2006 06:50:08 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_150.rrd --template intemp 1167245401:18
12/27/2006 06:50:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/intemp_151.rrd --template outemp 1167245401:4.9

[gandalf]

Simply delete the database (may be sth like delete database cacti; from mysql) and re-import the cacti.sql as stated in the installation procedure. Do not forget to delete all files in the rra folder. That should do.
Reinhard

[Cougar]

OK ill try that also.. in the meantime I have actually deleted all my DIMs and DSs and recreated them with unique internal vars and editf my existing RRDs to have unique vars also and it has gone past one "timeframe" without spiking. Next one is in 50 mins but ill be asleep! Hopefully that has sorted it... After that ill look at cleaning out my database and starting completely fresh also..

Thanks for the help! its is much apreciated!