[SOLVED] RRD files with blank padding due to timezone

[etienne66]

I noticed that a few of my RRD files weren't being updated, but were continuing to be accessed. I tried peforming a dump via

Code: Select all

rrdtool dump device.rrd device.xml

and discovered that the time stamps of the different RRA sections were now in the future. Performing the same dump on a non-affected file looked fine. I don't know what is doing it. I have my log set to High and it is showing no errors or warnings.

By padding I mean several rows are added to each section. It appears to be about about 6 hours worth of NaN values.

My thought is that it may have something to do with something using GMT/UTC time as I am CST which is -6 hours adjustment.

Thanks,
Etienne66

Cacti Version - 0.8.7
Plugin Architecture - 1.3
Poller Type - Spine v0.8.7a
Server Info - Windows NT 5.1
Web Server - Microsoft-IIS/5.1
PHP - 5.2.4
PHP Extensions - bcmath, calendar, com_dotnet, ctype, session, filter, ftp, hash, iconv, json, odbc, pcre, Reflection, date, libxml, standard, tokenizer, zlib, SimpleXML, dom, SPL, wddx, xml, xmlreader, xmlwriter, ISAPI, gd, mysql, sockets
MySQL - 5.0.27-community-nt
RRDTool - 1.2.15
SNMP - 5.4.1
Plugins

• Global Plugin Settings (settings - v0.3)
  Thresholds (thold - v0.3.
  Host Info (hostinfo - v0.2)

[BSOD2600]

Post your cacti.log file from ONE polling cycle. Also read http://docs.cacti.net/node/283

[etienne66]

It was stable for the last 3 days. I logged in this morning and it did it after 2 polling cycles. It almost always affects the same files. I thought previously it might have been because I was adding devices, but I made no changes this morning when it happened @ 10:10 and 10:50 am. When I dumped one of the affected files it had again jumped ahead 6 hours and the new entries didn't even have any data in them. I left it alone last Thursday and it began logging data again after the future times had elapsed. And yet there still were no errors or pertinant warnings in the log files. The one warning I get is about an asset that hasn't been affected by this that does not return a good snmp response for one SNMP OID.

I set the logging to debug and ran one polling cycle like you requested. I have not tried to review it yet. I set my log level back to high.

Thanks,
Etienne66

[TheWitness]

Are you running spine 0.8.7a? If not please update. It is in announcements.

TheWitness

[etienne66]

Yes, I am using Spine 0.8.7a binary I downloaded from your posting the other day.

By the way, thanks for all of the hard work you do. It is greatly appreciated.

Thanks,
Etienne66

[TheWitness]

Make sure, until 0.8.7a of Cacti is released, that you edit poller.php and set the cron_interval = 300;

TheWitness

[etienne66]

I assume you mean to add

Code: Select all

$cron_interval = 300;

around line 71 in poller.php so that it ignores all of the testing for cron_interval. Fortunately I only need a 5 minute polling interval anyway.

I'll try this for a while and see what happens.

I managed to dump one of the rrds and move the NaNs from the bottom of the list to the top after I reset the last update field so that it matched the last true values. I then restored it and it seems to be working. I don't fell like doing that for all of them though. I wish there was an easier way to roll back an rrd file.

Thanks,
Etienne66[/i]

[etienne66]

It did it again about an hour after I updated the Poller.php file. It did not seem to do any good. I'm getting tempted to leave the logging in debug mode so I can see what the rrdupdate syntax looks like when it gets messed up.

Thanks,
Etienne66

[TheWitness]

What is your php.ini's "memory_limit" set to? Check /var/log/messages for some interesting information.

TheWitness

[etienne66]

memory_limit was equal to 64M. I went ahead and doubled it to 128M and restarted my IIS to make sure it takes affect.

I'm not sure about your /var/log/messages comment. This is a windows XP platform.

[etienne66]

Doubling the memory_limit to 128M seemed to help, but it occured again last night. I have doubled it again to 256M

I figured out your /var/log/messages was equivalent to the Windows event log. I enabled all of the logging options in the "Event Logging" section but there were no errors around the time the problem occured last night.

I have turned on Debug logging in the "Poller Specific Logging" section which I will purge occasionally until the problem occurs again. I had turned it off last night prior to the last occurence because I though it was fixed.

To be honest I'm kinda hoping the Cacti 0.8.7a will fix this problem. I never had such a problem with 0.8.6j or prior versions.

Thanks,
Etienne66

[etienne66]

I've been trying for about two weeks in debug mode to capture the rrdtool update command when the time is shifted by 6 hours and it will not cause the problem. Within a few hours of disabling debug mode it does it again.

Why would setting the logging to debug mode resolve the issue?

This really isn't a viable solution as the debug builds up the log file by about 50 megabytes per day. Perhaps if there was some logfile rotation options I could do this.

Thanks,
Etienne66

[BSOD2600]

Thats odd. you're running cacti 0.8.7a? Does it make a difference if you use cmd.php or spine poller?

[etienne66]

Yes it is Cacit 0.8.7a with PIA 1.4 and Spine 0.8.7b. I don't know if it happens with cmd.php.

I switched over to cmd.php on Low level logging. I'll try it out for a few days to see if it happens again.

Thanks,
Etienne66

[etienne66]

This problem was resolved when I either used cmd.php instead of Spine or when I upgraded cygwin1.dll to the latest version 1.5.25-7. According to the last two updates they made some changes to the timezone functions. This is probably related to all of those patches Microsoft has been doing to timezone as well.