some data is missing from cacti graphs

[gsaray101]

Cacti server: Windows 32 bit
Cacti version: 0.8.7d
Rrdtool version: 1.2.25 (for windows)

Sporadically I am seeing that some data is missing from my cacti graphs. For example, if you look at the attached graph, I took a snapshot of a sever’s cpu utilization on 1/5/2010 and the data for 30 minute cpu utilization was there. But when I look at the same graph now, the data before November is missing.

This seems to be happening with a lot of my graphs. What cauces this? Is this an rrd file corruption and is there a way to fix it?

This is really important since reliablity of cacti is questioned. I cannot post the image here saying that I've reached the quote. Please let me know your personal email so that I can send you the image of what I am referring to.

thanks so much

[BSOD2600]

One could post the images on imageshack/other hosting sites... or delete attachments in old posts .

Anyways without a sample image its sort of hard to tell what the problem may be. If all graphs have gaps during the same time, then sounds like poller problem. If random graphs do from different devices, then could be some sporadic network latency issues where cacti though the queries took too long.

Missing data for weeks/months ago is likely due to incorrect rrdtool consolidation settings. More info in the cacti documentation site.

[gsaray101]

when I do rrdtool info, the graph in question's rrd file has a ds that I have never added to the rrd file, memAvailable MBytes. I have no idea how this data sources ended up in this rrd file. This is very strange. I already have the cacti templates for cpu, whenever there is a new server, I just applied the template and this works well. This is the response from Tobi (the creater of rrdtool) on this problem:

"the symptom you see can come from two things ...

a) you added an additional (empty) RRA into the rrd database at the
beginning of november ... whenever it gets selected for graphing
(due to matching resolution) you see this ...


b) cacti is forcing rrdtool to pick a non optimal RRA when drawing
the graph ... normally rrdtool will pick the RRA that best covers
the graphing range requested ..."

How do you explain "memAvailableMbytes" ds in cpu rrd file?

Code: Select all

D:\cacti\rra>c:\rrdtool\rrdtool info sever_cpuinfadvprocessor_26795.rrd

filename = "sb-mdcsql01_cpuinfadvprocessor_26795.rrd"

rrd_version = "0003"

step = 60

last_update = 1262972644

ds[cpuinfAdvProcessor].type = "GAUGE"

ds[cpuinfAdvProcessor].minimal_heartbeat = 120

ds[cpuinfAdvProcessor].min = 0.0000000000e+000

ds[cpuinfAdvProcessor].max = 1.0000000000e+002

ds[cpuinfAdvProcessor].last_ds = "22"

ds[cpuinfAdvProcessor].value = 8.8000000000e+001

ds[cpuinfAdvProcessor].unknown_sec = 0

ds[cpuinfAdvPrivileged].type = "GAUGE"

ds[cpuinfAdvPrivileged].minimal_heartbeat = 120

ds[cpuinfAdvPrivileged].min = 0.0000000000e+000

ds[cpuinfAdvPrivileged].max = 1.0000000000e+002

ds[cpuinfAdvPrivileged].last_ds = "2"

ds[cpuinfAdvPrivileged].value = 8.0000000000e+000

ds[cpuinfAdvPrivileged].unknown_sec = 0

ds[cpuinfAdvUser].type = "GAUGE"

ds[cpuinfAdvUser].minimal_heartbeat = 120

ds[cpuinfAdvUser].min = 0.0000000000e+000

ds[cpuinfAdvUser].max = 1.0000000000e+002

ds[cpuinfAdvUser].last_ds = "19"

ds[cpuinfAdvUser].value = 7.6000000000e+001

ds[cpuinfAdvUser].unknown_sec = 0

ds[memAvailableMBytes].type = "GAUGE"

ds[memAvailableMBytes].minimal_heartbeat = 120

ds[memAvailableMBytes].min = 0.0000000000e+000

ds[memAvailableMBytes].max = NaN

ds[memAvailableMBytes].last_ds = "U"

ds[memAvailableMBytes].value = NaN

ds[memAvailableMBytes].unknown_sec = 4

[BSOD2600]

How do you explain "memAvailableMbytes" ds in cpu rrd file?

The Data Template contained that DS, which cacti faithfully created as instructed

[gsaray101]

no, data template only has the cpu data, no memory data. Memory template is a separate data template.

[gsaray101]

plus, since I am using the same templates for all devices, if I had to modify the data template, all cpu graphs should be like this, right. I see this problem only one one servers's cpu graph right now. Just validated again, that the data template only has 3 ds defined: "cpuinfAdvPrivileged", "cpuinfAdvProcessor" and "cpuinfAdvUser".

[gsaray101]

is there a way to remove that rra from the rrd file without impacting the graph?

[gsaray101]

Once I've created the templates (data, and graph), and validated that they work, we just added 300+ servers to this cacti server. We do not modify the templates ever. this has to be a bug with cacti and really curious how did this really happen?

[BSOD2600]

if I had to modify the data template, all cpu graphs should be like this, right.

No.

At the time the rrd file is created, it uses the defined data sources in the Data Template. If the data template is changed after this, the rrd file is NOT touched in any way. So yes, at some point in the past, the data template you're using contained that memory DS too.

is there a way to remove that rra from the rrd file without impacting the graph?

There isn't any harm in leaving an unused DS in a rrd file -- it only consumes additional space. There might be a way to remove it with some custom scripts someone on the rrdtool mailing list has wrote...

[gsaray101]

There is a windows host which I noticed that history on the rrd file disappeared. I deleted the data source and recreated it. After running couple of weeks, I noticed again that cpu rrd file emptied out again. We have rra's configured for 1, 5, 30 and 60 minutes. Is the problem happening because the server is returning a bad data and this is causing problems on the rrd file? Has anybody seen this problem?

[BSOD2600]

Even with bad data, unless outside the DS min/max ranges, should be stored/graphed. Sounds like the rows/history for that data template / rrd is too small and/or using the wrong rra setting.

[gsaray101]

I have hundreds of windows servers using the same rra and data template. I only see this problem with one particular host only.

[BSOD2600]

rrdtool info show the rrd files are the same for this server compared to others which are working fine for the same data template?

[gsaray101]

they seem exactly the same. sql1 cpu graph is the problematic one.

[BSOD2600]

time to dive into the cacti.log for details. Change the logging level to medium or better. Look for your problem server. Is it timing out? snmp data queries returning data always?

I'm having a hunch this is all due to your custom rra definitions...