Unexplained gaps in graph data

[TheWitness]

Just the log.

[clangro]

Just the log.

There is nothing in the log for the hosts having this issue now.

I'm at my absolute wit's end with this. It just doesn't make any sense. The template is fine because 20+ devices use the template and about 6 of em are fine while the remaining are having an issue. It isn't an interface issue because some 10. devices are perfect. It's obviously a Cacti issue but even debug level logging provides no good information. Obviously the SNMP polling works as the logs show the results, it just decides not to graph it sometimes.

[TheWitness]

I'm sorry, I can't help you if you don't provide the information I request.

TheWitness

[greghaase]

Hi I read through this thread, and I am experiencing something similar.

I have gathered all of the info that has been requested, here it is:

(Image Below)



# rrdtool info rra/205/3306.rrd
filename = "rra/205/3306.rrd"
rrd_version = "0003"
step = 60
last_update = 1275081852
ds[usedReal].type = "GAUGE"
ds[usedReal].minimal_heartbeat = 120
ds[usedReal].min = 0.0000000000e+00
ds[usedReal].max = NaN
ds[usedReal].last_ds = "625008"
ds[usedReal].value = 7.5000960000e+06
ds[usedReal].unknown_sec = 0
ds[totalReal].type = "GAUGE"
ds[totalReal].minimal_heartbeat = 120
ds[totalReal].min = 0.0000000000e+00
ds[totalReal].max = NaN
ds[totalReal].last_ds = "1026792"
ds[totalReal].value = 1.2321504000e+07
ds[totalReal].unknown_sec = 0
ds[memCached].type = "GAUGE"
ds[memCached].minimal_heartbeat = 120
ds[memCached].min = 0.0000000000e+00
ds[memCached].max = NaN
ds[memCached].last_ds = "58048"
ds[memCached].value = 6.9657600000e+05
ds[memCached].unknown_sec = 0
ds[memBuffer].type = "GAUGE"
ds[memBuffer].minimal_heartbeat = 120
ds[memBuffer].min = 0.0000000000e+00
ds[memBuffer].max = NaN
ds[memBuffer].last_ds = "143092"
ds[memBuffer].value = 1.7171040000e+06
ds[memBuffer].unknown_sec = 0
ds[availSwap].type = "GAUGE"
ds[availSwap].minimal_heartbeat = 120
ds[availSwap].min = 0.0000000000e+00
ds[availSwap].max = NaN
ds[availSwap].last_ds = "1048476"
ds[availSwap].value = 1.2581712000e+07
ds[availSwap].unknown_sec = 0
ds[totalSwap].type = "GAUGE"
ds[totalSwap].minimal_heartbeat = 120
ds[totalSwap].min = 0.0000000000e+00
ds[totalSwap].max = NaN
ds[totalSwap].last_ds = "1048568"
ds[totalSwap].value = 1.2582816000e+07
ds[totalSwap].unknown_sec = 0
ds[availReal].type = "GAUGE"
ds[availReal].minimal_heartbeat = 120
ds[availReal].min = 0.0000000000e+00
ds[availReal].max = NaN
ds[availReal].last_ds = "200644"
ds[availReal].value = 2.4077280000e+06
ds[availReal].unknown_sec = 0
ds[usedSwap].type = "GAUGE"
ds[usedSwap].minimal_heartbeat = 120
ds[usedSwap].min = 0.0000000000e+00
ds[usedSwap].max = NaN
ds[usedSwap].last_ds = "92"
ds[usedSwap].value = 1.1040000000e+03
ds[usedSwap].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 500
rra[0].cur_row = 129
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[0].cdp_prep[2].value = NaN
rra[0].cdp_prep[2].unknown_datapoints = 0
rra[0].cdp_prep[3].value = NaN
rra[0].cdp_prep[3].unknown_datapoints = 0
rra[0].cdp_prep[4].value = NaN
rra[0].cdp_prep[4].unknown_datapoints = 0
rra[0].cdp_prep[5].value = NaN
rra[0].cdp_prep[5].unknown_datapoints = 0
rra[0].cdp_prep[6].value = NaN
rra[0].cdp_prep[6].unknown_datapoints = 0
rra[0].cdp_prep[7].value = NaN
rra[0].cdp_prep[7].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 600
rra[1].cur_row = 35
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[1].cdp_prep[2].value = NaN
rra[1].cdp_prep[2].unknown_datapoints = 0
rra[1].cdp_prep[3].value = NaN
rra[1].cdp_prep[3].unknown_datapoints = 0
rra[1].cdp_prep[4].value = NaN
rra[1].cdp_prep[4].unknown_datapoints = 0
rra[1].cdp_prep[5].value = NaN
rra[1].cdp_prep[5].unknown_datapoints = 0
rra[1].cdp_prep[6].value = NaN
rra[1].cdp_prep[6].unknown_datapoints = 0
rra[1].cdp_prep[7].value = NaN
rra[1].cdp_prep[7].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 700
rra[2].cur_row = 297
rra[2].pdp_per_row = 6
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 0.0000000000e+00
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = 0.0000000000e+00
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[2].cdp_prep[2].value = 0.0000000000e+00
rra[2].cdp_prep[2].unknown_datapoints = 0
rra[2].cdp_prep[3].value = 0.0000000000e+00
rra[2].cdp_prep[3].unknown_datapoints = 0
rra[2].cdp_prep[4].value = 0.0000000000e+00
rra[2].cdp_prep[4].unknown_datapoints = 0
rra[2].cdp_prep[5].value = 0.0000000000e+00
rra[2].cdp_prep[5].unknown_datapoints = 0
rra[2].cdp_prep[6].value = 0.0000000000e+00
rra[2].cdp_prep[6].unknown_datapoints = 0
rra[2].cdp_prep[7].value = 0.0000000000e+00
rra[2].cdp_prep[7].unknown_datapoints = 0
rra[3].cf = "AVERAGE"
rra[3].rows = 775
rra[3].cur_row = 251
rra[3].pdp_per_row = 24
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = 3.1355407333e+06
rra[3].cdp_prep[0].unknown_datapoints = 7
rra[3].cdp_prep[1].value = 5.1339600000e+06
rra[3].cdp_prep[1].unknown_datapoints = 7
rra[3].cdp_prep[2].value = 2.9000286667e+05
rra[3].cdp_prep[2].unknown_datapoints = 7
rra[3].cdp_prep[3].value = 7.4458773333e+05
rra[3].cdp_prep[3].unknown_datapoints = 7
rra[3].cdp_prep[4].value = 5.2423800000e+06
rra[3].cdp_prep[4].unknown_datapoints = 7
rra[3].cdp_prep[5].value = 5.2428400000e+06
rra[3].cdp_prep[5].unknown_datapoints = 7
rra[3].cdp_prep[6].value = 9.6382866667e+05
rra[3].cdp_prep[6].unknown_datapoints = 7
rra[3].cdp_prep[7].value = 4.6000000000e+02
rra[3].cdp_prep[7].unknown_datapoints = 7
rra[4].cf = "AVERAGE"
rra[4].rows = 797
rra[4].cur_row = 539
rra[4].pdp_per_row = 288
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = 3.7242394800e+07
rra[4].cdp_prep[0].unknown_datapoints = 72
rra[4].cdp_prep[1].value = 6.1607520000e+07
rra[4].cdp_prep[1].unknown_datapoints = 72
rra[4].cdp_prep[2].value = 3.5495743000e+06
rra[4].cdp_prep[2].unknown_datapoints = 72
rra[4].cdp_prep[3].value = 9.0216869667e+06
rra[4].cdp_prep[3].unknown_datapoints = 72
rra[4].cdp_prep[4].value = 6.2908560000e+07
rra[4].cdp_prep[4].unknown_datapoints = 72
rra[4].cdp_prep[5].value = 6.2914080000e+07
rra[4].cdp_prep[5].unknown_datapoints = 72
rra[4].cdp_prep[6].value = 1.1793863933e+07
rra[4].cdp_prep[6].unknown_datapoints = 72
rra[4].cdp_prep[7].value = 5.5200000000e+03
rra[4].cdp_prep[7].unknown_datapoints = 72
rra[5].cf = "MIN"
rra[5].rows = 600
rra[5].cur_row = 180
rra[5].pdp_per_row = 1
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = NaN
rra[5].cdp_prep[0].unknown_datapoints = 0
rra[5].cdp_prep[1].value = NaN
rra[5].cdp_prep[1].unknown_datapoints = 0
rra[5].cdp_prep[2].value = NaN
rra[5].cdp_prep[2].unknown_datapoints = 0
rra[5].cdp_prep[3].value = NaN
rra[5].cdp_prep[3].unknown_datapoints = 0
rra[5].cdp_prep[4].value = NaN
rra[5].cdp_prep[4].unknown_datapoints = 0
rra[5].cdp_prep[5].value = NaN
rra[5].cdp_prep[5].unknown_datapoints = 0
rra[5].cdp_prep[6].value = NaN
rra[5].cdp_prep[6].unknown_datapoints = 0
rra[5].cdp_prep[7].value = NaN
rra[5].cdp_prep[7].unknown_datapoints = 0
rra[6].cf = "MIN"
rra[6].rows = 700
rra[6].cur_row = 149
rra[6].pdp_per_row = 6
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = 6.2500800000e+05
rra[6].cdp_prep[0].unknown_datapoints = 0
rra[6].cdp_prep[1].value = 1.0267920000e+06
rra[6].cdp_prep[1].unknown_datapoints = 0
rra[6].cdp_prep[2].value = 5.8048000000e+04
rra[6].cdp_prep[2].unknown_datapoints = 0
rra[6].cdp_prep[3].value = 1.4309200000e+05
rra[6].cdp_prep[3].unknown_datapoints = 0
rra[6].cdp_prep[4].value = 1.0484760000e+06
rra[6].cdp_prep[4].unknown_datapoints = 0
rra[6].cdp_prep[5].value = 1.0485680000e+06
rra[6].cdp_prep[5].unknown_datapoints = 0
rra[6].cdp_prep[6].value = 2.0064400000e+05
rra[6].cdp_prep[6].unknown_datapoints = 0
rra[6].cdp_prep[7].value = 9.2000000000e+01
rra[6].cdp_prep[7].unknown_datapoints = 0
rra[7].cf = "MIN"
rra[7].rows = 775
rra[7].cur_row = 100
rra[7].pdp_per_row = 24
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = 6.2481140000e+05
rra[7].cdp_prep[0].unknown_datapoints = 7
rra[7].cdp_prep[1].value = 1.0267920000e+06
rra[7].cdp_prep[1].unknown_datapoints = 7
rra[7].cdp_prep[2].value = 5.7954000000e+04
rra[7].cdp_prep[2].unknown_datapoints = 7
rra[7].cdp_prep[3].value = 1.4309200000e+05
rra[7].cdp_prep[3].unknown_datapoints = 7
rra[7].cdp_prep[4].value = 1.0484760000e+06
rra[7].cdp_prep[4].unknown_datapoints = 7
rra[7].cdp_prep[5].value = 1.0485680000e+06
rra[7].cdp_prep[5].unknown_datapoints = 7
rra[7].cdp_prep[6].value = 1.8851600000e+05
rra[7].cdp_prep[6].unknown_datapoints = 7
rra[7].cdp_prep[7].value = 9.2000000000e+01
rra[7].cdp_prep[7].unknown_datapoints = 7
rra[8].cf = "MIN"
rra[8].rows = 797
rra[8].cur_row = 748
rra[8].pdp_per_row = 288
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = 6.0708000000e+05
rra[8].cdp_prep[0].unknown_datapoints = 72
rra[8].cdp_prep[1].value = 1.0267920000e+06
rra[8].cdp_prep[1].unknown_datapoints = 72
rra[8].cdp_prep[2].value = 5.7884000000e+04
rra[8].cdp_prep[2].unknown_datapoints = 72
rra[8].cdp_prep[3].value = 1.4309200000e+05
rra[8].cdp_prep[3].unknown_datapoints = 72
rra[8].cdp_prep[4].value = 1.0484760000e+06
rra[8].cdp_prep[4].unknown_datapoints = 72
rra[8].cdp_prep[5].value = 1.0485680000e+06
rra[8].cdp_prep[5].unknown_datapoints = 72
rra[8].cdp_prep[6].value = 1.7838000000e+05
rra[8].cdp_prep[6].unknown_datapoints = 72
rra[8].cdp_prep[7].value = 9.2000000000e+01
rra[8].cdp_prep[7].unknown_datapoints = 72
rra[9].cf = "MAX"
rra[9].rows = 500
rra[9].cur_row = 70
rra[9].pdp_per_row = 1
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = NaN
rra[9].cdp_prep[0].unknown_datapoints = 0
rra[9].cdp_prep[1].value = NaN
rra[9].cdp_prep[1].unknown_datapoints = 0
rra[9].cdp_prep[2].value = NaN
rra[9].cdp_prep[2].unknown_datapoints = 0
rra[9].cdp_prep[3].value = NaN
rra[9].cdp_prep[3].unknown_datapoints = 0
rra[9].cdp_prep[4].value = NaN
rra[9].cdp_prep[4].unknown_datapoints = 0
rra[9].cdp_prep[5].value = NaN
rra[9].cdp_prep[5].unknown_datapoints = 0
rra[9].cdp_prep[6].value = NaN
rra[9].cdp_prep[6].unknown_datapoints = 0
rra[9].cdp_prep[7].value = NaN
rra[9].cdp_prep[7].unknown_datapoints = 0
rra[10].cf = "MAX"
rra[10].rows = 600
rra[10].cur_row = 509
rra[10].pdp_per_row = 1
rra[10].xff = 5.0000000000e-01
rra[10].cdp_prep[0].value = NaN
rra[10].cdp_prep[0].unknown_datapoints = 0
rra[10].cdp_prep[1].value = NaN
rra[10].cdp_prep[1].unknown_datapoints = 0
rra[10].cdp_prep[2].value = NaN
rra[10].cdp_prep[2].unknown_datapoints = 0
rra[10].cdp_prep[3].value = NaN
rra[10].cdp_prep[3].unknown_datapoints = 0
rra[10].cdp_prep[4].value = NaN
rra[10].cdp_prep[4].unknown_datapoints = 0
rra[10].cdp_prep[5].value = NaN
rra[10].cdp_prep[5].unknown_datapoints = 0
rra[10].cdp_prep[6].value = NaN
rra[10].cdp_prep[6].unknown_datapoints = 0
rra[10].cdp_prep[7].value = NaN
rra[10].cdp_prep[7].unknown_datapoints = 0
rra[11].cf = "MAX"
rra[11].rows = 700
rra[11].cur_row = 352
rra[11].pdp_per_row = 6
rra[11].xff = 5.0000000000e-01
rra[11].cdp_prep[0].value = 6.2500800000e+05
rra[11].cdp_prep[0].unknown_datapoints = 0
rra[11].cdp_prep[1].value = 1.0267920000e+06
rra[11].cdp_prep[1].unknown_datapoints = 0
rra[11].cdp_prep[2].value = 5.8048000000e+04
rra[11].cdp_prep[2].unknown_datapoints = 0
rra[11].cdp_prep[3].value = 1.4309200000e+05
rra[11].cdp_prep[3].unknown_datapoints = 0
rra[11].cdp_prep[4].value = 1.0484760000e+06
rra[11].cdp_prep[4].unknown_datapoints = 0
rra[11].cdp_prep[5].value = 1.0485680000e+06
rra[11].cdp_prep[5].unknown_datapoints = 0
rra[11].cdp_prep[6].value = 2.0064400000e+05
rra[11].cdp_prep[6].unknown_datapoints = 0
rra[11].cdp_prep[7].value = 9.2000000000e+01
rra[11].cdp_prep[7].unknown_datapoints = 0
rra[12].cf = "MAX"
rra[12].rows = 775
rra[12].cur_row = 193
rra[12].pdp_per_row = 24
rra[12].xff = 5.0000000000e-01
rra[12].cdp_prep[0].value = 6.2997600000e+05
rra[12].cdp_prep[0].unknown_datapoints = 7
rra[12].cdp_prep[1].value = 1.0267920000e+06
rra[12].cdp_prep[1].unknown_datapoints = 7
rra[12].cdp_prep[2].value = 5.8056000000e+04
rra[12].cdp_prep[2].unknown_datapoints = 7
rra[12].cdp_prep[3].value = 1.5049266667e+05
rra[12].cdp_prep[3].unknown_datapoints = 7
rra[12].cdp_prep[4].value = 1.0484760000e+06
rra[12].cdp_prep[4].unknown_datapoints = 7
rra[12].cdp_prep[5].value = 1.0485680000e+06
rra[12].cdp_prep[5].unknown_datapoints = 7
rra[12].cdp_prep[6].value = 2.0064400000e+05
rra[12].cdp_prep[6].unknown_datapoints = 7
rra[12].cdp_prep[7].value = 9.2000000000e+01
rra[12].cdp_prep[7].unknown_datapoints = 7
rra[13].cf = "MAX"
rra[13].rows = 797
rra[13].cur_row = 2
rra[13].pdp_per_row = 288
rra[13].xff = 5.0000000000e-01
rra[13].cdp_prep[0].value = 6.3974906667e+05
rra[13].cdp_prep[0].unknown_datapoints = 72
rra[13].cdp_prep[1].value = 1.0267920000e+06
rra[13].cdp_prep[1].unknown_datapoints = 72
rra[13].cdp_prep[2].value = 6.2872000000e+04
rra[13].cdp_prep[2].unknown_datapoints = 72
rra[13].cdp_prep[3].value = 1.5431040000e+05
rra[13].cdp_prep[3].unknown_datapoints = 72
rra[13].cdp_prep[4].value = 1.0484760000e+06
rra[13].cdp_prep[4].unknown_datapoints = 72
rra[13].cdp_prep[5].value = 1.0485680000e+06
rra[13].cdp_prep[5].unknown_datapoints = 72
rra[13].cdp_prep[6].value = 2.1589840000e+05
rra[13].cdp_prep[6].unknown_datapoints = 72
rra[13].cdp_prep[7].value = 9.2000000000e+01
rra[13].cdp_prep[7].unknown_datapoints = 72
rra[14].cf = "LAST"
rra[14].rows = 500
rra[14].cur_row = 348
rra[14].pdp_per_row = 1
rra[14].xff = 5.0000000000e-01
rra[14].cdp_prep[0].value = NaN
rra[14].cdp_prep[0].unknown_datapoints = 0
rra[14].cdp_prep[1].value = NaN
rra[14].cdp_prep[1].unknown_datapoints = 0
rra[14].cdp_prep[2].value = NaN
rra[14].cdp_prep[2].unknown_datapoints = 0
rra[14].cdp_prep[3].value = NaN
rra[14].cdp_prep[3].unknown_datapoints = 0
rra[14].cdp_prep[4].value = NaN
rra[14].cdp_prep[4].unknown_datapoints = 0
rra[14].cdp_prep[5].value = NaN
rra[14].cdp_prep[5].unknown_datapoints = 0
rra[14].cdp_prep[6].value = NaN
rra[14].cdp_prep[6].unknown_datapoints = 0
rra[14].cdp_prep[7].value = NaN
rra[14].cdp_prep[7].unknown_datapoints = 0
rra[15].cf = "LAST"
rra[15].rows = 600
rra[15].cur_row = 134
rra[15].pdp_per_row = 1
rra[15].xff = 5.0000000000e-01
rra[15].cdp_prep[0].value = NaN
rra[15].cdp_prep[0].unknown_datapoints = 0
rra[15].cdp_prep[1].value = NaN
rra[15].cdp_prep[1].unknown_datapoints = 0
rra[15].cdp_prep[2].value = NaN
rra[15].cdp_prep[2].unknown_datapoints = 0
rra[15].cdp_prep[3].value = NaN
rra[15].cdp_prep[3].unknown_datapoints = 0
rra[15].cdp_prep[4].value = NaN
rra[15].cdp_prep[4].unknown_datapoints = 0
rra[15].cdp_prep[5].value = NaN
rra[15].cdp_prep[5].unknown_datapoints = 0
rra[15].cdp_prep[6].value = NaN
rra[15].cdp_prep[6].unknown_datapoints = 0
rra[15].cdp_prep[7].value = NaN
rra[15].cdp_prep[7].unknown_datapoints = 0
rra[16].cf = "LAST"
rra[16].rows = 700
rra[16].cur_row = 59
rra[16].pdp_per_row = 6
rra[16].xff = 5.0000000000e-01
rra[16].cdp_prep[0].value = 6.2500800000e+05
rra[16].cdp_prep[0].unknown_datapoints = 0
rra[16].cdp_prep[1].value = 1.0267920000e+06
rra[16].cdp_prep[1].unknown_datapoints = 0
rra[16].cdp_prep[2].value = 5.8048000000e+04
rra[16].cdp_prep[2].unknown_datapoints = 0
rra[16].cdp_prep[3].value = 1.4309200000e+05
rra[16].cdp_prep[3].unknown_datapoints = 0
rra[16].cdp_prep[4].value = 1.0484760000e+06
rra[16].cdp_prep[4].unknown_datapoints = 0
rra[16].cdp_prep[5].value = 1.0485680000e+06
rra[16].cdp_prep[5].unknown_datapoints = 0
rra[16].cdp_prep[6].value = 2.0064400000e+05
rra[16].cdp_prep[6].unknown_datapoints = 0
rra[16].cdp_prep[7].value = 9.2000000000e+01
rra[16].cdp_prep[7].unknown_datapoints = 0
rra[17].cf = "LAST"
rra[17].rows = 775
rra[17].cur_row = 353
rra[17].pdp_per_row = 24
rra[17].xff = 5.0000000000e-01
rra[17].cdp_prep[0].value = 6.2500800000e+05
rra[17].cdp_prep[0].unknown_datapoints = 7
rra[17].cdp_prep[1].value = 1.0267920000e+06
rra[17].cdp_prep[1].unknown_datapoints = 7
rra[17].cdp_prep[2].value = 5.8048000000e+04
rra[17].cdp_prep[2].unknown_datapoints = 7
rra[17].cdp_prep[3].value = 1.4309200000e+05
rra[17].cdp_prep[3].unknown_datapoints = 7
rra[17].cdp_prep[4].value = 1.0484760000e+06
rra[17].cdp_prep[4].unknown_datapoints = 7
rra[17].cdp_prep[5].value = 1.0485680000e+06
rra[17].cdp_prep[5].unknown_datapoints = 7
rra[17].cdp_prep[6].value = 2.0064400000e+05
rra[17].cdp_prep[6].unknown_datapoints = 7
rra[17].cdp_prep[7].value = 9.2000000000e+01
rra[17].cdp_prep[7].unknown_datapoints = 7
rra[18].cf = "LAST"
rra[18].rows = 797
rra[18].cur_row = 676
rra[18].pdp_per_row = 288
rra[18].xff = 5.0000000000e-01
rra[18].cdp_prep[0].value = 6.2500800000e+05
rra[18].cdp_prep[0].unknown_datapoints = 72
rra[18].cdp_prep[1].value = 1.0267920000e+06
rra[18].cdp_prep[1].unknown_datapoints = 72
rra[18].cdp_prep[2].value = 5.8048000000e+04
rra[18].cdp_prep[2].unknown_datapoints = 72
rra[18].cdp_prep[3].value = 1.4309200000e+05
rra[18].cdp_prep[3].unknown_datapoints = 72
rra[18].cdp_prep[4].value = 1.0484760000e+06
rra[18].cdp_prep[4].unknown_datapoints = 72
rra[18].cdp_prep[5].value = 1.0485680000e+06
rra[18].cdp_prep[5].unknown_datapoints = 72
rra[18].cdp_prep[6].value = 2.0064400000e+05
rra[18].cdp_prep[6].unknown_datapoints = 72
rra[18].cdp_prep[7].value = 9.2000000000e+01
rra[18].cdp_prep[7].unknown_datapoints = 72

Plugins Being used:
aggregate
autom8
boost
spine

Cacti Logs for this period:

05/28/2010 01:50:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out(DS[3316])
05/28/2010 01:50:22 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:50:22 PM - SPINE: Poller[0] Host[205] DS[3316] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:50:30 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:50:30 PM - SPINE: Poller[0] Host[205] DS[3313] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:50:48 PM - SYSTEM STATS: Time:46.9399 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:51:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out(DS[3316])
05/28/2010 01:51:21 PM - SPINE: Poller[0] Host[205] DS[3308] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:51:21 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:51:29 PM - SPINE: Poller[0] Host[205] DS[3307] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:51:29 PM - SPINE: Poller[0] Host[212] DS[3401] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:51:29 PM - SYSTEM STATS: Time:27.8124 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:52:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out(DS[3316])
05/28/2010 01:52:21 PM - SPINE: Poller[0] Host[205] DS[3313] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:52:21 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:52:29 PM - SPINE: Poller[0] Host[205] DS[3312] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:52:29 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:52:29 PM - SYSTEM STATS: Time:27.8223 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:53:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out(DS[3316])
05/28/2010 01:53:20 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:53:28 PM - SPINE: Poller[0] Host[212] DS[3401] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:53:29 PM - SYSTEM STATS: Time:27.8111 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:54:20 PM - SPINE: Poller[0] Host[205] DS[3308] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:54:21 PM - SPINE: Poller[0] Host[212] DS[3395] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:54:28 PM - SPINE: Poller[0] Host[205] DS[3307] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:54:29 PM - SYSTEM STATS: Time:27.8511 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:55:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out(DS[3316])
05/28/2010 01:55:20 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:55:20 PM - SPINE: Poller[0] Host[205] DS[3308] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:55:28 PM - SPINE: Poller[0] Host[212] DS[3401] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:55:28 PM - SPINE: Poller[0] Host[205] DS[3307] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:55:29 PM - SYSTEM STATS: Time:27.7979 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:56:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out(DS[3316])
05/28/2010 01:56:21 PM - SPINE: Poller[0] Host[212] DS[3395] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:56:22 PM - SYSTEM STATS: Time:20.8108 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:57:21 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:57:21 PM - SPINE: Poller[0] Host[205] DS[3308] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:57:29 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:57:29 PM - SPINE: Poller[0] Host[205] DS[3307] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:57:30 PM - SYSTEM STATS: Time:28.7687 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:57:59 PM - SYSTEM BOOST STATS: Time:28.7167 RRDUpdates:216515
05/28/2010 01:58:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: traffic_out(DS[3316])
05/28/2010 01:58:20 PM - SPINE: Poller[0] Host[205] DS[3307] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:58:21 PM - SPINE: Poller[0] Host[212] DS[3395] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:58:28 PM - SPINE: Poller[0] Host[205] DS[3316] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:58:28 PM - SYSTEM STATS: Time:27.5357 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
05/28/2010 01:59:20 PM - SPINE: Poller[0] Host[205] DS[3308] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:59:20 PM - SPINE: Poller[0] Host[212] DS[3399] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:59:28 PM - SPINE: Poller[0] Host[205] DS[3307] WARNING: SNMP timeout detected [2000 ms], ignoring host 'superhans.l'
05/28/2010 01:59:28 PM - SPINE: Poller[0] Host[212] DS[3398] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l'
05/28/2010 01:59:28 PM - SYSTEM STATS: Time:27.8142 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0

[BSOD2600]

greghaase: fix the snmp timeout for those two hosts?

[clangro]

I'm sorry, I can't help you if you don't provide the information I request.

TheWitness

Here is the log file. This is a low-level log for the past few days. Here are some of the host IDs that are still showing gaps in graph data:

225
209 - Big gap around 5-31 19:15 to 22:30
210 - Tons of big gaps
211 - Tons of big gaps here as well

If you need more info, let me know.

[greghaase]

greghaase: fix the snmp timeout for those two hosts?

When I increase the snmp timeout for these hosts it still gets ignored. Note in the log below I increased the timeout for Superhans.


06/01/2010 01:20:01 PM - SYSTEM STATS: Time:59.8469 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
06/01/2010 01:20:01 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
06/01/2010 01:20:21 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l.'
06/01/2010 01:20:29 PM - SPINE: Poller[0] Host[212] DS[3401] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l.'
06/01/2010 01:20:48 PM - SPINE: Poller[0] Host[205] DS[3307] WARNING: SNMP timeout detected [9000 ms], ignoring host 'superhans.l.'
06/01/2010 01:21:00 PM - POLLER: Poller[0] Maximum runtime of 58 seconds exceeded. Exiting.
06/01/2010 01:21:00 PM - SYSTEM STATS: Time:58.9076 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
06/01/2010 01:21:00 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
06/01/2010 01:21:20 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l.'
06/01/2010 01:21:29 PM - SPINE: Poller[0] Host[212] DS[3401] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l.'
06/01/2010 01:21:49 PM - SPINE: Poller[0] Host[205] DS[3316] WARNING: SNMP timeout detected [9000 ms], ignoring host 'superhans.l.'
06/01/2010 01:22:00 PM - POLLER: Poller[0] Maximum runtime of 58 seconds exceeded. Exiting.
06/01/2010 01:22:00 PM - SYSTEM STATS: Time:58.8989 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
06/01/2010 01:22:01 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
06/01/2010 01:22:21 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l.'
06/01/2010 01:22:29 PM - SPINE: Poller[0] Host[212] DS[3401] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l.'
06/01/2010 01:22:48 PM - SPINE: Poller[0] Host[205] DS[3308] WARNING: SNMP timeout detected [9000 ms], ignoring host 'superhans.l.'
06/01/2010 01:23:00 PM - POLLER: Poller[0] Maximum runtime of 58 seconds exceeded. Exiting.
06/01/2010 01:23:00 PM - SYSTEM STATS: Time:58.8806 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
06/01/2010 01:23:01 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
06/01/2010 01:23:20 PM - SPINE: Poller[0] Host[212] DS[3403] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l.'
06/01/2010 01:23:28 PM - SPINE: Poller[0] Host[212] DS[3401] WARNING: SNMP timeout detected [2000 ms], ignoring host 'dnsldap01.l.'
06/01/2010 01:23:47 PM - SPINE: Poller[0] Host[205] DS[3316] WARNING: SNMP timeout detected [9000 ms], ignoring host 'superhans.l.'
06/01/2010 01:24:00 PM - POLLER: Poller[0] Maximum runtime of 58 seconds exceeded. Exiting.
06/01/2010 01:24:00 PM - SYSTEM STATS: Time:59.8642 Method:spine Processes:1 Threads:16 Hosts:92 HostsPerProcess:92 DataSources:3577 RRDsProcessed:0
06/01/2010 01:24:01 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal

[clangro]

So does anyone have any ideas based on the log data? Does anyone need any more data? I can provide whatever you need, since I'm at a total loss on this.

[clangro]

Wow, guess this problem even bested the developers.

[TheWitness]

Not really, the developers are busy "developing" there are only a few of us. BSOD2600 is an "expert" and knows what your problem is. He tried to help. Your problem is more than likely that you have some hosts that don't respond too well.

This can be caused by:

1) Crappy device
2) MAX OID's too high
3) Overloaded network backbone
...

If you fix those devices, you fix cacti.

Now there are a few things you can do. Make sure you have enough cores.

1) Increase thread count
2) Increase process count
3) Both

That way if one host times out after 9 seconds (1/6) of your poller interval, no harm done. Remember to make sure mysql can handle the number of connections.

So, for you to say that your at your "wits end" reminds me of that commercial where someone misses the garbage can and everyone just stands around appalled by the fact that someone littered. It occurred to no one, that it would have been simply easier to pick up the garbage and toss it in the can and end all the whining.

So much time spent on complaining about your situation rather than getting yourself out of it, through the use of the engineering method.

Sorry,

TheWitness

[TheWitness]

Last thing, those perl scripts are taking too much time. However, I can not re-design them for you.

TheWitness

[clangro]

With all due respect, I have to disagree with your assessment for reasons I mentioned previously.

We've got Cacti monitoring devices on two different networks on 2 different interfaces. We have about 180 Juniper firewalls on the same network/interface that are all graphing perfectly. On the other network/interface we have about 30 devices, 10 of which are graphing perfectly, 20 of which have gaps that range from 1 poll being missed in 6 hours to 4 hours of no polling and everything in between. Of these 30 devices, 24 of them are exactly the same (Cisco ASA 5520s) using the exact same data queries and templates.

If it is a problem with Cacti being under too heavy of a load, how come the problem is limited to only a specific set of devices? If Cacti couldn't handle the requests, shouldn't every device be impacted?

If it is a problem with "crappy devices", how come identical devices are graphing perfectly and when the problem devices do graph they're constantly under a very light load?

If it is a problem with too many OIDs being requested, how come an identical device on the same network path is graphing fine but its high availability counterpart isn't?

If it is a problem with the network, how come I can see in the log the devices are returning data from an SNMP query and it simply isn't being graphed?

These are rhetorical questions, but I did think through all of them before I posted. I go looking for help in forums when I've feel I've exhausted all options. It's not the hardware because only a small handful of devices is impacted. It isn't the queries because devices using the same queries are working fine. It isn't the devices because identical devices are deployed and are working fine. It isn't the network since I can see in logs where the query is returning valid data but it isn't being graphed. That leads me to believe it is a problem with Cacti not graphing the data for whatever the reason, and I have no idea why.

[TheWitness]

Please use boost 4.1 that I will post to the Cacti Docs web site. I did not see that you were using boost.

TheWitess

[TheWitness]

Here it is just in case. There are two a known issues that cause major gapping. They are:

1) A major bug in MySQL 5.0.x where records are not properly deleted from Memory tables
2) A bug in boost where if a user is viewing graphs while the poller is running, those graphs may gap.

To upgrade, simply copy this file to your plugin directory, untar it and then goto Plugin Management. The upgrade will then be complete.

Regards,

TheWitness