Cacti Graphs -nan after upgrade from 0.8.7e to 0.8.7h

[chrisc543]

Hello, I upgraded from 0.8.7e to 0.8.7h and we are using spine and it seems like it is working but the graphs are not populating. We have lots of entry's in our poller cache but it is not writing to the graphs. I am not seeing any errors in the cacti log even at debug level. Here is a snippit of log:

02/08/2013 04:35:39 PM - WEBLOG: Poller[0] CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start=1360361138 --end=1360362938 --title='HP Procurve Switch 2 - Traffic - 48' --rigid --base=1000 --height=120 --width=500 --alt-autoscale-max --lower-limit=0 COMMENT:"From 2013/02/08 16\:05\:38 To 2013/02/08 16\:35\:38\c" COMMENT:" \n" --vertical-label='bits per second' --slope-mode --font TITLE:10: --font AXIS:7: --font LEGEND:8: --font UNIT:7: DEF:a="/usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_177.rrd":traffic_in:AVERAGE DEF:b="/usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_177.rrd":traffic_out:AVERAGE CDEF:cdefa=a,8,* CDEF:cdefe=b,8,* AREA:cdefa#00CF00FF:"Inbound" GPRINT:cdefa:LAST:" Current\:%8.2lf %s" GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" LINE1:cdefe#002A97FF:"Outbound" GPRINT:cdefe:LAST:"Current\:%8.2lf %s" GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s"
02/08/2013 04:35:39 PM - WEBLOG: Poller[0] CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start=1360361138 --end=1360362938 --title='HP Procurve Switch 2 - Traffic - 47' --rigid --base=1000 --height=120 --width=500 --alt-autoscale-max --lower-limit=0 COMMENT:"From 2013/02/08 16\:05\:38 To 2013/02/08 16\:35\:38\c" COMMENT:" \n" --vertical-label='bits per second' --slope-mode --font TITLE:10: --font AXIS:7: --font LEGEND:8: --font UNIT:7: DEF:a="/usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_176.rrd":traffic_in:AVERAGE DEF:b="/usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_176.rrd":traffic_out:AVERAGE CDEF:cdefa=a,8,* CDEF:cdefe=b,8,* AREA:cdefa#00CF00FF:"Inbound" GPRINT:cdefa:LAST:" Current\:%8.2lf %s" GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" LINE1:cdefe#002A97FF:"Outbound" GPRINT:cdefe:LAST:"Current\:%8.2lf %s" GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s"
02/08/2013 04:35:38 PM - WEBLOG: Poller[0] CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start=1360361138 --end=1360362938 --title='HP Procurve Switch 2 - Traffic - 45' --rigid --base=1000 --height=120 --width=500 --alt-autoscale-max --lower-limit=0 COMMENT:"From 2013/02/08 16\:05\:38 To 2013/02/08 16\:35\:38\c" COMMENT:" \n" --vertical-label='bits per second' --slope-mode --font TITLE:10: --font AXIS:7: --font LEGEND:8: --font UNIT:7: DEF:a="/usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_174.rrd":traffic_in:AVERAGE DEF:b="/usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_174.rrd":traffic_out:AVERAGE CDEF:cdefa=a,8,* CDEF:cdefe=b,8,* AREA:cdefa#00CF00FF:"Inbound" GPRINT:cdefa:LAST:" Current\:%8.2lf %s" GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" LINE1:cdefe#002A97FF:"Outbound" GPRINT:cdefe:LAST:"Current\:%8.2lf %s" GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s"
02/08/2013 04:35:38 PM - WEBLOG: Poller[0] CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start=1360361138 --end=1360362938 --title='HP Procurve Switch 2 - Traffic - 46' --rigid --base=1000 --height=120 --width=500 --alt-autoscale-max --lower-limit=0 COMMENT:"From 2013/02/08 16\:05\:38 To 2013/02/08 16\:35\:38\c" COMMENT:" \n" --vertical-label='bits per second' --slope-mode --font TITLE:10: --font AXIS:7: --font LEGEND:8: --font UNIT:7: DEF:a="/usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_175.rrd":traffic_in:AVERAGE DEF:b="/usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_175.rrd":traffic_out:AVERAGE CDEF:cdefa=a,8,* CDEF:cdefe=b,8,* AREA:cdefa#00CF00FF:"Inbound" GPRINT:cdefa:LAST:" Current\:%8.2lf %s" GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" LINE1:cdefe#002A97FF:"Outbound" GPRINT:cdefe:LAST:"Current\:%8.2lf %s" GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s"
02/08/2013 04:35:38 PM - WEBLOG: Poller[0] CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start=1360361138 --end=1360362938 --title='HP Procurve Switch 2 - MAC Count' --base=1000 --height=120 --width=500 --alt-autoscale-max --lower-limit=0 COMMENT:"From 2013/02/08 16\:05\:38 To 2013/02/08 16\:35\:38\c" COMMENT:" \n" --vertical-label='MAC'\''s' --slope-mode --font TITLE:10: --font AXIS:7: --font LEGEND:8: --font UNIT:7: DEF:a="/usr/share/cacti/site/rra/hp_procurve_switch_2_snmp_oid_170.rrd":snmp_oid:AVERAGE AREA:a#FF0000FF:"Number of MAC's" GPRINTLAST:"Cur\:%8.0lf" GPRINTAVERAGE:"Avg\:%8.0lf" GPRINTMAX:"Max\:%8.0lf" GPRINTMIN:"Min\:%8.0lf"
02/08/2013 04:35:38 PM - WEBLOG: Poller[0] CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start=1360361138 --end=1360362938 --title='HP Procurve Switch 2 - MEM Usage' --base=1024 --height=120 --width=500 --alt-autoscale-max --lower-limit=0 COMMENT:"From 2013/02/08 16\:05\:38 To 2013/02/08 16\:35\:38\c" COMMENT:" \n" --vertical-label='bytes' --slope-mode --font TITLE:10: --font AXIS:7: --font LEGEND:8: --font UNIT:7: DEF:a="/usr/share/cacti/site/rra/hp_procurve_switch_2_mem_used_173.rrd":mem_used:AVERAGE DEF:b="/usr/share/cacti/site/rra/hp_procurve_switch_2_mem_free_171.rrd":mem_free:AVERAGE DEF:c="/usr/share/cacti/site/rra/hp_procurve_switch_2_mem_total_172.rrd":mem_total:AVERAGE AREA:a#4444FFFF:"MEM used" GPRINTLAST:" Cur\:%8.2lf %s" GPRINTAVERAGE:"Avg\:%8.2lf %s" GPRINTMAX:"Max\:%8.2lf %s" GPRINTMIN:"Min\:%8.2lf %s\n" AREA:b#7CB3F1FF:"MEM free":STACK GPRINTLAST:" Cur\:%8.2lf %s" GPRINTAVERAGE:"Avg\:%8.2lf %s" GPRINTMAX:"Max\:%8.2lf %s" GPRINTMIN:"Min\:%8.2lf %s\n" LINE1:c#000000FF:"MEM total" GPRINT:c:LAST:"Cur\:%8.2lf %s" GPRINT:c:AVERAGE:"Avg\:%8.2lf %s" GPRINT:c:MAX:"Max\:%8.2lf %s" GPRINT:c:MIN:"Min\:%8.2lf %s"
02/08/2013 04:35:38 PM - WEBLOG: Poller[0] CACTI2RRD: /usr/bin/rrdtool graph - --imgformat=PNG --start=1360361138 --end=1360362938 --title='HP Procurve Switch 2 - CPU Usage' --rigid --base=1000 --height=120 --width=500 --alt-autoscale-max --lower-limit=0 COMMENT:"From 2013/02/08 16\:05\:38 To 2013/02/08 16\:35\:38\c" COMMENT:" \n" --vertical-label='percent' --slope-mode --font TITLE:10: --font AXIS:7: --font LEGEND:8: --font UNIT:7: DEF:a="/usr/share/cacti/site/rra/hp_procurve_switch_2_snmp_oid_169.rrd":snmp_oid:AVERAGE AREA:a#FF0000FF:"CPU Usage" GPRINTLAST:"Current\:%8.0lf" GPRINTAVERAGE:"Average\:%8.0lf" GPRINTMAX:"Maximum\:%8.0lf"
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_snmp_oid_169.rrd --template snmp_oid 1360362906:1
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_snmp_oid_170.rrd --template snmp_oid 1360362906:83
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_mem_free_171.rrd --template mem_free 1360362906:109416864
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_mem_total_172.rrd --template mem_total 1360362906:151975344
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_mem_used_173.rrd --template mem_used 1360362906:42558480
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_174.rrd --template traffic_in:traffic_out 1360362906:0:0
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_175.rrd --template traffic_in:traffic_out 1360362906:0:0
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_176.rrd --template traffic_in:traffic_out 1360362906:0:0
02/08/2013 04:35:06 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/site/rra/hp_procurve_switch_2_traffic_in_177.rrd --template traffic_in:traffic_out 1360362906:603746698073:2747665073469
02/08/2013 04:35:06 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
02/08/2013 04:35:06 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
02/08/2013 04:35:06 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
02/08/2013 04:35:06 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
02/08/2013 04:35:06 PM - SPINE: Poller[0] Host[10] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function


I've also attached a picture of graphs for same device.

Any help is appreciated.

Thanks,
Chris

[gandalf]

The graph is not related to the rrdupdate statements seens in the log. Please see 2nd link of my sig for an in-depth debugging procedure
R.

[chrisc543]

Thank you. I went through the debugging and none of the stood out except the RRD file numbers. I see mine is much higher than 1.000000e2 which I believe is 100. So I changed to U and still not getting graphs. Here is an output of the rratoolinfo

This is just on the local host rra.
filename = "localhost_load_1min_5.rrd"
rrd_version = "0003"
step = 300
last_update = 1360014256
ds[load_1min].type = "GAUGE"
ds[load_1min].minimal_heartbeat = 600
ds[load_1min].min = 0.0000000000e+00
ds[load_1min].max = 5.0000000000e+02
ds[load_1min].last_ds = "0.03"
ds[load_1min].value = NaN
ds[load_1min].unknown_sec = 256
ds[load_5min].type = "GAUGE"
ds[load_5min].minimal_heartbeat = 600
ds[load_5min].min = 0.0000000000e+00
ds[load_5min].max = 5.0000000000e+02
ds[load_5min].last_ds = "0.09"
ds[load_5min].value = NaN
ds[load_5min].unknown_sec = 256
ds[load_15min].type = "GAUGE"
ds[load_15min].minimal_heartbeat = 600
ds[load_15min].min = 0.0000000000e+00
ds[load_15min].max = 5.0000000000e+02
ds[load_15min].last_ds = "0.07"
ds[load_15min].value = NaN
ds[load_15min].unknown_sec = 256
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[0].cur_row = 91
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[0].cdp_prep[2].value = NaN
rra[0].cdp_prep[2].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 700
rra[1].cur_row = 451
rra[1].pdp_per_row = 6
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 2
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 2
rra[1].cdp_prep[2].value = NaN
rra[1].cdp_prep[2].unknown_datapoints = 2
rra[2].cf = "AVERAGE"
rra[2].rows = 775
rra[2].cur_row = 601
rra[2].pdp_per_row = 24
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 20
rra[2].cdp_prep[1].value = NaN
rra[2].cdp_prep[1].unknown_datapoints = 20
rra[2].cdp_prep[2].value = NaN
rra[2].cdp_prep[2].unknown_datapoints = 20
rra[3].cf = "AVERAGE"
rra[3].rows = 797
rra[3].cur_row = 143
rra[3].pdp_per_row = 288
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = NaN
rra[3].cdp_prep[0].unknown_datapoints = 260
rra[3].cdp_prep[1].value = NaN
rra[3].cdp_prep[1].unknown_datapoints = 260
rra[3].cdp_prep[2].value = NaN
rra[3].cdp_prep[2].unknown_datapoints = 260
rra[4].cf = "MAX"
rra[4].rows = 600
rra[4].cur_row = 17
rra[4].pdp_per_row = 1
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = NaN
rra[4].cdp_prep[0].unknown_datapoints = 0
rra[4].cdp_prep[1].value = NaN
rra[4].cdp_prep[1].unknown_datapoints = 0
rra[4].cdp_prep[2].value = NaN
rra[4].cdp_prep[2].unknown_datapoints = 0
rra[5].cf = "MAX"
rra[5].rows = 700
rra[5].cur_row = 17
rra[5].pdp_per_row = 6
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = NaN
rra[5].cdp_prep[0].unknown_datapoints = 2
rra[5].cdp_prep[1].value = NaN
rra[5].cdp_prep[1].unknown_datapoints = 2
rra[5].cdp_prep[2].value = NaN
rra[5].cdp_prep[2].unknown_datapoints = 2
rra[6].cf = "MAX"
rra[6].rows = 775
rra[6].cur_row = 27
rra[6].pdp_per_row = 24
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = NaN
rra[6].cdp_prep[0].unknown_datapoints = 20
rra[6].cdp_prep[1].value = NaN
rra[6].cdp_prep[1].unknown_datapoints = 20
rra[6].cdp_prep[2].value = NaN
rra[6].cdp_prep[2].unknown_datapoints = 20
rra[7].cf = "MAX"
rra[7].rows = 797
rra[7].cur_row = 435
rra[7].pdp_per_row = 288
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = NaN
rra[7].cdp_prep[0].unknown_datapoints = 260
rra[7].cdp_prep[1].value = NaN
rra[7].cdp_prep[1].unknown_datapoints = 260
rra[7].cdp_prep[2].value = NaN
rra[7].cdp_prep[2].unknown_datapoints = 260
root@monitor01:/usr/share/cacti/site/rra#

I don't have a user called cactiuser would that make a difference? I log into our server with root and the actual user is administrator so should root have the full permission or administrator?

Thanks,
Chris

[chrisc543]

Also attached is the debug of my SPINE log from your debugging. I had to change directories to /usr/sbin and then could run this ./spine --verbosity=5 5 8 is what I used and it came back not what I expected. This should have been polling an ASA with the 5 min CPU data template.

[chrisc543]

One other thing I forgot to add. It looks like my original e install had the RRA files under /var/lib/cacti/rra I copied this files to /usr/share/cacti/site/rra

[chrisc543]

Ok I have found out it is a permissions issue. I ran top and found that it had tons of spine processes running as www-data so I checked the old rra directory and found the permissions were set as www-data so I changed it to chown www-data on the /usr/share/cacti/site/rra files and it started graphing. I rebooted the server and now it has stopped again. I tried to add www-data:www-data to the chown and same thing. Not sure why it graphed for little while then stopped. Any input is appreciated.

[chrisc543]

ok changing the permissions to www-data:root got some but not all of my devices to graph. I checked the permissions on the rrd files that are not graphing and they are the same. ?????????

[gandalf]

One other thing I forgot to add. It looks like my original e install had the RRA files under /var/lib/cacti/rra I copied this files to /usr/share/cacti/site/rra

Copying those files won't help. At least you will have to symlink <path_cacti>/rra to the new directory.
R.

[chrisc543]

I am not sure what you mean.

I am having more problems with the server. it is locking up on me when I just checked top there was over 200 spine processes its like they are never terminating properly. Any advice on that? I have to reboot the server to get it back up again and showing the graphs. I am getting chunking in my graphs and it is over using the system pretty heavily.

[chrisc543]

I am still having problems. I have had to turn off the poller to get the server to not lock up. It always has zombie processes now since upgrading to 0.8.7h. Any advice on how I should move forward? I am running updates to the ubuntu server now to try and see if it makes a difference.

[chrisc543]

Ok ran all the updates to ubuntu and now on v 12.04LTS and cacti 8.7i all works well. Went to install PIA and over wrote files in directory and it changed where it was writing out the rrd files. by default it looks like ubuntu puts the files in /var/lib/cacti/rra when i copy the files for PIA into the cacti directory it changes that to /var/share/cacti/site/rra then i get the NAN graphs.

Does anyone have any idea how to install PIA on ubuntu or upgrade to 8.8a on ubuntu as it seems like it only works if you leave it alone not sure what files i would have to edit to get it changed from /var/share/cacti/site/rra to /var/lib/cacti/rra when done.

[gandalf]

The Ubuntu HowTo in the related forum section may help.
The "symlink" I mentioned will install a "pointer" from the old directory to the new one as a workaround
R.