Graphs not populating.

[dilbert3k]

This is a brand new install on a CentOS 6.3 using the latest stable Cacti build.

When I first installed it I could not even get the RRD files to be created but during troubleshooting I was able to force their creation but I have never gotten any data in them. I am seeing no problems with polling, and the logs show that everything seems to be working... but all graphs only have nan.

Following Gandalfs troubleshooting guide.

Check Cacti Log File

01/11/2013 06:40:01 PM - POLLER: Poller[0] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300', Max Runtime '298', Poller Runs: '1'
01/11/2013 06:40:01 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q "/usr/share/cacti/cmd.php" 0 4]
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] PING: UDP Ping Success (0.15 ms)
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[3] CMD: perl /usr/share/cacti/scripts/linux_memory.pl MemFree:, output: 1304428
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_mem_buffers_3.rrd --template mem_buffers 1357947601:1304428
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[4] CMD: perl /usr/share/cacti/scripts/linux_memory.pl SwapFree:, output: 1015800
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_mem_swap_4.rrd --template mem_swap 1357947601:1015800
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[5] CMD: perl /usr/share/cacti/scripts/loadavg_multi.pl, output: 1min:0.00 5min:0.00 10min:0.00
01/11/2013 06:40:01 PM - POLLER: Poller[0] Parsed MULTI output field '1min:0.00' [map 1min->load_1min]
01/11/2013 06:40:01 PM - POLLER: Poller[0] Parsed MULTI output field '5min:0.00' [map 5min->load_5min]
01/11/2013 06:40:01 PM - POLLER: Poller[0] Parsed MULTI output field '10min:0.00' [map 10min->load_15min]
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_load_1min_5.rrd --template load_1min:load_5min:load_15min 1357947601:0.00:0.00:0.00
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[6] CMD: perl /usr/share/cacti/scripts/unix_users.pl , output: 1
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_users_6.rrd --template users 1357947601:1
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[7] CMD: perl /usr/share/cacti/scripts/unix_processes.pl, output: 128
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_proc_7.rrd --template proc 1357947601:128

No errors.

Check Basic Data Gathering

Unable to run these as the Cacti user. It always says the user is unavailable.

Check Cacti's Poller

This seems to be aimed at someone who has 1 or 2 devices failing, none the less results look fine.

01/11/2013 06:50:56 PM - CMDPHP: Poller[0] Host[2] SNMP: Host responded to SNMP
01/11/2013 06:50:56 PM - CMDPHP: Poller[0] Host[2] RECACHE: Processing 1 items in the auto reindex cache for 'xxx.xxx.xxx.xxx'.
01/11/2013 06:50:56 PM - CMDPHP: Poller[0] Host[2] RECACHE DQ[1] OID: .1.3.6.1.2.1.1.3.0
01/11/2013 06:50:56 PM - CMDPHP: Poller[0] Host[2] RECACHE DQ[1] OID: .1.3.6.1.2.1.1.3.0, output: 284128000
01/11/2013 06:50:56 PM - CMDPHP: Poller[0] Host[2] DS[8] SNMP: v2: xxx.xxx.xxx.xxx, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, output: 4276299224

Check Bulkwalk Behaviour (SNMP Data Queries only)

My issue is with more than just SNMP, but I went ahead and reduced it to 1. No help.

Check MySQL Update

I dug through all the tables in MySQL and could not see anything that looked wrong.

Check RRD File Update

No problems seen.

01/11/2013 06:50:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/XXXXX_traffic_in_47.rrd --template traffic_out:traffic_in 1357948201:0:0
01/11/2013 06:50:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/XXXXX_traffic_in_48.rrd --template traffic_in:traffic_out 1357948201:0:0
01/11/2013 06:50:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/XXXXX_traffic_in_49.rrd --template traffic_out:traffic_in 1357948201:0:0

Check RRD File Ownership

-rw-r--r--. 1 cacti cacti 94816 Jan 11 18:55 XXXXX_traffic_in_10.rrd

Note it has been running for days now and the file size has not changed (Not sure if it is supposed to)

Check RRD File Numbers

Sure enough we have -nan here.

1357948800: -nan -nan
1357949100: -nan -nan
1357949400: -nan -nan
1357949700: -nan -nan
1357950000: -nan -nan
1357950300: -nan -nan
1357950600: -nan -nan
1357950900: -nan -nan
1357951200: -nan -nan
1357951500: -nan -nan
1357951800: -nan -nan
1357952100: -nan -nan
1357952400: -nan -nan

But it looks ok on the next step.

[root@xxxxxxx01 cacti]# rrdtool info /var/lib/cacti/rra/xxxxx_traffic_in_10.rrd | egrep ds
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "0"
ds[traffic_in].value = 0.0000000000e+00
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "0"
ds[traffic_out].value = 0.0000000000e+00
ds[traffic_out].unknown_sec = 0

Check RRDTool Graph Statement

Checking this one I noticed I have SOME data in the graphs now. Something I did during troubleshooting put two short bursts of data into them... but then stopped again. Maybe I have the cron entry in the wrong place? (I did a lot of work on cron)

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title='Sec01 - Traffic - ethernet0/0' \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit='0' \
--vertical-label='bits per second' \
--slope-mode \
--font TITLE:10: \
--font AXIS:7: \
--font LEGEND:8: \
--font UNIT:7: \
DEF:a="/usr/share/cacti/rra/xxx01_traffic_in_8.rrd":'traffic_in':AVERAGE \
DEF:b="/usr/share/cacti/rra/xxx01_traffic_in_8.rrd":'traffic_out':AVERAGE \
CDEF:cdefa='a,8,*' \
CDEF:cdefe='b,8,*' \
AREA:cdefa#00CF00FF:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97FF:"Outbound" \
GPRINT:cdefe:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s\n"

RRDTool Says:

OK

I will try working with Cron some more, but suspect the data may have come from my manually running things. At least now I know data is being stored and displayed.

Any suggestions would be wonderful as this is driving me nuts.

[gandalf]

This is a brand new install on a CentOS 6.3 using the latest stable Cacti build.

When I first installed it I could not even get the RRD files to be created but during troubleshooting I was able to force their creation but I have never gotten any data in them. I am seeing no problems with polling, and the logs show that everything seems to be working... but all graphs only have nan.

Following Gandalfs troubleshooting guide.

Check Cacti Log File

01/11/2013 06:40:01 PM - POLLER: Poller[0] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300', Max Runtime '298', Poller Runs: '1'
01/11/2013 06:40:01 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q "/usr/share/cacti/cmd.php" 0 4]
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] PING: UDP Ping Success (0.15 ms)
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[3] CMD: perl /usr/share/cacti/scripts/linux_memory.pl MemFree:, output: 1304428
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_mem_buffers_3.rrd --template mem_buffers 1357947601:1304428
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[4] CMD: perl /usr/share/cacti/scripts/linux_memory.pl SwapFree:, output: 1015800
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_mem_swap_4.rrd --template mem_swap 1357947601:1015800
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[5] CMD: perl /usr/share/cacti/scripts/loadavg_multi.pl, output: 1min:0.00 5min:0.00 10min:0.00
01/11/2013 06:40:01 PM - POLLER: Poller[0] Parsed MULTI output field '1min:0.00' [map 1min->load_1min]
01/11/2013 06:40:01 PM - POLLER: Poller[0] Parsed MULTI output field '5min:0.00' [map 5min->load_5min]
01/11/2013 06:40:01 PM - POLLER: Poller[0] Parsed MULTI output field '10min:0.00' [map 10min->load_15min]
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_load_1min_5.rrd --template load_1min:load_5min:load_15min 1357947601:0.00:0.00:0.00
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[6] CMD: perl /usr/share/cacti/scripts/unix_users.pl , output: 1
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_users_6.rrd --template users 1357947601:1
01/11/2013 06:40:01 PM - CMDPHP: Poller[0] Host[1] DS[7] CMD: perl /usr/share/cacti/scripts/unix_processes.pl, output: 128
01/11/2013 06:40:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/localhost_proc_7.rrd --template proc 1357947601:128

No errors.

Looks fine, indeed. Data is fetched and rrdupdate commands are created.

Check Basic Data Gathering

Unable to run these as the Cacti user. It always says the user is unavailable.

That might be an issue. Which user is supposed to work as the Cacti user (cacti, cactiuser, www-data, ...)?

Check RRD File Update

No problems seen.

01/11/2013 06:50:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/XXXXX_traffic_in_47.rrd --template traffic_out:traffic_in 1357948201:0:0
01/11/2013 06:50:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/XXXXX_traffic_in_48.rrd --template traffic_in:traffic_out 1357948201:0:0
01/11/2013 06:50:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /usr/share/cacti/rra/XXXXX_traffic_in_49.rrd --template traffic_out:traffic_in 1357948201:0:0

You did run this as the "Cacti user"?

Check RRD File Ownership

-rw-r--r--. 1 cacti cacti 94816 Jan 11 18:55 XXXXX_traffic_in_10.rrd

Note it has been running for days now and the file size has not changed (Not sure if it is supposed to)

Is "cacti" the user that's supposed to be the "Cacti user"? Is "cacti" the user that runs the poller from crontab?

R.

[dilbert3k]

Yes, I used cacti for the cacti user at every point I was asked for a user name.

Some install guides say cactiuser, others just cacti.

[dilbert3k]

And now it is working... no idea why. I have changed far too many things to be able to keep track of them all

I still have one minor issue, the time on the graphs is off by 1 hour. When I first installed cacti the system was on Eastern time I have changed it to be Central, however it looks like Cacti did not get that notice

It always shows the last hour as having no data, how can I change it's time?

[gandalf]

Assuming you are talking about the 4 default graphs created by a fresh install: It is known, that they don't make it into the poller cache. You once have to rebuild that, either under "System Utilities" or via cli script. This bug is fixed for next upcoming version
R.

[dilbert3k]

Ok, fully working now. Looks like I needed to update the timezone in mysql, that or the restart of mysqld did something

At any rate I now have data flowing into my graphs and all with the correct time stamps. Thanks a ton for the troubleshooting guide!