2 out of 3 Nginx data sources are nan but why?

[iberkner]

For some time we've been using the templates provided by http://code.google.com/p/mysql-cacti-templates/ to monitor Nginx, MySQL and Memcached.

After a recent upgrade to latest stable Cacti release (0.8.7h) and the latest templates, 2 of our 3 Nginx graphs have stopped reporting. I've checked everything from permissions, to RRD configuration and update calls, everything looks good. The data is retrieved through SSH and running it in shell produces the correct results as well as looking at the log data provided by Cacti.

When I review the actual data stored in the relevant RRA file, for 2 of the graph templates the values show "nan", even though both the Cacti log file and the command line call show correct value returned from the request. I've also looked at the "max" settings for these 2 data sources and they are set at: 18446744073709551615. The values that I'm returning for these data sources, are well below this maximum limit, for example: 107886056.

"Assuming" that the data returned is valid (meaning the SSH query works as it should), how do I further debug the problem between Cacti and RRD? If I'm exceeding a maximum, how can I tell? If for some reason the data returned is invalid to RRD, how can I tell? Is this a Cacti issue? a RRD issue? I followed this protocol for debugging: http://www.cacti.net/downloads/docs/html/debugging.html and got to the point that I can tell that the data RRD is processing from the RRA is in fact "nan".

Its important to note that other data sources that use this type of query are working fine.

I'm happy to provide additional data points if needed.

Thanks

[iberkner]

I wanted to give more specific examples from the log:

-- Line which shows values returned in output (b0, b1):

10/30/2011 12:30:11 PM - CMDPHP: Poller[0] Host[30] DS[389] CMD: /usr/bin/php -q /var/www/html/cacti/scripts/ss_get_by_ssh.php --host #.#.#.# --type nginx --items b0,b1 --server --url --http-user --http-password , output: b0:32737954 b1:32737954

-- Following line which shows the "rrdtool update" command. In this case, I don't see the variable name or values, only the timestamp followed by a "U". What does that mean and where's the actual problem located?

10/30/2011 12:30:11 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/html/cacti/rra/nginx_web_server_nginx_server_accpts_389.rrd --template 1319995808:U

Thanks

[TheWitness]

Grab the latest branches/0.8.7/lib/functions.php from SVN, or if using the PIA, you would get PIA/trunk/lib/functions.php. There was a is_hexadecimal() function that was detecting the output as a hex string, when it is in fact not.

TheWitness

[iberkner]

Thanks, the updated functions.php appears to have fixed this issue (see updated log data below), however, graphing has now completed stopped, across all devices. Why would that be the case?

10/30/2011 01:10:10 PM - CMDPHP: Poller[0] Host[30] DS[389] CMD: /usr/bin/php -q /var/www/html/cacti/scripts/ss_get_by_ssh.php --host #.#.#.# --type nginx --items b0,b1 --server --url --http-user --http-password , output: b0:32845645 b1:32845645

10/30/2011 01:10:10 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/html/cacti/rra/nginx_web_server_nginx_server_accpts_389.rrd --template NGINX_server_accpts:NGINX_server_handld 1319998209:32845645:32845645

[iberkner]

This is one of the RRD's that was correctly reporting prior to upgrading to the latest functions.php from the repository. None of the previously working RRDs are now showing correct values, nothing was changed, other than the update of the functions.php library.

[server]# fetch s1_nginx_waiting_385.rrd AVERAGE

1319991000: 4.4863000000e+02 7.8500000000e+00 4.3340000000e+01 4.9982000000e+02
1319991300: 4.2860000000e+02 2.4490000000e+01 1.7810000000e+01 4.7090000000e+02
1319991600: nan nan nan nan
1319991900: nan nan nan nan
1319992200: nan nan nan nan

Any suggestions?

Thanks

[TheWitness]

Just update the single function is_hexadecimal() instead of the whole file.

[iberkner]

Thanks, simply replacing the is_hexadecimal function (as opposed to the whole file) fixed the problem.