last_ds = "U"

[trezemauro]

Hi all,

got this from RRD TOOL info

rrd_version = "0003"
step = 300
last_update = 1222140022
ds[snmp_oid].type = "GAUGE"
ds[snmp_oid].minimal_heartbeat = 600
ds[snmp_oid].min = 0.0000000000e+00
ds[snmp_oid].max = 5.0000000000e+03
ds[snmp_oid].last_ds = "U"
ds[snmp_oid].value = NaN
ds[snmp_oid].unknown_sec = 22

As you can see there is:

ds[snmp_oid].last_ds = "U"
ds[snmp_oid].value = NaN

I have actually compare it with another RRD which gives me instead:

rrd_version = "0003"
step = 300
last_update = 1222140316
ds[snmp_oid].type = "GAUGE"
ds[snmp_oid].minimal_heartbeat = 600
ds[snmp_oid].min = 0.0000000000e+00
ds[snmp_oid].max = 1.9780000000e+03
ds[snmp_oid].last_ds = "1978"
ds[snmp_oid].value = 3.1648000000e+04
ds[snmp_oid].unknown_sec = 0

Any suggestion?



Also this can be of help, from rrdtool dump


Wrong one
[.....]
<!-- PDP Status -->
<last_ds> U </last_ds>
[.....]


Correct one
[.....]
<!-- PDP Status -->
<last_ds> 60 </last_ds>
[.....]


I can't sort it out



Thanks in advance for any hint

[gandalf]

This is usually shown if the last update is mssing of failed. Please tell us the whole story. YOu may find help at the 2nd link of my sig
Reinhard

[arun.mittalb]

Hi

In every forum i am seeing that you mentioned see in my second link what is that. i am very confused.

i will appreciate your response
thanks

[gandalf]

Login to find the links underneath all of my posts
R.

[ryepup]

I have gone through http://docs.cacti.net/manual:087:4_help.2_debugging and still have a problem with getting "U"s in my rrd.

My script/command data input method queries a database and returns multiple values. I've changed the field names, but otherwise this is the exact output:

Code: Select all

field1:145414 field2:35260 field3:106029 field4:141289 field5:4125 field6:12224 field7:126201 field8:138425

My data template creates GAUGE data source items for each return value, and one DERIVE data source item for "field7". I name these like "field7_g" for the GAUGE item and "field7_d" for the DERIVE item. The GAUGEs are configured in cacti with "0" as the max value, to indicate no upper limit.

Cacti creates everything OK, but I seem to randomly get "U"s in my RRD for some of the gauges. Here's rrdtool info output:

Code: Select all

ds[field7_g].type = "GAUGE"
ds[field7_g].minimal_heartbeat = 600
ds[field7_g].min = 0.0000000000e+00
ds[field7_g].max = NaN
ds[field7_g].last_ds = "U"
ds[field7_g].value = NaN
ds[field7_g].unknown_sec = 24
ds[field8_g].type = "GAUGE"
ds[field8_g].minimal_heartbeat = 600
ds[field8_g].min = 0.0000000000e+00
ds[field8_g].max = NaN
ds[field8_g].last_ds = "138425"
ds[field8_g].value = 3.3222000000e+06
ds[field8_g].unknown_sec = 0
ds[field7_d].type = "DERIVE"
ds[field7_d].minimal_heartbeat = 600
ds[field7_d].min = -1.0000000000e+05
ds[field7_d].max = 1.0000000000e+05
ds[field7_d].last_ds = "126201"
ds[field7_d].value = 0.0000000000e+00
ds[field7_d].unknown_sec = 0

The field7 GAUGE is unparsable, but the field7 DERIVE has the right last_ds. Those came from the same output field in my script, so I'm confused why one works and one fails. For comparison, the field8 GAUGE works fine, so I think that means I configured my gauges correctly.

I've run into this 2 of 3 times I've tried script/command with multiple output parameters, and I've got a workaround that seems wrong: I create another data template for the data source items that didn't work the first time, and that usually works. This ends up running my script/command twice per polling cycle, which is wasteful.

What am I doing wrong here?

[BSOD2600]

Turn the cacti.log level to debug and watch the output of the script.
1) script return all the data to cacti?
2) you see the multi-value parsing entries?
3) correct rrdtool update entries?

[ryepup]

Turn the cacti.log level to debug and watch the output of the script.
1) script return all the data to cacti?
2) you see the multi-value parsing entries?
3) correct rrdtool update entries?

Thanks for the quick response. I've been debugging using spine --verbosity=5, as suggested on http://docs.cacti.net/manual:087:4_help.2_debugging, and I do NOT see log entries for multi-value parsing and rrdtool updates.

Here's the output (with some param name changes to protect the innocent and comments inline):

Code: Select all

 > sudo spine --verbosity=5 70 70
[sudo] password for ryepup:
SPINE: Using spine config file [/etc/cacti/spine.conf]
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The path_php_server variable is /usr/share/cacti/site/script_server.php
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The path_cactilog variable is /var/log/cacti/cacti.log
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The log_destination variable is 1 (FILE)
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The path_php variable is /usr/bin/php
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The availability_method variable is 2
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The ping_recovery_count variable is 3
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The ping_failure_count variable is 2
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The ping_method variable is 2
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The ping_retries variable is 1
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 400
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The snmp_retries variable is 3
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 0
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The boost_redirect variable is 0
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The log_pstats variable is 0
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The threads variable is 10
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The polling interval is 300 seconds
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The number of concurrent processes is 1
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The script timeout is 25
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The number of php script servers to run is 1
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: StartHost='70', EndHost='70', TotalPHPScripts='0'
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Required
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size is 10
07/28/2010 03:09:23 PM - SPINE: Poller[0] Version 0.8.7e starting
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
07/28/2010 03:09:23 PM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.
07/28/2010 03:09:23 PM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(s)
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads is 0
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 2
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[0] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
07/28/2010 03:09:23 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[70] No Host Availability Method Selected
07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[70] Host has no information for recache.

That all seems to be boilerplate stuff.

Code: Select all

07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[70] NOTE: There are '3' Polling Items for this Host
07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[70] DEBUG: The NIFTY POPEN returned the following File Descriptor 5
07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[70] DS[713] SCRIPT: /path/to/script1.sh, output: field1:3 field2:5 field3:1521 field4:479 field5:138110
07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[70] DEBUG: The NIFTY POPEN returned the following File Descriptor 5
07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[70] DS[723] SCRIPT: /path/to/script1.sh, output: field1:3 field2:5 field3:1521 field4:479 field5:138110

Here I have two calls to /path/to/script1.sh, filling datasources 713 and 723. 713 has all the items I want, but one GAUGE and one COUNTER always report ds_last="U", so I run it again in data source 723, which reads those two values with no problems.

Code: Select all

07/28/2010 03:09:23 PM - SPINE: Poller[0] Host[70] DEBUG: The NIFTY POPEN returned the following File Descriptor 5
07/28/2010 03:09:24 PM - SPINE: Poller[0] Host[70] DS[765] SCRIPT: /path/to/script2.sh, output: field1:145414 field2:35260 field3:106029 field4:141289 field5:4125 field6:12224 field7:126201 field8:138425
07/28/2010 03:09:24 PM - SPINE: Poller[0] Host[70] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function

This is my script, and as I reported before I have one GAUGE for field7 that reports unparsable, even though a DERIVE reads field7 with no problems.

Code: Select all

07/28/2010 03:09:24 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
07/28/2010 03:09:24 PM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
07/28/2010 03:09:24 PM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
07/28/2010 03:09:24 PM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
07/28/2010 03:09:24 PM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
07/28/2010 03:09:24 PM - SPINE: Poller[0] Time: 0.4391 s, Threads: 10, Hosts: 2

More standard entries.

My best guess is that I'm not allowed to use the same script output field twice in the same RRD. I haven't dug into cacti source code yet to see how the actual rrdtool update statements are built; maybe cacti doesn't support a one-to-many mapping of output fields?

I'll set cacti to DEBUG and run using cmd.php to see if I get more debugging info.

[ryepup]

I put cacti into "DEVEL" mode, and ran cmd.php. The results don't show anything weird, just a bunch of insert statements that match my expectations.

I do not see any rrdtool update entries. I'm currently running 0.8.7e, so maybe an upgrade is needed.

[ryepup]

Ok, I dug through cacti source, and I think I found out what's happening.

The basic issue is Cacti does not support mapping an multi output field to more than one RRD field. Here's how it's working:

In poller.php, process_poller_output, it detects multiple output, and uses queries the database to get a list of output fields and RRD fields. In my case this looks something like

Code: Select all

data_source_name | data_name
-----------------+----------
...
field7_g         | field7
field7_d         | field7
...

process_poller_output then uses array_rekey to loop over resultset and make a dictionary mapping of script output variable to RRD field name. In my case, a simple dictionary won't work. array_rekey first adds an entry for field7->field7_g, then overwrites it when processing the next row, so I get field7->field7_d in my mapping. That mapping is used to create the actual rrdtool update statement, and that's why I never see any values for one of my RRD fields.

Looks like 8.7.f has the same issue, so upgrading my install won't fix it. Going to hack a little PHP and change these loops around a bit.

[gandalf]

My best guess is that I'm not allowed to use the same script output field twice in the same RRD. I haven't dug into cacti source code yet to see how the actual rrdtool update statements are built; maybe cacti doesn't support a one-to-many mapping of output fields?

If you explain what you want to achive doing so and how you did it, I suppose my answer would be: yes, that's not allowed.
R.

[ryepup]

If you explain what you want to achive doing so and how you did it, I suppose my answer would be: yes, that's not allowed.
R.

I'll try to be clear, and use terminology I see in the cacti interface more consistently.

I want to make two graphs based on the same Data Input Method Output Field : one GAUGE and one DERIVE. I am interested in both the actual value, and in how that value changes.

In my Data Template (on /data_templates.php?action=template_edit), in my list of Data Source Items, I have two items (tabs) with the same "Output Field" value (same option selected in the dropdown).

Currently the /lib/poller.php process_poller_output function does not support mapping a Data Input Method Output Field to more than one Data Source Item. It seems basically to be doing: "for each Data Input Method Output Field, find the single matching Data Source Item and update the rrd value"

I've attached a patch that re-orders this loop, so it does: "for each Data Source Item, find the single matching Data Input Method Output Field, and update the rrd value". This change in ordering permits the same Data Input Method Output Field to be used by more than one Data Source Item.

I am not experienced with PHP nor with Cacti coding standards, so there are likely problems with it, but this patch works for me. Feel free to use it however you wish.

Is there a more appropriate way for me to make two graphs off the same Data Input Method Output Field?

[gandalf]

This approach is indeed new to me. But your use case is valid, but a cornercase. The current workaround would be to make the DIM spit out the same value twice, which, in my eyes is a valid workaround.
Please post your case and patch to http://ww.cacti.net/bugs.php to increase the chance to have it integrated. You know, this is for tracking purpose
R.

[ryepup]

Bug reported: http://bugs.cacti.net/view.php?id=1820

I summarized the workarounds on there, too, so anyone else having this problem should check it out: http://bugs.cacti.net/view.php?id=1820