Script/Command is not consistent

[ehelm]

I have a Data Input Method using Script/Command that will graph data occasionally, but mostly I get the "SCRIPT not valid. Partial Result" log messages. I've doublechecked my output and it all looks good. I have several other Script/Command Data Input Methods doing almost the exact same thing and they all work great. Any ideas?
If you need more info, please let me know.

/Eric

[BSOD2600]

Turn the logging level in cacti to debug and watch what the script is returning when it says it's invalid.

[ehelm]

Here is from a debug:

04/07/2006 11:30:03 AM - CACTID: Poller[0] Host[0] DS[6529] WARNING: Result from SCRIPT not valid. Partial Result: multi.uribl.com_byte...
04/07/2006 11:30:03 AM - CACTID: Poller[0] Host[0] DS[6529] SCRIPT: /usr/bin/ssh -i /root/.ssh/id_rsa dccd.blah.net cat /usr/local/doublecheck/cacti/cacti_rbldnsbytes.txt, output: U
04/07/2006 11:30:03 AM - CACTID: Poller[0] Host[0] DEBUG: The POPEN returned the following File Descriptor 14

If it helps, I am running cactid 0.8.6g (Although a cactid -v shows 0.8.6f, even though I've recompiled 6g just to be sure.) OS is Fedora Core 4.
The script output looks like this when run manually:
multi.uribl.com_bytes:107232 multi.uribl.com_bytesin:36470 multi.uribl.com_bytesout:70762 black.uribl.com_bytes:12275 black.uribl.com_bytesin:4145 black.uribl.com_bytesout:8130 red.uribl.com_bytes:3059 red.uribl.com_bytesin:1036 red.uribl.com_bytesout:2023 grey.uribl.com_bytes:3101 grey.uribl.com_bytesin:1057 grey.uribl.com_bytesout:2044 white.uribl.com_bytes:809 white.uribl.com_bytesin:290 white.uribl.com_bytesout:519 bl.dcrbl.com_bytes:22152 bl.dcrbl.com_bytesin:8226 bl.dcrbl.com_bytesout:13926 multi.surbl.org_bytes:16030 multi.surbl.org_bytesin:5473 multi.surbl.org_bytesout:10557

[gandalf]

1) Is all output on one line? No <CR>?
2) Perhaps "." in the output vars is the problem. May you have a go replacing this by some other chars or omitting it?
Reinhard

[ehelm]

1) Is all output on one line? No <CR>?
2) Perhaps "." in the output vars is the problem. May you have a go replacing this by some other chars or omitting it?
Reinhard

1) There are no <CR>.
2) I thought of that, but I have 2 other nearly identical scripts, 1 for RBL DNS BPS, and another for RBL DNS Requests. Both of these are working flawlessly.

When looking through some debug logs, it appears that the script output is not always complete. Although, sometimes it is. When its not complete its never truncated at the same spot.

[gandalf]

Did you already test the script using http://forums.cacti.net/viewtopic.php?t=7738? This may give further insight
Reinhard

[ehelm]

That post talks about testing the PHP Script server.
The script in question is not running from the script server. In the poller cache it is listed as "Action 1:", not "Action 2:".
The Data Input Method is of type "Script/Command" Not Script Server.
Is there any way to test the script in a similar manner aside from running the command manually?

Attached are screenshots of two graphs. The one labeled DNS Bytes is the script that is having a problem. The one labeled DNS Requests is working fine.
The script for DNS Bytes when run manually, returns the following output:
/usr/bin/ssh -i /root/.ssh/id_rsa dccd.ruraltel.net cat /usr/local/doublecheck/cacti/cacti_rbldnsbytes.txt
multi.uribl.com_bytes:49090 multi.uribl.com_bytesin:15706 multi.uribl.com_bytesout:33384 black.uribl.com_bytes:4017 black.uribl.com_bytesin:1274 black.uribl.com_bytesout:2743 red.uribl.com_bytes:189 red.uribl.com_bytesin:31 red.uribl.com_bytesout:158 grey.uribl.com_bytes:630 grey.uribl.com_bytesin:173 grey.uribl.com_bytesout:457 white.uribl.com_bytes:193 white.uribl.com_bytesin:33 white.uribl.com_bytesout:160 bl.dcrbl.com_bytes:14290 bl.dcrbl.com_bytesin:5302 bl.dcrbl.com_bytesout:8988 multi.surbl.org_bytes:7721 multi.surbl.org_bytesin:2366 multi.surbl.org_bytesout:5355

The script for DNS Requests when run manually is almost exactly the same, just different data:
/usr/bin/ssh -i /root/.ssh/id_rsa dccd.ruraltel.net cat /usr/local/doublecheck/cacti/cacti_rbldnsrequests.txt
multi.uribl.com_reqs:291 multi.uribl.com_list:126 multi.uribl.com_nolist:165 black.uribl.com_reqs:25 black.uribl.com_list:12 black.uribl.com_nolist:13 red.uribl.com_reqs:2 red.uribl.com_list:1 red.uribl.com_nolist:1 grey.uribl.com_reqs:1 grey.uribl.com_list:1 grey.uribl.com_nolist:0 white.uribl.com_reqs:0 white.uribl.com_list:0 white.uribl.com_nolist:0 bl.dcrbl.com_reqs:116 bl.dcrbl.com_list:0 bl.dcrbl.com_nolist:116 multi.surbl.org_reqs:34 multi.surbl.org_list:11 multi.surbl.org_nolist:23

The DNS Requests graph is working perfectly, but the DNS Bytes graph is very broken.

Let me know if you need anymore info about the setup.

/Eric

[gandalf]

That post talks about testing the PHP Script server.
The script in question is not running from the script server. In the poller cache it is listed as "Action 1:", not "Action 2:".
The Data Input Method is of type "Script/Command" Not Script Server.

Sorry for being imprecise

Attached are screenshots of two graphs. The one labeled DNS Bytes is the script that is having a problem. The one labeled DNS Requests is working fine.

Fine, that helps a lot. The failing one gives intermittend response, eh? This may be due to a runtime problem. Please time your scripts from cli. It may be necessary to adjust the script maximum runtime, if this exceeds the Script and Script Server Timeout Value at Settings->Poller
Reinhard

[ehelm]

Currently, I have the Script and Script Server Timeout Value set at 60 seconds. Do you think this value is too low? I had previously adjusted it thinking it may help.
The scripts, when run manually, only take milliseconds.
time /usr/bin/ssh -i /root/.ssh/id_rsa dccd.ruraltel.net cat /usr/local/doublecheck/cacti/cacti_rbldnsbytes.txt
multi.uribl.com_bytes:68816 multi.uribl.com_bytesin:22726 multi.uribl.com_bytesout:46090 black.uribl.com_bytes:3693 black.uribl.com_bytesin:1242 black.uribl.com_bytesout:2451 red.uribl.com_bytes:148 red.uribl.com_bytesin:56 red.uribl.com_bytesout:92 grey.uribl.com_bytes:123 grey.uribl.com_bytesin:38 grey.uribl.com_bytesout:85 white.uribl.com_bytes:0 white.uribl.com_bytesin:0 white.uribl.com_bytesout:0 bl.dcrbl.com_bytes:13594 bl.dcrbl.com_bytesin:5049 bl.dcrbl.com_bytesout:8545 multi.surbl.org_bytes:12997 multi.surbl.org_bytesin:4013 multi.surbl.org_bytesout:8984
real 0m0.169s
user 0m0.048s
sys 0m0.008s

[gandalf]

time /usr/bin/ssh -i /root/.ssh/id_rsa dccd.ruraltel.net cat /usr/local/doublecheck/cacti/cacti_rbldnsbytes.txt

I only just noticed, that this is really TWO commands. And "cat" should be give will full path (I suppose this is required when running the latest cactid as a poller). But you told us, that this is currently working for the second example. This is quite unusal, but this does not mean that it won't work.
Am I right assuming that the first command fills some txt file and the second displays the outcome? Then there might be some timing problem between both commands?
Reinhard

[ehelm]

There should only be 1 command that the poller is running, ssh, since cat is being run on the remote box. Would I need to provide the path to cat on the remote box?
The 'time' command in my previous post was just to record the resources used, and to show how long the actual command takes to run, which in this case is less than 1 second. So my point was that the script timeout value in the poller settings may not actually affect why this is happening.

The file does have the output data in it. There is a script that runs on the remote boxes that records its output into a file. I setup the Data Input Method to just cat this file for speed purposes. This is our way of using cacti to gather and graph data on remote boxes.
I probably should have mentioned all of this earlier in the thread.
Let me know if you have anymore questions.

[gandalf]

There should only be 1 command that the poller is running, ssh, since cat is being run on the remote box. Would I need to provide the path to cat on the remote box?

No, as this command is not directly run by the poller.
Reinhard

[ehelm]

So, we know the script runs. And, when I run it manually, it is fine. The other 2 nearly identical scripts are working perfectly.
The question is: What happens when the poller returns "Result from SCRIPT not valid. Partial Result:"? And, why does it work sometimes?

/Eric

[gandalf]

So, we know the script runs. And, when I run it manually, it is fine. The other 2 nearly identical scripts are working perfectly.
The question is: What happens when the poller returns "Result from SCRIPT not valid. Partial Result:"?

This occurs, if the result is not valid or no result at all

And, why does it work sometimes?

That's the question! I'm at a loss
Reinhard

[rony]

Could it be that SSH is timing out? Or that the file is locked or incorrectly formated, because it's being updated when the poller hits it?

I would be very curious to see full debug on the polling cycle when you get the partial results error.