Spine not processing all script output

[Cor]

I just upgraded from cmd.php to Spine for performance reasons, and now all hosts polled with a custom script display partial or no data in their graphs and suffer recurring "Poller Output Table not Empty" errors. The affected hosts were fine under cmd.php. Rebuilding the poller cache does not correct the problem. Running poller_output_empty.php does not help either. Deleting and recreating hosts/graphs does not help, nor does restarting the system.

Running spine by itself in debug mode against one of the problem hosts, I see that only the first name:value pair in the script output is being captured. This matches what is being displayed in the graphs. This looks like a bug to me, but I'd love to hear suggestions if anyone has them. I'm running Cacti 0.8.7e, Spine 0.8.7e, CentOS 5.4.

Here is an example:

---- script run from the command line ----------
[cacti@localhost ~]$ /bin/sh /usr/local/src/cacti-0.8.7e/scripts/DataNodeStatistics.sh service:jmx:rmi:///jndi/rmi://111.222.333.444:10003/jmxrmi monitorRole password hadoop:service=DataNode,name=DataNodeActivity-DS-758376126-111.222.333.444-50010-1254818525508
BytesWritten:2814528012 ReadsFromRemoteClient:2453 ReplaceBlockOpNum:0 ReplaceBlockOpAverageTime:0 ReplaceBlockOpMinTime:-1 ReplaceBlockOpMaxTime:0 BlockVerificationFailures:0 BlocksRead:2457 HeartbeatsNum:100 HeartbeatsAverageTime:12 HeartbeatsMinTime:0 HeartbeatsMaxTime:57595 CopyBlockOpNum:0 CopyBlockOpAverageTime:0 CopyBlockOpMinTime:0 CopyBlockOpMaxTime:155358 BlocksWritten:61 ReadBlockOpNum:2457 ReadBlockOpAverageTime:31971 ReadBlockOpMinTime:0 ReadBlockOpMaxTime:207781535 ReadsFromLocalClient:4 WritesFromRemoteClient:40 BlocksVerified:2 BlockReportsNum:0 BlockReportsAverageTime:0 BlockReportsMinTime:498 BlockReportsMaxTime:110621 WritesFromLocalClient:22 BlocksReplicated:1 BytesRead:203509970 BlocksRemoved:26 WriteBlockOpNum:62 WriteBlockOpAverageTime:3122 WriteBlockOpMinTime:0 WriteBlockOpMaxTime:823785234 ReadMetadataOpNum:0 ReadMetadataOpAverageTime:0 ReadMetadataOpMinTime:-1 ReadMetadataOpMaxTime:0 blockChecksumOpNumOps:0 blockChecksumOpAvgTime:0 blockChecksumOpMinTime:-1 blockChecksumOpMaxTime:0
---- End script run from the command line ------

---- debug output from spine run against same script/host ---------
Host[226] DS[3131] SCRIPT: /bin/sh /usr/local/src/cacti-0.8.7e/scripts/DataNodeStatistics.sh service:jmx:rmi:///jndi/rmi://111.222.333.444:10003/jmxrmi monitorRole password hadoop:service=DataNode,name=DataNodeActivity-DS-758376126-111.222.333.444-50010-1254818525508, output: BytesWritten:2814528012
---- End debug output from spine run against same script/host ---------

[gandalf]

A guess, but a good one is that those scripts don't spit out the data on a single print statement but use multiple prints. Change this and you will be happy
R.

[Cor]

Preliminary indications are that suggestion helped, thanks! Is that documented anywhere? It should be.

In my case the output issue was caused in Java by "System.out.println" in a loop. Changing the wrapper shell script to write the output of the Java command to a shell variable and then echo that to STDOUT in one gulp seems to have cured the immediate problem. Most of the "Poller Output Table not Empty" errors are gone and I expect I'll be able to ferret out the rest.

Again, thanks! Life is good when graphs are filling with pretty colors.

[gandalf]

Is that documented anywhere? It should be.

Oops. Added.
R.

[TheWitness]

Yea, it's in the main doco but, as Gandalf explains, wasn't until recently added to the Troubleshooting guide.

You must output the entire contents at once as I do non-blocking reads and don't wait on EOL. So, you have to write the entire buffer in one shot.

You might also want to use the latest spine w/patches of course.

TheWitness

[doughairfield]

Sorry, I'm having the same problem with my hadoop graphs. Pardon my n00biness but I can't find the documentation describing how to change what your referring to. Can someone steer me in the right direction?

[gandalf]

Please see 2nd link of my sig for help
R.

[doughairfield]

Sorry, does sig=signature? And if so, where is that?

OK, the was strange, once I posted this, your signature showed up, i'm checking it out now.