Spine: Invalid output from script using regex

[erusenger]

Hello all,

I have a non-prod instance of cacti (recently upgraded to 1.2.26 after running into the issue below) and recently changed to running SPINE (1.2.25) as the poller.


I have a couple of .py scripts that run and each returns multiple values (both are data input as script/command).
Both work fine on the php poller, but one of them gives the following error when using spine:

Code: Select all

2024-05-10 21:15:03 - POLLER: Poller[Main Poller] PID[2995] WARNING: Invalid output! MULTI DS[rxagg] Graphs[rxaggtest] Encountered [U] Expected[Rxagg:value]
2024-05-10 16:15:02 - SPINE: Poller[Main Poller] PID[3002] PT[140037561440320] WARNING: Device[<hostname>] polling sleeping while waiting for 1 Threads to End
2024-05-10 16:15:02 - SPINE: Poller[Main Poller] PID[3002] PT[140037561440320] WARNING: Device[<hostname>] polling sleeping while waiting for 1 Threads to End
2024-05-10 16:15:02 - SPINE: Poller[Main Poller] PID[3002] PT[140037394978560] WARNING: Invalid Response, Device[<hostname>] HT[1] DS[rxagg] Graphs[rxaggtest] SCRIPT: /usr/bin/python3 /usr/share/cacti/scripts/total_CX_throughput.py '<host_ip>', output: U
2024-05-10 16:15:02 - SPINE: Poller[Main Poller] PID[3002] PT[140037394978560] Device[<hostname>] DS[rxagg] Graphs[rxaggtest] ERROR: Empty result [<host_ip>]: '/usr/bin/python3 /usr/share/cacti/scripts/total_CX_throughput.py '<host_ip>''  

This only happens for one of the two scripts. The other is apparently fine.
As soon as I switch the poller to php, both scripts start acting normally.

Both scripts output correctly when run from command line:

Code: Select all

txagg:21.9 rxagg:11.12 count:5

Code: Select all

scgopr:-1570 scgopt:5

Both scripts have the same ownership.
Only issue is that spine doesn't like one of them.


The command for both of the data input methods for both scripts is essentially the same:

Code: Select all

<python_binary_path>/python3 <cacti_path>/cacti/scripts/<scriptname>.py <host>

Both scripts do very similar functions, and both of them execute in less than 3 seconds depending on host.
The only difference is that one of them uses a regex function (the non-working one) and the other does not.
I have tried several other scripts and none of them that use regex in the script seem to work, which I find odd.

I am also not 100% certain that the data-source is not the issue.

I am not sure what is going wrong here, but for the time being this is non-prod so I can mess with it at will.
Any guidance is very much appreciated.

Thank you!

[erusenger]

I should mention I am running centOS 7.9.2009

[TheWitness]

It could be that the user account running that script does not have permission. Not that Spine does not load the user profile. So, you have to include the full path to python if you call it that way.

[erusenger]

Hello, thank you for your reply!

Both data input methods have the Python3 full path in the command.
I also tried the base python binary, the explicit 3 and 3.6 binarys. Both with and without their full paths (re-creating the input method, with a new source, with a new template each time).

I would assume that running the PHP poller would also break if the path were to be broken on one.

I also tried modifying the scripts to have a different output string in case spine didn't like something in the string.
I also tried several different scripts that have the same regex function and they all were broken on spine but OK on the PHP poller.

I'll likely just write a new script that doesn't

Code: Select all

import re

and go from there.

[erusenger]

As an update:

I wrote a new script that imports re and does some other stuff, then outputs two variables in name:value pairs.
I commented out the portion where re is used, but the module is still imported.

Put this in the path/cacti/scripts dir and ran it and it worked fine.

I un-commented the regex statement and kept the variable names the same, verifying that the output is normal when running from command line, leaving all cacti configs the same, and it broke (keeping the same filename etc).
Re-commented again, tested on command line to ensure it was working, and cacti is again working with it.

Seems to be pointing more towards the RE import being the issue.

[macan]

Is it broken only with "import re" or any other import?

I don't know python .. is it posible to import with full path?

[erusenger]

Re is part of the python package so it should be native.
I am also importing sys, subprocess, and argparse. All of which are working properly.

Importing RE breaks when RE is called (so maybe it is RE that is breaking and non the import, but if there is no statement in the script that uses RE then I don't think it ever properly gets called beyond just checking that the library exists).
Having the "import re" in there without RE ever being called doesn't break anything, it is just once RE is called.

I would have also assumed that once python is called that it is no longer running under the spine process but rather spine is calling python > Python executes script w/ command > script returns string and passes string back to spine, at least that is what I would assume.
Though, that is a lot of assumptions. I'm a bit out of my depth here.

[erusenger]

So I was able to get the script to return valid values:

Code: Select all

 2024-05-22 08:10:03 - SPINE: Poller[Main Poller] PID[31575] PT[140303005861632] Device[**hostname**] HT[1] DS[CXintaggDS] Graphs[CX **hostname** - agg traffic] TT[199.09] SCRIPT: /usr/bin/python3 /usr/share/cacti/scripts/allint.py 'host.ip.v.4', output: txagg:0.00 rxagg:0.00 count:0 

but the values are all 0s. As soon as I switch to PHP it works again.
If I hard-code a value, it returns it properly, so the scrip is definitely running but it is not doing so properly.

Still baffled. I also tried making a .py script that calls the other script but that also returns nothing.

I am going to drop this and attempt a different way of doing the same thing.
With that said, if anyone knows how to aggregate all graph of a given type that a given host has, in a way that can be templated and automated and is dynamic, please let me know.

This script got SNMP indexes, parsed UP interfaces, parsed which ones were of a given type, grabbed the Tx/Rx octets, and provided a total of all interfaces. In short, it showed how much traffic a host was passing, and how many interfaces were UP/UP.

[erusenger]

Still looking for a solution to this.
Aggregate templates are a no-go as there are 400+ devices for which to create specific graphs.
Really not understanding why spine is adverse to this specific query when it is working elsewhere, and working under the PHP poller.

[TheWitness]

Well, spine does not source your bashrc or your profile. It could be the poller user lacks a permission too. If you run the poller as root, or run spine in read only mode as root, does it work?

Code: Select all

./spine -R -S -V 3 -H host_id

[erusenger]

Output from that shows:


sudo /usr/local/spine/bin/spine -R -S -V 3 -H <host_id>
FATAL: Unable to read configuration file! (Spine init)


I should mention, I am running this on a VM, not on dedicated hardware.

I will also mention that other Python scripts do execute successfully from spine, but not any script that has either regex or uses dictionaries/lists. Given this, I did not suspect a permission issue since other scripts are running fine with the same py binary being called.

I am beginning to suspect a possible memory handling issue given that it is a VM, and the scripts run when called with hard-coded values for the return in stead of having the data hit a dictionary or list, or being parsed through regex.

[erusenger]

Here is my eventual workaround, in case others fall into the same issue.
I never found the root cause, but I ruled out permissions as the root cause.
I suspect, but cannot confirm, that this is a memory issue wherein the values stored in dictionaries or lists are unable to be stored in memory for some reason. May be due to this being a VM, or it could just be that I am overlooking something obvious (most likely)

Workaround:
Save all data to files, not lists or dictionaries.
Iterate over the files, make new as needed, and overwrite.
so:

Code: Select all

data = (oid_query(host, tx_toms))

becomes

Code: Select all

    with open('{0}{1}_tx_toms.txt'.format(directory,file), "w") as data:
        data.write(oid_query(host, tx_toms))
    if not data.closed:
        data.close()

Looping etc works just as well and there's no need for iterating over a slice of a list instead of using the list itself.
Less convenient, but still usable.

The scripts run remarkably quickly using this method and they leave their last-run data.
This also allows for the creation of host-specific profiles to be stored and iterated upon (separate indexes from what cacti does, unique re-index prompts, compare/diff functions, trailing average, etc.).
It isn't the most elegant solution, but it works, is fast, and seems to use modest system resources when monitoring via BPYTOP.
Ended up with about a 180 line script after rewriting it to use files vs 130 line w/ using lists and dictionaries.