Issue w/external script using spine (vs cmd.php)

[doogles]

Hello! It's been a number of years, but I'm glad to be back in cacti-ville.

This past weekend, I spun up a cacti 1.1.37 to serve a specific monitoring need. The application I'm interfacing with has their own application-specific API to fetch the data I want. So to solve this I created a Python script that does what I want and taught Cacti about it. I require reasonably high resolution of this data, so I created a Data Profile which calls for polling every 1 minute and keeping the result for 1 year. This alone is working great.

Where I seem to be running in to a pickle, now, is with performance. With cmd.php, the external script appears to be called in a serial fashion -- ie, it waits for completion of the script before calling it again with the new parameters. So now that I have ~75 data sources with a poll every minute, the poller does not complete in time and the RRDs towards the end of the poller process are not updated.

CMD.PHP QUESTION: I tried adjusting "Maximum Concurrent Poller Processes", but that doesn't seem to do anything. Should it?

So given the above challenge, I figured I would simply switch to spine and run a bunch of threads concurrently. However, it appears that my script and spine don't quite get along. It appears that the data sent to my script gets cut off with spine, while it doesn't with cmd.php.

Example from cacti.log (DEBUG) using cmd.php:
2018/04/16 12:28:32 - POLLER: Poller[1] Device[1] DS[43] CMD: /usr/share/cacti/scripts/cucm_perfmon_sum.py 'soicucmpub.example.com' USER_HARDCODED PASS_PHARDCODED '\\soicucmsub1.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_1)\CallsActive' '\\soicucmsub2.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_1)\CallsActive' '\\soicucmsub3.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_1)\CallsActive' '\\soicucmsub4.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_1)\CallsActive' '\\soicucmsub1.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_2)\CallsActive' '\\soicucmsub2.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_2)\CallsActive' '\\soicucmsub3.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_2)\CallsActive' '\\soicucmsub4.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_2)\CallsActive' '' '' '' '', output: 91

Example of the same data source fetch from cacti.log (DEBUG) using spine:
2018/04/16 11:55:30 - SPINE: Poller[1] Device[1] TH[1] DS[43] SCRIPT: /usr/share/cacti/scripts/cucm_perfmon_sum.py 'soicucmpub.example.com' USER_HARDCODED PASS_PHARDCODED '\\soicucmsub1.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_1)\CallsActive' '\\soicucmsub2.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_1)\CallsActive' '\\soicucmsub3.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_1)\CallsActive' '\\soicucmsub4.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_1)\CallsActive' '\\soicucmsub1.example.com\Cisco SIP(SIP_I3_DFR_IVR_PROD_TF_2)\CallsActive' '\\soicucmsub2.example.com\Cis, output: U

The only change I need to make to the system to cause the above behavior is to switch from cmd.php to spine (and vice-versa to resolve it).

Back of the napkin math (ok, pasting in to Notepad++) suggests that the issue might be 512-byte specific? I notice that the cut off version (spine, above) makes it to column 511.

Can this cacti/spine behavior be tweaked such that spine executes the exact same command as cmd.php? I realize the command is long but I have to pass up to 16 parameters consisting of long'ish "counter names" in order to get all of the data back that I need.

Thanks!,
-jd

[camerabob]

Not quite an answer but possibly a workaround.

I see you pass a similar FQDN 16 times (I didn't count, but saw several occurrences). Perhaps create really short 'CNAME's in the host file (or just new names that point to the same IPs if they're not dynamic) that map to the same locations. This will shrink your query somewhat, getting you past one hurdle.

[doogles]

Not quite an answer but possibly a workaround.

I see you pass a similar FQDN 16 times (I didn't count, but saw several occurrences). Perhaps create really short 'CNAME's in the host file (or just new names that point to the same IPs if they're not dynamic) that map to the same locations. This will shrink your query somewhat, getting you past one hurdle.

Thanks for the suggestion, but unfortunately the syntax for the API requires that each counter is called using the internal name for the counter -- which is \\host\group\counter (with host being a FQDN). I unfortunately can't abbreviate it and still get the data; that's just how this application's "PerfMon API" works.

-jd

[netniV]

I would agree with the above suggestion. I would modify the python script in question to have the FQDN as a parameter of it's own and then build up the other parameters using it. That would considerably shorten the command line. If that script is not your own script, maybe consider copying and customising it.

[doogles]

I would agree with the above suggestion. I would modify the python script in question to have the FQDN as a parameter of it's own and then build up the other parameters using it. That would considerably shorten the command line. If that script is not your own script, maybe consider copying and customising it.

Thank you, but that is not an option. The parameter(s) are the counter(s) I wish to fetch, and because which counter(s) I want to fetch varies depending on what data I'm trying to fetch, it cannot be programatically built.

I appreicate the suggested workarounds, but surely there must be some way to fix spine to pass the full command ling string without cutting off at 512 characters (like cmd.php)?

-jd

[netniV]

OK so what you are going to have to do is build yourself a custom version of spine. You are going to have to increase the size of arg1 in the target_t structure. However, please note that you should definitely check where arg1 is used and make sure that ALL places are correctly handling the new size of arg1. Also, you will need to make sure that any copies of arg1 are also correctly handled, ie, if there is a copy made to a buffer, you need to make sure that the buffer can handle your new size.

Also note, that on Windows the maximum command line size is 8,191 characters. On my Ubuntu box it is 2,097,152 characters (obtained via getconf ARG_MAX). So different platforms have different limits.

[doogles]

OK so what you are going to have to do is build yourself a custom version of spine. You are going to have to increase the size of arg1 in the target_t structure. However, please note that you should definitely check where arg1 is used and make sure that ALL places are correctly handling the new size of arg1. Also, you will need to make sure that any copies of arg1 are also correctly handled, ie, if there is a copy made to a buffer, you need to make sure that the buffer can handle your new size.

Also note, that on Windows the maximum command line size is 8,191 characters. On my Ubuntu box it is 2,097,152 characters (obtained via getconf ARG_MAX). So different platforms have different limits.

Thank you, that worked wonderfully. I bumped arg1 in two spots in spine.h to 32768 (I'm on CentOS 7.4) and I'm off and running.

Any reason to not crank this up to 8191 on the default spine distribution?

-jd