Strange behavior with cmd.php/spine and a script

[nebj00la]

For some reason, which I can't figure out, this is working with cmd.php, and not spine. Thought it works with cmd.php, it throws errors every time, which leads me to believe something is not right either way.

Here's the data query:

Code: Select all

<interface>
        <name>Get Cisco ASA/PIX VPN Statistics</name>
        <script_path>perl |path_cacti|/scripts/query_lan2lan_cisco.pl</script_path>
        <arg_prepend>|host_snmp_community| |host_hostname| ASA</arg_prepend>
        <arg_get>get</arg_get>
        <arg_index>index</arg_index>
        <arg_query>query</arg_query>
        <output_delimeter>:</output_delimeter>
        <index_order>PeerIP</index_order>
        <index_order_type>numeric</index_order_type>
        <index_title_format>|chosen_order_field| IPSec Traffic</index_title_format>

        <fields>
                <PeerIP>
                        <name>Peer IP</name>
                        <direction>input</direction>
                        <query_name>index</query_name>
                </PeerIP>

                <RX>
                        <name>Received Traffic</name>
                        <direction>output</direction>
                        <query_name>RX</query_name>
                </RX>
                <TX>
                        <name>Transmitted Traffic</name>
                        <direction>output</direction>
                        <query_name>TX</query_name>
                </TX>
        </fields>
</interface>

Here's what spine returns in the log, and nothing works:

Code: Select all

SPINE: Poller[0] Host[3] ERROR: Empty result [10.2.3.4]: 'perl /usr/local/cacti/scripts/query_lan2lan_cisco.pl community 10.2.3.4 ASA get index 10.2.3.4' 

Here's what cmd.php returns in the log, and it works (yet the error shows):

Code: Select all

CMDPHP: Poller[0] ASSERT: '10.2.3.4=' failed. Recaching host '10.2.3.4', data query #11

Here is the output of a verbose query:

Code: Select all

+ Running data query [11].
+ Found type = '4 '[script query].
+ Found data query XML file at '/usr/local/cacti/resource/script_queries/cisco_asa_vpn_tunnel.xml'
+ XML file parsed ok.
+ Executing script for list of indexes 'perl /usr/local/cacti/scripts/query_lan2lan_cisco.pl community 1.2.3.4 ASA index'
+ Executing script query 'perl /usr/local/cacti/scripts/query_lan2lan_cisco.pl community 1.2.3.4 ASA query index'
+ Found item [PeerIP='10.1.2.3'] index: 10.1.2.3
+ Found item [PeerIP='10.1.2.4'] index: 10.1.2.4
+ Found item [PeerIP='10.2.3.8'] index: 10.2.4.8
+ Found item [PeerIP='10.4.8.12'] index: 10.4.8.12
+ Found data query XML file at '/usr/local/cacti/resource/script_queries/cisco_asa_vpn_tunnel.xml'
+ Found data query XML file at '/usr/local/cacti/resource/script_queries/cisco_asa_vpn_tunnel.xml'
+ Found data query XML file at '/usr/local/cacti/resource/script_queries/cisco_asa_vpn_tunnel.xml'
+ Found data query XML file at '/usr/local/cacti/resource/script_queries/cisco_asa_vpn_tunnel.xml'

Here's the output of various commands from the script:

perl query_lan2lan_cisco.pl community 10.2.3.4 ASA index
1.2.3.4
5.6.7.8
1.2.3.1
4.3.2.1

I believe this is where the issue is...

perl query_lan2lan_cisco.pl community 10.2.3.4 ASA query index
1.2.3.4:1.2.3.4
5.6.7.8:5.6.7.8
1.2.3.1:1.2.3.1
4.3.2.1:4.3.2.1

I believe this should be returning an index value, and not the same thing twice.

I have attached the script. I gathered it from http://forums.cacti.net/viewtopic.php?t=27211

Any suggestions?

[TheWitness]

The script is defective. Since it uses multiple prints to output the ASA specs, it will fail under spine. You need to, internal to the script, maintain a buffer and output all at once.

TheWitness

[nebj00la]

The script is defective. Since it uses multiple prints to output the ASA specs, it will fail under spine. You need to, internal to the script, maintain a buffer and output all at once.

TheWitness

Thanks for the information!

[Setarcos]

Hi folks,

I'm one of the authors (to be honest, I tweaked the original author's script a bit) of said script and was somewhat surprised to hear TheWitness answer, as I do use both spine and the script in my production environment.

There are quite a few other script query scripts (including those shipped with Cacti) that employ the multiple print method to do their thing, so the implication here is that a more fundamental issue must exist.

Larry, would you mind explaining the multiple print thing in a bit more detail?

Thanks!

[nebj00la]

I'd like to bring this discussion back into light, as I must change from cmd.php to spine. Performance issues are preventing this from working efficiently.

Setarcos, can you let me know what's going on when the following is executed? It's taking 30 seconds to return the value:

perl /usr/local/cacti/scripts/query_lan2lan_cisco.pl community a.b.c.d ASA get TX w.x.y.z

I know it's gathering the TX statistics for peer host w.x.y.z, what specifically is happening in the background?

[Setarcos]

I'd like to bring this discussion back into light, as I must change from cmd.php to spine. Performance issues are preventing this from working efficiently.

Setarcos, can you let me know what's going on when the following is executed? It's taking 30 seconds to return the value:

perl /usr/local/cacti/scripts/query_lan2lan_cisco.pl community a.b.c.d ASA get TX w.x.y.z

I know it's gathering the TX statistics for peer host w.x.y.z, what specifically is happening in the background?

Hi nebj00la,

The script walks the peer (cikeTunRemoteValue), IKE peer index (cipSecTunIkeTunnelIndex), TX (cipSecTunOutOctets) and RX (cipSecTunInOctets) OIDs to get the data needed to present a sum of the IPSec TX and RX for each IKE peer. If it is taking 30 seconds for a single TX value, you must have both a lot of IKE peers/IPSec tunnels and/or high latency between your Cacti box and the ASA.

The original script which I started with used SNMPv1, which will definitely have performance issues in this type of situation. Try the attached version to see if it improves anything. If not, the way the tables are being walked will need to be re-worked to do bulkwalks instead to decrease the polling time.

[nebj00la]

Thanks, I'll try this first thing tomorrow morning.

[nebj00la]

The v2c improves response time by 8 seconds, ie. 12 seconds instead of 20 seconds.

This particular ASA has 90ms delay with 15 tunnel groups.

Code: Select all

smokeping: min:92.420 avg:92.496 max:92.598 dev:0.416 loss:0

[nebj00la]

I'd like to understand why cacti/spine is trying to get the following information, which clearly fails due to no logic for the argument "get index" in the script:

w.x.y.z = ASA
a.b.c.d = Peer IP

Code: Select all

06/02/2010 12:03:10 PM - SPINE: Poller[0] Host[7] ERROR: Empty result [w.x.y.z]: 'perl /usr/local/cacti/scripts/query_lan2lan_cisco_v2.pl community w.x.y.z ASA get index a.b.c.d'

I understand "query index" is valid, why is "get index" followed by one of the peer IP addresses being passed?

[gandalf]

if "index" is a valid object, this should be ok. But it depends on the exact script code.
R.

[nebj00la]

if "index" is a valid object, this should be ok. But it depends on the exact script code.
R.

Yeah, that's the problem. index is not a valid get object. Why would cacti be asking for it? Is it a standard that the script needs to be adjusted to meet?

Right now the index displays the available peer IP addresses on the ASA. Cacti is trying to run "get index" for each of those peer IP addresses, in addition to the original index. Logically, I don't think there is such a thing. Once the PeerIP is found, the only things we care about are the RX/TX values which we're already getting.

Let me know if I'm not making any sense.

[Setarcos]

if "index" is a valid object, this should be ok. But it depends on the exact script code.
R.

Yeah, that's the problem. index is not a valid get object. Why would cacti be asking for it? Is it a standard that the script needs to be adjusted to meet?

Right now the index displays the available peer IP addresses on the ASA. Cacti is trying to run "get index" for each of those peer IP addresses, in addition to the original index. Logically, I don't think there is such a thing. Once the PeerIP is found, the only things we care about are the RX/TX values which we're already getting.

Let me know if I'm not making any sense.

Yea, some scripts implement this and others do not (and things work fine). I too am not quite sure how it could be used, but could easily add it to this script.

As for your ASA, I imagine the script even with the v2c enhancement is still not going to work well for you due to the latency. Spine will try to poll all the data sources for this host with a single thread, eventually running out of time to get them all before the next polling cycle starts. Also, both the TX and RX for an IKE peer must be polled individually, as indexed script queries can only have a single return value.

Things are a little hectic right now, but I might be able to spend some cycles in the next few days re-working the code to do bulkwalks instead of the way it is being done currently.

[nebj00la]

That would be great. Thanks for the update!

[gandalf]

... but I might be able to spend some cycles in the next few days re-working the code to do bulkwalks instead of the way it is being done currently.

cact_snmp_walk does the bulkwalks for you. But you will have to make your code SNMP V3 compliant (add some SNMP V3 parameters to the "old" function call).
R.

[Setarcos]

... but I might be able to spend some cycles in the next few days re-working the code to do bulkwalks instead of the way it is being done currently.

cact_snmp_walk does the bulkwalks for you. But you will have to make your code SNMP V3 compliant (add some SNMP V3 parameters to the "old" function call).
R.

Thanks Gandalf,

I think Cisco finally added support for SNMP V3 in later versions of the ASA firmware, but it isn't supported on the 8.0.x versions I have (only v1, v2c). Also i believe this utility function is only available to PHP scripts (this script is written in Perl)