spine 1.2.4 cannot run snmp queries that succeed in cmd.php

[ciela2]

Hello,
Trying to migrate Cacti Version 0.8.8h to Cacti 1.2.23. Have upgraded spine to 1.2.24

cmd.php overruns 5 minute polling cycle, but spine is failing to run a number of queries - in particular queries to F5 BigIP (which often have long snmp indexes). This occurs on F5 NAT devices and F5 Load Balancers.

Any help would be greatly appreciated.
Thanks

./spine -R -S -V 3 -f 98 -l 98

SPINE: Using spine config file [spine.conf]
Version 1.2.24 starting
Total[0.0178] Spine will support multithread device polling.
Total[0.0181] DEBUG: Initial Value of Available Threads is 6 (0 outstanding)
Total[0.0184] DEBUG: Available Threads is 5 (1 outstanding)
Total[0.0188] DEBUG: Available Threads is 4 (2 outstanding)
Total[0.0194] DEBUG: Device[98] HT[1] In Poller, About to Start Polling
Total[0.0203] Device[0] HT[1] Total Time: 0.0011 Seconds
Total[0.0210] Device[98] INFO: SNMP Device nat-a1:161' has a timeout of 1000000 (1000), with 5 retries
Total[0.0257] Device[98] DEBUG: Entering ICMP Ping
Total[0.0276] Device[98] DEBUG: Attempting to ping nat-a1, seq 0 (Retry 0 of 1)
Total[0.0282] Device[98] DEBUG: ICMP Device Alive, Try Count:1, Time:0.6061 ms
Total[0.0284] Device[98] DEBUG: Entering SNMP Ping
Total[0.0293] Device[98] Checking for System Information Update
Total[0.0294] Device[98] Updating Full System Information Table
Total[0.0340] DEBUG: Device[98] HT[1] RECACHE: Processing 6 items in the auto reindex cache for 'nat-a1'
Total[0.0421] Device[98] HT[1] DQ[1] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 16 = output: 16)
Total[0.0429] Device[98] HT[1] DQ[25] Legacy Uptime Result: 272704697, Is Numeric: 1
Total[0.0435] Device[98] HT[1] DQ[25] Extended Uptime Result: 272704700, Is Numeric: 1
Total[0.0435] Device[98] HT[1] DQ[25] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 272688100 < output: 272704700)
Total[0.0445] Device[98] HT[1] DQ[27] RECACHE OID: .1.3.6.1.4.1.3375.2.2.14.6.2, (assert: 5 = output: U)
Total[0.0445] Device[98] HT[1] DQ[27] RECACHE ASSERT FAILED: '5=U'
Total[0.0445] Device[98] HT[1] DQ[28] RECACHE OID: .1.3.6.1.4.1.3375.2.2.14.6.2, (assert: 5 = output: U)
Total[0.0445] Device[98] HT[1] DQ[28] RECACHE ASSERT FAILED: '5=U'
Total[0.0445] Device[98] HT[1] DQ[30] RECACHE OID: .1.3.6.1.4.1.3375.2.2.14.6.2, (assert: 5 = output: U)
Total[0.0446] Device[98] HT[1] DQ[30] RECACHE ASSERT FAILED: '5=U'
Total[0.0446] DEBUG: walk starts at OID .1.3.6.1.4.1.3375.2.1.1.2.20.30
Total[0.0465] Device[98] HT[1] DQ[31] RECACHE OID COUNT: .1.3.6.1.4.1.3375.2.1.1.2.20.30, output: 1
Total[0.0495] Device[98] HT[1] NOTE: There are '66' Polling Items for this Device
Total[0.0497] Device[98] HT[1] Total Time: 0.03 Seconds
Total[0.0505] DEBUG: Device[98] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
Total[0.5292] Time: 0.5283 s, Threads: 6, Devices: 2

[TheWitness]

Do a verbose query on this Data Query and post the output.

[ciela2]

Thanks - here is the verbose output from data query 30:

Data Query Debug Information
Total: 0.000000, Delta: 0.000000, Found data query XML file at '/apps/local/cacti-1.2.23/resource/snmp_queries/F5BIGIP_ltmLsnPoolStat11_6.xml'
Total: 0.000000, Delta: 0.000000, Running Data Query [30].
Total: 0.000000, Delta: 0.000000, Found Type = '3' [SNMP Query].
Total: 0.000000, Delta: 0.000000, XML file parsed ok.
Total: 0.000000, Delta: 0.000000, Auto Bulk Walk Size Selected.
Total: 0.030000, Delta: 0.030000, Tested Bulk Walk Size 1 with a response of 0.0324.
Total: 0.070000, Delta: 0.030000, Tested Bulk Walk Size 5 with a response of 0.0321.
Total: 0.090000, Delta: 0.020000, Tested Bulk Walk Size 10 with a response of 0.0228.
Total: 0.110000, Delta: 0.020000, Tested Bulk Walk Size 15 with a response of 0.0202.
Total: 0.130000, Delta: 0.020000, Tested Bulk Walk Size 20 with a response of 0.0202.
Total: 0.160000, Delta: 0.030000, Tested Bulk Walk Size 25 with a response of 0.0293.
Total: 0.160000, Delta: 0.000000, Bulk Walk Size selected was 20.
Total: 0.190000, Delta: 0.030000, Executing SNMP get for num of indexes @ '.1.3.6.1.4.1.3375.2.2.14.6.2' Index Count:
Total: 0.190000, Delta: 0.000000, Executing SNMP walk for list of indexes @ '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1' Index Count: 5
Total: 0.190000, Delta: 0.000000, Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.13.47.67.111.109.109.111.110.47.103.117.101.115.116' value: '/Common/guest'
Total: 0.190000, Delta: 0.000000, Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.16.47.67.111.109.109.111.110.47.97.99.97.100.101.109.105.99' value: '/Common/academic'
Total: 0.190000, Delta: 0.000000, Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.16.47.67.111.109.109.111.110.47.112.97.116.95.112.111.111.108' value: '/Common/pat_pool'
Total: 0.190000, Delta: 0.000000, Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.16.47.67.111.109.109.111.110.47.112.98.97.95.112.111.111.108' value: '/Common/pba_pool'
Total: 0.190000, Delta: 0.000000, Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.19.47.67.111.109.109.111.110.47.114.101.115.105.100.101.110.116.105.97.108' value: '/Common/residential'
Total: 0.190000, Delta: 0.000000, Filtering list of indexes @ '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1' Index Count: 5
Total: 0.190000, Delta: 0.000000, Filtered Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.13.47.67.111.109.109.111.110.47.103.117.101.115.116' value: '13.47.67.111.109.109.111.110.47.103.117.101.115.116'
Total: 0.190000, Delta: 0.000000, Filtered Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.16.47.67.111.109.109.111.110.47.97.99.97.100.101.109.105.99' value: '16.47.67.111.109.109.111.110.47.97.99.97.100.101.109.105.99'
Total: 0.190000, Delta: 0.000000, Filtered Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.16.47.67.111.109.109.111.110.47.112.97.116.95.112.111.111.108' value: '16.47.67.111.109.109.111.110.47.112.97.116.95.112.111.111.108'
Total: 0.190000, Delta: 0.000000, Filtered Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.16.47.67.111.109.109.111.110.47.112.98.97.95.112.111.111.108' value: '16.47.67.111.109.109.111.110.47.112.98.97.95.112.111.111.108'
Total: 0.190000, Delta: 0.000000, Filtered Index found at OID: '.1.3.6.1.4.1.3375.2.2.14.6.3.1.1.19.47.67.111.109.109.111.110.47.114.101.115.105.100.101.110.116.105.97.108' value: '19.47.67.111.109.109.111.110.47.114.101.115.105.100.101.110.116.105.97.108'
Total: 0.190000, Delta: 0.000000, Inserting index data for field 'LsnPSIndex' [value='13.47.67.111.109.109.111.110.47.103.117.101.115.116']
Total: 0.190000, Delta: 0.000000, Inserting index data for field 'LsnPSIndex' [value='16.47.67.111.109.109.111.110.47.97.99.97.100.101.109.105.99']
Total: 0.190000, Delta: 0.000000, Inserting index data for field 'LsnPSIndex' [value='16.47.67.111.109.109.111.110.47.112.97.116.95.112.111.111.108']
Total: 0.190000, Delta: 0.000000, Inserting index data for field 'LsnPSIndex' [value='16.47.67.111.109.109.111.110.47.112.98.97.95.112.111.111.108']
Total: 0.190000, Delta: 0.000000, Inserting index data for field 'LsnPSIndex' [value='19.47.67.111.109.109.111.110.47.114.101.115.105.100.101.110.116.105.97.108']
Click to show Data Query output for field 'LsnPoolStatName'
Total: 0.220000, Delta: 0.010000, Checking for Sort Field change. No changes detected.
Total: 0.220000, Delta: 0.000000, Update Data Query Sort Cache complete
Total: 0.220000, Delta: 0.000000, Found 8 Local Data ID's to Verify
Total: 0.220000, Delta: 0.010000, Verification of 8 Local Data ID's Complete
Total: 0.220000, Delta: 0.000000, Index Association with Local Data complete
Total: 0.230000, Delta: 0.010000, No Index Changes Detected, Skipping Re-Index and Poller Cache Re-population
Total: 0.230000, Delta: 0.000000, Automation Executing for Data Query complete
Total: 0.230000, Delta: 0.000000, Plugin hooks complete

[TheWitness]

What happens when you do an SNMP get for this oid? .1.3.6.1.4.1.3375.2.2.14.6.2

[ciela2]

snmpget:
SNMPv2-SMI::enterprises.3375.2.2.14.6.2 = No Such Instance currently exists at this OID

A snmpget doesn't work, but a snmpwalk does.
SNMPv2-SMI::enterprises.3375.2.2.14.6.2.0 = INTEGER: 5

A walk appends the .0 to the OID (and so does a snmpget in my MIB browser) which is necessary.
Thanks.

[TheWitness]

Does the data query use the Walk or Get method? If you are using Get, you might want to consider using Walk. Or get that .0 to the end of the OID for the index or index count.

[ciela2]

First I should mention that we have the F5 NAT boxes which these queries are on, but we also have F5 load balancers and none of those queries work either - in spine. All of these queries work fine in Cacti 0.8.8h - cmd.php

I added a 0 to the index count oid as you suggested and can now get back the correct number of indexes when I do a verbose query. But still seeing -nan in the graphs.

All of the entries in the .xml file are walks.

The simplest F5 query is the CPU one - this one also fails in spine, even though I can change all of the fields in the .xml query file into input fields and see correct data come back in a verbose query.

As this is mostly an F5 issue (also see some failures in some of the RFC1213 graphs), I tried increasing the timeout, but that did not help.

It's baffling, If there's anything else I can try or check, please let me know. Appreciate your help with this.

[TheWitness]

Post the resource XML.

[ciela2]

<ltmLsnPoolStat>
<name>Get F5 Big-IP ltmLsnPoolStat v11.6 stats</name>
<oid_index>.1.3.6.1.4.1.3375.2.2.14.6.3.1.1</oid_index>
<oid_index_parse>OID/REGEXP:^.*\.3375\..{12}\.\d+\.(.*)$</oid_index_parse>
<index_order>LsnPSIndex</index_order>
<index_order_type>alphabetic</index_order_type>
<fields>
<LsnPSIndex>
<name>Index</name>
<source>index</source>
<direction>input</direction>
</LsnPSIndex>
<LsnPoolStatName>
<name>Pool Name</name>
<method>walk</method>
<source>value</source>
<direction>input</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.1</oid>
</LsnPoolStatName>
<LsnPSTotEndPts>
<name>Maximum EndPoints</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.12</oid>
</LsnPSTotEndPts>
<LsnPSActPortBlocks>
<name>Active Port Blocks</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.24</oid>
</LsnPSActPortBlocks>
<LsnPSActClientAtLim>
<name>Active Clients at Limit</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.25</oid>
</LsnPSActClientAtLim>
<LsnPSActZombiePBs>
<name>Active Zombie Port Blocks</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.26</oid>
</LsnPSActZombiePBs> <LsnPSTotClientLim>
<name>Total Clients at Limit</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.27</oid>
</LsnPSTotClientLim>
<LsnPSTotalPBAs>
<name>Total Port Blocks Allocated</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.28</oid>
</LsnPSTotalPBAs>
<LsnPSTotPBAsFails>
<name>Total Port Block Alloc Failures</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.29</oid>
</LsnPSTotPBAsFails>
<LsnPSTotPBDealloc>
<name>Total Port Block Deallocations</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.30</oid>
</LsnPSTotPBDealloc>
<LsnPSTotZombiePBCre>
<name>Total Zombie PBs Created</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.31</oid>
</LsnPSTotZombiePBCre>
<LsnPSTotZombiePBDel>
<name>Total Zombie PBs Deleted</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.32</oid>
</LsnPSTotZombiePBDel>
<LsnPSTotZombiePBKld>
<name>Total Zombie PBs Killed</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.33</oid>
</LsnPSTotZombiePBKld>
</fields>
</ltmLsnPoolStat>

[TheWitness]

I don't see that OID in that resource file:

1.3.6.1.4.1.3375.2.2.14.6.2

Are you sure you have the right Resource XML file. Grep the directory for that OID.

[ciela2]

Sorry - that's the OID that returns how many indexes there are - I had removed it from the .xml file to see if it would help. Here's the original xml file:

<ltmLsnPoolStat>
<name>Get F5 Big-IP ltmLsnPoolStat v11.6 stats</name>
<index_order_type>numeric</index_order_type>
<oid_index>.1.3.6.1.4.1.3375.2.2.14.6.3.1.1</oid_index>
<oid_index_parse>OID/REGEXP:^.*\.3375\..{12}\.\d+\.(.*)$</oid_index_parse>
<oid_num_indexes>.1.3.6.1.4.1.3375.2.2.14.6.2</oid_num_indexes>
<index_order>LsnPSIndex</index_order>
<index_order_type>alphabetic</index_order_type>
<fields>
<LsnPSIndex>
<name>Index</name>
<source>index</source>
<direction>input</direction>
</LsnPSIndex>
<LsnPoolStatName>
<name>Pool Name</name>
<method>walk</method>
<source>value</source>
<direction>input</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.1</oid>
</LsnPoolStatName>
<LsnPSTotEndPts>
<name>Maximum EndPoints</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.12</oid>
</LsnPSTotEndPts>
<LsnPSActPortBlocks>
<name>Active Port Blocks</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.24</oid>
</LsnPSActPortBlocks>
<LsnPSActClientAtLim>
<name>Active Clients at Limit</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.25</oid>
</LsnPSActClientAtLim>
<LsnPSActZombiePBs>
<name>Active Zombie Port Blocks</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.26</oid>
</LsnPSActZombiePBs>
<LsnPSTotClientLim>
<name>Total Clients at Limit</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.27</oid>
</LsnPSTotClientLim>
<LsnPSTotalPBAs>
<name>Total Port Blocks Allocated</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.28</oid>
</LsnPSTotalPBAs>
<LsnPSTotPBAsFails>
<name>Total Port Block Alloc Failures</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.29</oid>
</LsnPSTotPBAsFails>
<LsnPSTotPBDealloc>
<name>Total Port Block Deallocations</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.30</oid>
</LsnPSTotPBDealloc>
<LsnPSTotZombiePBCre>
<name>Total Zombie PBs Created</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.31</oid>
</LsnPSTotZombiePBCre>
<LsnPSTotZombiePBDel>
<name>Total Zombie PBs Deleted</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.32</oid>
</LsnPSTotZombiePBDel>
<LsnPSTotZombiePBKld>
<name>Total Zombie PBs Killed</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.3375.2.2.14.6.3.1.33</oid>
</LsnPSTotZombiePBKld>
</fields>
</ltmLsnPoolStat>

[TheWitness]

You need to add the .0 after the 2, then reindex.

So, restore the original file and the make the change, the verbose query.

[ciela2]

Followed your suggestions - restored original xml files, added zero to the num_indexes OID, but this didn't solve the problem.

I ran a diff on the files & I found that tabs had been used in some of the lines in the xml files as alignment - while most of the lines have leading spaces. Also ran dos2unix on the xml file since I found CR/LF endings in the file.

When I replaced all of the tabs with spaces the query began working on six of the eight graphs on this device.

However two of the graphs continue to show 'nan' even though I can see values for the variables in the log file (I have debug on for both NAT boxes). Realtime graphs on these two graphs also show 'nan'.

Example from log file of variable showing nan in graph:
dsname: LsnPSTotPBDealloc, oid: .1.3.6.1.4.1.3375.2.2.14.6.3.1.30.16.47.67.111.109.109.111.110.47.97.99.97.100.101.109.105.99, value: 5191816

We have two NAT boxes. One of them is partially working, but nothing works on the second box & it has also has two non-existent data queries showing up in the log file #25 & #26 - which don't exist - don't know how to fix this.

Thanks for helping with this - any suggestions greatly welcome.

[ciela2]

Screen shot attached of F5 Nat graphs running in cmd.php and broken in spine.
Two of the eight graphs are blank - nan - they don't run in Cacti 1.2.23 cmp.php either, but they work in Cacti 0.8.8h.

Below is partial output from spine : as you can see the "host ignore flag is active"
Do not see a Data Query[25] or Data Query[26] in our list of Data Queries - ??? Don't know if something got out of sync when the db was imported from Cacti 0.8.8h and upgraded ?

Is there anything I can try or test ?
It looks like we'll have to keep our production Cacti on version 0.8.8h, which is disappointing.
We have a dev version of Cacti on version 1.2.23 that I can set these graphs up on in case this gets straightened out at some point.

Thanks,

Screen Shot 2023-03-30 at 3.18.01 PM.png (718.41 KiB) Viewed 838 times

Result: ICMP: Device is Alive
Total[0.0285] Device[99] SNMP Result: Device responded to SNMP
Total[0.0285] Device[99] Checking for System Information Update
Total[0.0285] Device[99] Updating Full System Information Table
Total[0.0333] DEBUG: Device[99] HT[1] RECACHE: Processing 6 items in the auto reindex cache for [IP address]
Total[0.0342] ERROR: No such Instance for oid '.1.3.6.1.4.1.3375.2.2.14.6.2' for Device[99] with Status[1]
Total[0.0342] Device[99] HT[1] DQ[25] RECACHE OID: .1.3.6.1.4.1.3375.2.2.14.6.2, (assert: = output: U)
Total[0.0342] Device[99] HT[1] DQ[25] RECACHE ASSERT FAILED: '=U'
Total[0.0342] WARNING: Skipped oid '.1.3.6.1.4.1.3375.2.2.14.6.2' for Device[99] as host ignore flag is active
Total[0.0342] Device[99] HT[1] DQ[26] RECACHE OID: .1.3.6.1.4.1.3375.2.2.14.6.2, (assert: = output: U)
Total[0.0343] Device[99] HT[1] DQ[26] RECACHE ASSERT FAILED: '=U'
Total[0.0343] WARNING: Skipped oid '.1.3.6.1.4.1.3375.2.2.14.6.2.0' for Device[99] as host ignore flag is active
Total[0.0343] Device[99] HT[1] DQ[27] RECACHE OID: .1.3.6.1.4.1.3375.2.2.14.6.2.0, (assert: 5 = output: U)
Total[0.0343] Device[99] HT[1] DQ[27] RECACHE ASSERT FAILED: '5=U'
Total[0.0343] WARNING: Skipped oid '.1.3.6.1.4.1.3375.2.2.14.6.2.0' for Device[99] as host ignore flag is active
Total[0.0343] Device[99] HT[1] DQ[28] RECACHE OID: .1.3.6.1.4.1.3375.2.2.14.6.2.0, (assert: 5 = output: U)
Total[0.0343] Device[99] HT[1] DQ[28] RECACHE ASSERT FAILED: '5=U'
Total[0.0343] WARNING: Skipped oid '.1.3.6.1.4.1.3375.2.2.14.6.2.0' for Device[99] as host ignore flag is active
Total[0.0343] Device[99] HT[1] DQ[30] RECACHE OID: .1.3.6.1.4.1.3375.2.2.14.6.2.0, (assert: 5 = output: U)
Total[0.0343] Device[99] HT[1] DQ[30] RECACHE ASSERT FAILED: '5=U'
Total[0.0343] DEBUG: walk starts at OID .1.3.6.1.4.1.3375.2.1.1.2.20.30
Total[0.0361] Device[99] HT[1] DQ[31] RECACHE OID COUNT: .1.3.6.1.4.1.3375.2.1.1.2.20.30, output: 1
Total[0.0387] Device[99] HT[1] NOTE: There are '139' Polling Items for this Device