Upgrade 1.2.18 to 1.2.21 - Error: Maximum runtime of 298 seconds exceeded

[TheWitness]

Does the problem diminish if you change Max OIDs to say 10?

[warben61]

Good question. Testing now. I ran a query in mysql to change a whole bunch of the max_oids from 50 to 10.

[TheWitness]

Yea, I resort to doing things that way myself, and when I find one of those, I add it to the 'Change Settings' dropdown. But when you have > 10k devices, it's easier to just hit the database directly.

[warben61]

Yep.
Ok, well changing max oids doesn't appear to have had any impact. I've had 2 of the double free or corruption errors in the last 30 minutes.

[TheWitness]

Ugh. Okay. Working hours now. I'll dig into their API soon. Was this working before? What spine version?

[warben61]

Sounds good, thanks. Yeah it was fine before the updates. I was on Cacti 1.2.18 and I'm almost certain Spine 1.2.18 as well.

[TheWitness]

Oh crap, run the following Query:

Code: Select all

select distinct snmp_auth_protocol, snmp_priv_protocol from host;

Send the results back.

[warben61]

Results below. The blank row made me curious... so I did the next queries as well. The blank rows are all from hosts not using snmpv3. Not sure if that has anything to do with anything, but I figured I'd include that too.

Code: Select all

mysql> select distinct snmp_auth_protocol, snmp_priv_protocol from host;
+--------------------+--------------------+
| snmp_auth_protocol | snmp_priv_protocol |
+--------------------+--------------------+
| SHA                | AES128             |
|                    |                    |
| SHA                | DES                |
| MD5                | DES                |
+--------------------+--------------------+
4 rows in set (0.01 sec)

mysql> select count(*) from host where snmp_auth_protocol = '';
+----------+
| count(*) |
+----------+
|     3256 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from host where snmp_priv_protocol = '';
+----------+
| count(*) |
+----------+
|     3256 |
+----------+
1 row in set (0.00 sec)

select count(*) from host where snmp_version != '3';
+----------+
| count(*) |
+----------+
|     3256 |
+----------+
1 row in set (0.00 sec)

[TheWitness]

How many hosts at AES128? What I thought might have happened was not the case though.

[warben61]

2710 hosts at AES128. We did notice there are some new options in the dropdowns like AES128, 192, 256, whereas before we just chose AES. Same for SHA. I haven't seen any problems with the defaults though and I've just had our guys using AES128 since it seems to connect to our Mikrotiks no problem.

Code: Select all

mysql> mysql> select count(*) from host where snmp_priv_protocol = 'AES128';
+----------+
| count(*) |
+----------+
|     2710 |
+----------+
1 row in set (0.00 sec)

[TheWitness]

What's your poller interval and how close to the line are you?

[warben61]

Poller interval is 5 minutes. Typical runtime is 20-30 seconds.

[TheWitness]

So, looking at the net-snmp API, they seem to maintain a user list, which is a linked list of unique users for a session, and the free corruption happens there. So, now I have to find out if this always happens, :or if there is some condition where it's happening twice for a single device. That would be unusual.

So, do this, run the following query:

Code: Select all

SELECT host_id, COUNT(DISTINCT CONCAT(snmp_username, snmp_password, snmp_auth_protocol, snmp_priv_passphrase, snmp_priv_protocol, snmp_context, snmp_engine_id)) AS `combos` 
FROM poller_item 
WHERE snmp_priv_protocol='AES128' 
GROUP BY host_id 
HAVING combos > 1;

Let's hope it comes back empty.

[warben61]

That's gonna be interesting to track down. Yeah, no results on that query.

Code: Select all

mysql> SELECT host_id, COUNT(DISTINCT CONCAT(snmp_username, snmp_password, snmp_auth_protocol, snmp_priv_passphrase, snmp_priv_protocol, snmp_context, snmp_engine_id)) AS `combos` FROM poller_item WHERE snmp_priv_protocol='AES128' GROUP BY host_id HAVING combos > 1;
Empty set (0.07 sec)

[TheWitness]

Okay, taking a different look. Open poller.c and look for the string snmp_host_cleanup. That function will be called three times there. Comment all three and test running that way. Hand running first without installing.