Upgrade 1.2.18 to 1.2.21 - Error: Maximum runtime of 298 seconds exceeded

[warben61]

Hey there. Half good news - I haven't seen any segfaults yet. But it did do the corruption thing again. At 2:55pm the errors below shows up in cacti_stderr.log. I had a debug running on spine but I didn't notice it until about 3:04, almost 10 minutes after the error. I hope that's not too much time passing. GDB log attached.

2:55pm:
double free or corruption (fasttop)
FATAL: Spine Interrupted by Abort Signal
Unable to flush stdout: Broken pipe

[TheWitness]

Follow the link. Is that's what's logged

https://sourceware.org/legacy-ml/gdb/20 ... 00001.html

[warben61]

...yes? I think so. I see some of the same lines in my last trace:

Reading symbols from ./spine...
[Thread debugging using libthread_db enabled]
[New Thread 0x7f14c4627700 (LWP 1213819)]
Thread 4 "spine" received signal SIG33, Real-time event 33.
[Switching to Thread 0x7f4d6bfff700 (LWP 1190359)]

I looked back through other traces I've done. Sometimes it says thread 4 received sig33, other times it says thread 35 or thread 36. Not sure if that's a thing or not but I figured I'd mention.

[TheWitness]

Have you been able to capture any core files?

[warben61]

I'll try to get that for you now. I've never done a core dump before... did I set this up right? Just checking to make sure I get the right information.

ulimit -S -c 0
ulimit -S -c unlimited spine
gdb ./spine
run

And then after it happens: generate-core-file

I did that once so far and got a file that's 3.6Gb in size. Does that sound correct?

[TheWitness]

You should also get that running outside of gdb right?

[TheWitness]

What I would do is add that and the core dump pattern to the Apache user and then make sure we get cores from real runs.

[warben61]

Ok yeah, I see what you're saying. I've got it set up. Now just waiting for it to fail again.

[warben61]

Got something. I walked away for a few minutes and came back to 2 core dumps. Not a segfault yet. Both of these show this in the log:

double free or corruption (fasttop)
FATAL: Spine Interrupted by Abort Signal

https://drive.google.com/drive/folders/ ... sp=sharing
core-spine.2110396
core-spine.2132369

[warben61]

I think I caught a segfault in the core dumps. They're added to the google drive folder. There have been 5 failures since I last checked and 2 of them say seg fault in the log. Based on the timestamps of the core dumps and the fact that the segfaults were the first 2 of 5 failures, I believe they're capture in these files:
core-spine.2858258
core-spine.2971815

[TheWitness]

Cool. Do a backtrace and backtrace full on each of them and upload.

[warben61]

Attached. Thanks!

[TheWitness]

This is looking more and more like a net-snmp v3 client bug. What version of net-snmp libraries are you using? What's your OS version? Can you send a note out to the net-snmp team and see if they are aware of such issues?

[TheWitness]

Lastly, let me know the snmpv3 details: Page capture you can obfuscate the details on the passphrases and passwords, but what I would like to know:

- Obfuscated screen captures
- Length of passwords and passphrases
- Length of username
- Special character use, and the special characters you used in either the username, password, or passphrases
- Auth Proto and Privacy proto?
- Using Engine ID?
- Using Context?

[warben61]

Interesting. Assuming I'm looking at the right thing... I've got net-snmp libraries version 5.8. Ubuntu 20.04. I'm happy to reach out to them, but I don't know what I need to ask? Is it crashing on the queries?

root@oss:/var/log/remote# net-snmp-config --version
5.8

root@oss:/var/log/remote# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

Screen captures attached of a device config
SNMPv3 username, password, and passphrases do not have special characters. They're a mix of lowercase letters and numbers.
Username: 9 characters long
Password: SHA, 13 characters long
Passphrase: AES-128, 13 characters long
Screenshot of SNMP config is in Device Details 02.
Not using EngineID or Context