[SOLVED] Gaps / "nan" returned

[khufure]

Sadly, my gaps came back again. Found a table corruption. After running the fix, poller is processing better. Well, so far. It's been about 1 hour.

I am now running hacks to both kill old poller processes and repair tables, every 10 minutes. I'd prefer not to run these, but better these than the machine runs out of memory and OOMs important stuff. And cacti is too useful to turn off.

# kill old cacti processes
*/10 * * * cacti /usr/local/bin/find_kill_old_cacti_procs.sh
*/10 * * * root /usr/local/bin/repair_cacti_tables.sh

</usr/local/bin/repair_cacti_tables.sh>
#!/bin/bash
# repair cacti tables
/usr/bin/myisamchk --silent --force --fast --update-state --key_buffer_size=64M --sort_buffer_size=64M --read_buffer_size=1M --write_buffer_size=1M /var/lib/mysql/cacti/*.MYI
</>

[khufure]

That makes things more complicated. Will try to investigate. But no promise made ...
Reinhard

Yeah understood, it's frustrating though

I think the hung processes was from corruption/locks in the DB. The tech support page found some entries. Hope so anyway!

[gandalf]

So I'll drop it, then.
Reinhard

[khufure]

Update : my hacks are successful in making Cacti stable now. The gaps prove to be on the bandwidth graphs of one server.

What can be done to troubleshoot whatever might be wrong on the bandwidth graphs? If I had to guess, I suspect it is a 64-bit problem.

[gandalf]

First, see what cacti.log says about it.
Reinhard

[foo]

Same problem. Cacti hangs randomly on certain hosts, usually the same hosts. Gives SNMP timeout errors, however, snmpd is working fine on those boxes. I can manually poll that information, and get it instantly. Also tried raising the SNMP timeout for those machines to 5+ seconds. No change.

Switching back to cmd.php works, but it's not ideal.

This started when I upgraded to 0.8.7b. I am running the latest spine.

[ccogdill]

I too ran into the issue of Cacti hanging on various hosts then eventually timing out but traced it down to a problem with using SNMP v2 in the Cacti interface. Once I changed the SNMP version to V1. The problem went away and the host no longer timed out.

[foo]

I too ran into the issue of Cacti hanging on various hosts then eventually timing out but traced it down to a problem with using SNMP v2 in the Cacti interface. Once I changed the SNMP version to V1. The problem went away and the host no longer timed out.

That's a workaround, not a resolution. Doing manual queries works fine with snmpv2. The bug is with Cacti. I actually stopped using Cacti due to this bug, and several, several others.

[gandalf]

I too ran into the issue of Cacti hanging on various hosts then eventually timing out but traced it down to a problem with using SNMP v2 in the Cacti interface. Once I changed the SNMP version to V1. The problem went away and the host no longer timed out.

Are you using php-snmp or net-snmp libraries, then? Please report versions of those packages
Reinhard

[ccogdill]

Net-SNMP version 5.0.9.

I used the SNMP agent that comes pre-installed on Solaris 10.

[gandalf]

I suspect an issue with snmpbulkwalk. This is only triggered when using SNMP V2. Net-snmp 5.0.9 is waaay old. Perhaps bulkwalk is broken, there.
Reinhard

[aleto]

After upgrading to cacti 0.8.7g this probleem seems to have returned. Except that graphs are not updating AT ALL. Sometimes it updates..

Any suggestions? Ive seen the "poller bug". Poller time is 1 minute cron and 1 minute interval.