Cactid processes climbing and gaps in graphs

[lard]

Hi,

Thanks for that - though I know why they are failing - it's the poller timeout after 296 that is backing up everything and causing the gaps in the graphs, the invalid SNMP queries have been with me for a while (before and after the gaps in the graphs)

Thanks,

Lard

[lard]

Also of note is that when this happens I have a cactid.exe process that hangs...stopped them climbing indefinately by removing all graphs that use scripts but so far have not had much luck in resolving this,

Thanks,

Lard

[lard]

O.K - some more information which may shed some light on this problem - this is from running php poller from a DOS prompt to try to see where this fails:

CACTID: Host[205] DS[5257] SNMP: v1: hhhhhhhh, dsname: traffic
_out, oid: .1.3.6.1.2.1.2.2.1.16.4, value: 4013775797
CACTID: Host[178] DS[5096] SNMP: v1: hhhhhhhh, dsname: traffic_
out, oid: .1.3.6.1.2.1.2.2.1.16.40, value: 880
CACTID: Host[206] DS[5264] SNMP: v1: hhhhhhhhh, dsname: traffi
c_in, oid: .1.3.6.1.2.1.2.2.1.10.4, value: 3835694595 261 [unknown (0x1AF4)]
cactid 11044 handle_exceptions: Exception: STATUS_ACCESS_VIOLATION

1239 [unknown (0x1AF4)] cactid 11044 open_stackdumpfile: Dumping stack trace
to cactid.exe.stackdump

Stackdump file shows:

Exception: STATUS_ACCESS_VIOLATION at eip=610AA5A5
eax=55000000 ebx=55000000 ecx=610E88A0 edx=00750000 esi=00000000 edi=54FFFFF8
ebp=22B2B988 esp=22B2B960 program=e:\cactid\cactid.exe, pid 11044, thread unknown (0x1AF4)
cs=001B ds=0023 es=0023 fs=003B gs=0000 ss=0023
Stack trace:
Frame Function Args
22B2B988 610AA5A5 (00000000, 00DE0090, 22B2B9C8, 6104EC66)
22B2B9B8 6104EC66 (55000000, 008B1E38, 22B2B9F8, 00417E42)
22B2B9C8 610844FF (008B1E38, 35363639, 30363234, 282C2927)
22B2B9F8 00417E42 (0087DDC0, 007EE420, 22B2EB58, 00407E20)
22B2BA08 00404114 (0087DDC0, 01C7A430, 00000032, 0000002C)
22B2EB58 00407E20 (00000061, 000003FF, 004AB930, 00000000)
22B2EF88 00404AC1 (007CD8F8, FFFFFFFF, 22B2EFC8, 61003B7D)
22B2EFC8 6109D17E (007EE420, 22B2F000, 6109D110, 00000000)
22B2EFF8 61003E94 (00000000, 00000000, 00000000, 00000000)
22B2FF98 61003EDA (00000000, 00000000, 00000000, 00000000)
End of stack trace

Does this mean anything to anyone?

Thanks,

lard

[lard]

..hmm...and the following strange file created in the cactid folder

[lard]

And this file contains the following:

12/06/2005 09:43:49 AM - POLLER: Poller[0] CACTI2RRD: E:\rrdtool/rrdtool.exe update E:\Apache2\htdocs\cacti\rra\pl2b13548_traffic_in_3612.rrd --template traffic_out:traffic_in 1133862180:1369378239:84300126

[lard]

Well - all has been stable for a few hours and I have not seen the "time exceeded" message,

I am also now seeing:

ERROR: HOST EVENT: Host is DOWN Message: Host did not respond to SNMP

Which I never did before - could this have been the route of the problem??

Oh well - fingers crossed that this doesnt return - I'll now load back in all my script based graphs and wish for the best!

Thanks,

Lard

[BSOD2600]

Are you using the latest version of Cactid? If so, forward that stack dump to TheWitness.

As per your downed host, can you use GetIF or snmpwalk on it from the cacti server? What do you have your uptime detection method set to in Cacti? Try changing it.

[lard]

Hi,

The cactid version is f rather than f-1 so may already have been resolved - either way the downed host notification is good as previously it wouldn't show this in the log - just the "296 seconds exceeded" message,

I am using SNMP for testing the devices as I have a few firewalls with ICMP restrictions - though one query is that some of the devices I manage are very busy and can timeout mid SNMP query as they have a lot of interfaces (sub interfaces and vlans on 7300's) and am wondering if this would cause the timeout whilst mid poll (after checking it was up then starting the snmp get for each interface) or is it the case that if any failure to respond to an SNMP query rather than just the up/down check will cause the host to timeout and move onto the next?

Many thanks,

Lard

[lard]

In fact - just thinking that a few of my devices have got 1200+ items from the SNMP interface query....if it started an SNMP get for each one at a timeout of 500ms (plus retry) would it move onto the next host or the next SNMP target on the same host?

Thanks,

Lard

[BSOD2600]

Yea, cactid 0.8.6f had some memory problems or something, hence the quick f-1 release. Try that or the 0.8.6g beta in the annoucement forum.

You're probably experiance the timeouts with so many snmp gets. Cactid 0.8.6g and Cacti 0.8.6h will support SNMP BULKGETS, which should dramatically speed up the querying of large hosts.

[lard]

Excellent news - have seen the improvement bulkgets have on other SNMP platforms!

I may well wander over and try out the beta of cactid - are you aware of when cacti 0.8.6h is likely to be released?

You've cheered me up no end!

Thanks,

Lard

[BSOD2600]

cacti 0.8.6h is in beta testing right now... if you feel brave, download the SVN and use it in a preproduction enviroment, of course.