CACTID Issues - Windows [RESOLVED]

[mcutting]

I am trying to get CACTID to work. I have downloaded the 0.8.6.i version (Cywin 1.5.21), and have this, along with the other required dll's (note, I have not installed Cygwin).

Sometimes, CACTID will work fine fine, giving a poll time of around 47 seconds (great !). However, most of the time, it will stall, and I see the process of SH.EXE hanging in task manager, taking 99% of the CPU. The poller then eventually times out.

The poller then promptly starts again, and I end up with a stack of cactid.exe processes running. I have also had to place the SH.EXE file in a directory on the root of C: called "Bin" otherwise the custom PERL and VBSCRIPTS I have will NOT work, despite the paths having "/" and not "\"

I'm pulling my hair out with this one ! Can anyone help ?

Thanks

[BSOD2600]

cactid is a picky thing. It requires the full path for binaries if its executing any scripts (i.e. instead of simply perl.exe in the parameter line, you must specify c:\perl\bin\perl.exe). Are you running any scripts or is it purely snmp queries?

It's possible you're duplicate/different versions of sh.exe are a problem. Why not engineer the scripts to use the sh.exe from the system path or the one installed with cactid.exe.

[TheWitness]

If you don't want full paths to binaries, use the SVN branch version. I have reverted that behavior. Also, if you want to run error free, read the readme on making it. Here are some brief instructions

1) Download net-snmp 5.1.x source. Don't go later than that, it won't compile on Windows
2) Compile net-snmp with the following options:

./configure --wihtout-agent --disable-mib-loading
make
make install

3) Download MySQL Unix Source (Not Windows)
4) Compile as following:

./configure --without-server
make
make install

5) Compile Cactid

./configure
make
make install

Please note, the full paths thing is no longer a requirement. It was a mistake.

TheWitness

[mcutting]

Larry - Thanks for the info. Clearly, I need Cygwin to compile a new version of Cactid, but have some questions:

1. Do I need to compile cactid on the machine that will be running it ?
2. If the answer to the above is yes, do I need to uninstall the version of net-snmp I am already running ?
3. Is there (in SVN) a stable cactid.exe for Windows already compiled for use ?

I'd really like to have this working, but do not really want to install Cygwin on my live box if I don't have to. I tried to use the pre-compiled cactid from BSOD2600, but this exhibited the same behaviour:

sh.exe hangs at 99 percent cpu
The POPEN timed out error messages

SNMP polling with cactid works great (super fast) - it just seems to stumble on scripts. At this point, I have to use pskill to terminate the sh.exe process, and the zombie cactid.exe processes.

If I can compile it on another machine, this would make me happy. What other files do I need to copy over to the live box along with

cactid.exe
crypto dll
cygwin1.dll
sh.exe
cactid.conf

From other posts, I understand that a handful of files are needed.

Any help with this would be much appreciated.

Thanks

[TheWitness]

No.
No.
Yes.

Here is a compiled version against the latest SVN. But you will need a good cygwin install with cryptography installed as well to run it.

TheWitness

If you want to go cheap. The following Cygwin DLL's are required:
cygcrypto-0.9.8.dll
cygz.dll
cygwin1.dll
sh.exe
cygreadline6.dll
cygintl-8.dll

[mcutting]

Hi Larry,

Thanks for the info + files. I installed Cygwin on our productive box, and have tested the CACTID poller. It worked well for a couple of runs, then got stuck on sh.exe at 99% again - I also had to kill off this process, and the tons of zombie cactid.exe processes.

I have managed to get down to 1 copy of sh.exe (in the Cygwin\Bin directory). All of my externmal scripts (mostly PERL) are running.

If I can get this hanging sh.exe sorted, then I will be done !

Thanks

[mcutting]

UPDATE

Ok. I think I've cracked it. I ran CACTID in DEBUG with verbosity 5 on the hosts that were causing the issues. I found a PERL script that collects WMI data from Exchange. This runs like a rocket locally, but takes a while over a slow link to a remote site, and causes CACTID to fail.

I've removed this graph, and CACTID now seems to be stable. I also only have one instance of SH.exe, which is in the Cygwin\Bin directory. I know that there is only supposed to be one copy of Cygwin1.dll, but where does this live ? Cygwin\Bin, or the CACTID directory ?

Thanks

[BSOD2600]

I know that there is only supposed to be one copy of Cygwin1.dll, but where does this live ? Cygwin\Bin, or the CACTID directory ?

Since you've got cygwin installed, it belongs in cygwin\bin\.

[mcutting]

Thanks. Looks like I've still got one host that runs a VBScript (written by me), which gives a POPEN timed out issue. I don't think it's the script, as it is very basic, and works fine with CACTID 99% of the time.

[mcutting]

Just for the sheer fun of it, I decided to downgrade CACTID to version "g". Surprisingly, this runs like a rocket, and works first time every time !! It seems strange, but this also does not seem to require or call sh.exe ???

Another thing that is odd is that I don't have a full copy of Cygwin installed - just the basic (and I mean BASIC) dll files that are required for it to run.

Even my PERL and VBSCRIPTS are working, and the data is collected extremely quickly.

Thanks for everyone's help !

[mcutting]

Ok. I really wanted the latest and greatest version of CACTID, so I took the plunge, and compiled it myself, using the instructions from TheWitness in this post.

The end result is: so far so good...it's using sh.exe for scripts such as PERL and CSCRIPT etc. I have two processes, with 15 threads. A total of 47 hosts are completing in around 20-35 seconds (average), which is what I wanted and expected from CACTID.

I have not installed CYGWIN - I have simply created a tree structure on the C: drive for CYGWIN\BIN, and dumped these files below inside the BIN folder (note, that I had to create a directory under CYGWIN called "tmp", as CACTID complained it could not see this)

cygcrypto-0.9.8.dll
cygfreetype-6.dll
cygiconv-2.dll
cygintl-3.dll
cygintl-8.dll
cygncurses-8.dll
cygpng12.dll
cygreadline6.dll
cygwin1.dll
cygz.dll
sh.exe

Inside the CACTID directory are two files

cactid.conf
cactid.exe

Environment Variables are set as

C:\CYGWIN\BIN; has been appended to the SYSTEM PATH variable

MIBDIRS=C:\USR\SHARE\SNMP\MIBS (this directory is actually empty at the suggestion of TheWitness)

MIBS=ALL This has no real impact, as there are no MIBS to load anyway, and it stops the error messages from appearing in the poller when running from CLI

I installed CYGWIN on a spare machine, downloaded the NET-SNMP and MYSQL source, and compiled CACTID here. The required files were then transferred to the production CACTI box after testing on a development machine.

The poller has been running as a scheduled task for three hours now with no issues at all. I will leave this running for the weekend, and will keep a close eye on it. My only last question is:

The end result is that I am now using CACTID 0.8.6i (SVN build) as the poller.

Open Question : Why does the poller complete quicker when run from CLI as opposed to a Scheduled Task in Windows ???

[BSOD2600]

What type of time differences are we talking about? Is the scheduled task user the same user as when you manually run it?

[mcutting]

Thanks for the reply. We are talking about 10 seconds at the worst, and about 5-6 at the best. The poller when run from CLI is being run under the same account as the scheduled task.

[TheWitness]

You know that I've seen this before. And it had something to do with domain authentication. I never solved it, I was just more pissed off at Microsoft than ever before.

The way to isolate this at the protocol level is to use Wireshark and listen on the wire during a polling cycle. Then, save and analyze it. If you want, you can PM it to me.

Larry

[mcutting]

An interesting deviation: Over the weekend, the newly compiled version of cactid decided to hang on sh.exe - it actually did this 8 times.

On the 9th occasion, the shell completely froze on the box. I had to use pskill to terminate the running processes. Cactid run ok for a couple of hours after that. When I checked the box again this morning, the sh.exe process was again stuck at 99 percent, with about 18 cactid.exe processes trailing after it.

I will be rolling back to version G of the poller, as my new version is simply too unstable (which is a pity). Version G of cactid doesn't seem to need sh.exe, amd this seems to be the root of the problem.