Problem: Cactid and multiple threads

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Developers, Moderators

Post Reply
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Problem: Cactid and multiple threads

Post by drose12 »

---
running cacti 0.8.6g, cactid 0.8.6f-1 and rrdtool-1.2.11.
not running SELINUX
oh yea, running the 0.8.6g pre-patches too
---

Hi all...just upgraded my server to Red Hat Enterprise Linux AS release 4, and now all my graphs that are dependent on scripts are broken. I did some playing around, and found that if I set the poller config to only spawn 1 concurrent poller and 1 thread for cactid, everything is fine, except that it takes >200seconds to complete my run. As soon as I up it to 2 threads or more, I start getting the following:

------------------
[root@nasux bin]# ./cactid
CACTID: Using cactid config file [cactid.conf]
CACTID: Version 0.8.6f starting
CACTID: Host[1] ERROR: The function was interrupted before any of the selected events occurred and before the timeout interval expired.
CACTID: Host[1] ERROR: The POPEN timed out
CACTID: Host[1] DS[7] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: ERROR: A non blocked signal was caught.
CACTID: Host[8] DS[78] WARNING: Result from SERVER not valid. Partial Result: ...
CACTID: Host[8] DS[85] WARNING: Could not call script server
CACTID: Host[8] DS[87] WARNING: Could not call script server
CACTID: Host[8] DS[88] WARNING: Could not call script server
CACTID: Host[8] DS[2143] WARNING: Could not call script server
CACTID: Host[8] DS[85] WARNING: Could not call script server
CACTID: Host[8] DS[87] WARNING: Could not call script server
CACTID: Host[8] DS[88] WARNING: Could not call script server
CACTID: Host[8] DS[2143] WARNING: Could not call script server
CACTID: Host[8] ERROR: The function was interrupted before any of the selected events occurred and before the timeout interval expired.
CACTID: Host[8] ERROR: The POPEN timed out
CACTID: Host[15] DS[7375] WARNING: Could not call script server
CACTID: Host[15] DS[179] WARNING: Could not call script server
CACTID: Host[15] DS[182] WARNING: Could not call script server
CACTID: Host[15] DS[183] WARNING: Could not call script server
CACTID: Host[15] DS[184] WARNING: Could not call script server
CACTID: Host[15] DS[180] WARNING: Could not call script server
CACTID: Host[15] DS[179] WARNING: Could not call script server
CACTID: Host[15] DS[182] WARNING: Could not call script server
CACTID: Host[15] DS[183] WARNING: Could not call script server
CACTID: Host[15] DS[184] WARNING: Could not call script server
CACTID: Host[15] DS[180] WARNING: Could not call script server
CACTID: Host[16] DS[190] WARNING: Could not call script server
CACTID: Host[16] DS[185] WARNING: Could not call script server
CACTID: Host[16] DS[186] WARNING: Could not call script server
CACTID: Host[16] DS[187] WARNING: Could not call script server
CACTID: Host[16] DS[188] WARNING: Could not call script server
CACTID: Host[16] DS[189] WARNING: Could not call script server
CACTID: Host[16] DS[185] WARNING: Could not call script server
CACTID: Host[16] DS[186] WARNING: Could not call script server
CACTID: Host[16] DS[187] WARNING: Could not call script server
CACTID: Host[16] DS[188] WARNING: Could not call script server
CACTID: Host[16] DS[189] WARNING: Could not call script server
CACTID: Host[17] DS[200] WARNING: Could not call script server
CACTID: Host[17] DS[7373] WARNING: Could not call script server
CACTID: Host[17] DS[196] WARNING: Could not call script server
CACTID: Host[17] DS[197] WARNING: Could not call script server
CACTID: Host[17] DS[198] WARNING: Could not call script server
CACTID: Host[17] DS[7374] WARNING: Could not call script server
CACTID: Host[17] DS[7373] WARNING: Could not call script server
CACTID: Host[17] DS[196] WARNING: Could not call script server
CACTID: Host[17] DS[197] WARNING: Could not call script server
CACTID: Host[17] DS[198] WARNING: Could not call script server
CACTID: Host[17] DS[7374] WARNING: Could not call script server

<CTL-C>
-----------------

Now it appears that host 1 is having a problem, but if I run it by itself it is ok:

[root@nasux bin]# ./cactid 1 1
CACTID: Using cactid config file [cactid.conf]
CACTID: Version 0.8.6f starting
CACTID: Execution Time: 0.4166 s, Threads: 1, Hosts: 2


Any ideas?
Last edited by drose12 on Wed Oct 26, 2005 1:57 pm, edited 1 time in total.
Top
User avatar
rony
Developer/Forum Admin
Posts: 6022
Joined: Mon Nov 17, 2003 6:35 pm
Location: Michigan, USA
Contact:
Contact rony

Post by rony »

What is you host down detection method are you using?
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]
Top
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Post by drose12 »

Ping and SNMP - Most Recommended.
Top
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Post by drose12 »

bump
Top
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Post by drose12 »

Rony,

When I set the Downed Host Detection to SNMP - Reliable, I don't get the errors if I bump up the threads....is there a problem with ping detection? I've always had it set that way. I did upgrade my OS to AS4 though...
Top
User avatar
TheWitness
Developer
Posts: 17047
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Please try the nft_open.c in SVN and let me know if it changes anything.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
User avatar
rony
Developer/Forum Admin
Posts: 6022
Joined: Mon Nov 17, 2003 6:35 pm
Location: Michigan, USA
Contact:
Contact rony

Post by rony »

Also, how many threads are you using?

And how many processes do you have set?
[size=117][i][b]Tony Roman[/b][/i][/size]
[size=84][i]Experience is what causes a person to make new mistakes instead of old ones.[/i][/size]
[size=84][i]There are only 3 way to complete a project: Good, Fast or Cheap, pick two.[/i][/size]
[size=84][i]With age comes wisdom, what you choose to do with it determines whether or not you are wise.[/i][/size]
Top
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Post by drose12 »

2 poller processes and 15 threads.
running on a dual xeon 3ghz HT box with 2gb mem

11/01/2005 08:40:36 AM - SYSTEM STATS: Time:34.8318 Method:cactid Processes:2 Threads:15 Hosts:181 HostsPerProcess:91 DataSources:11702 RRDsProcessed:6104
Top
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Post by drose12 »

TheWitness wrote:Please try the nft_open.c in SVN and let me know if it changes anything.

TheWitness
Sorry for being ignorant, but I'm not sure about that....where are the SVN archives?
Top
User avatar
TheWitness
Developer
Posts: 17047
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

I think you need to look at your glibc version installed and maybe backoff a release or two. Also, what version of gcc did you use to make the project.

It appears like a threading issue with your OS.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Post by drose12 »

Interestingly enough I seem to have two glibc rpms installed:

glibc-2.3.2-95.37
glibc-2.3.4-2.13

I'm going to get rid of the 2.3.2-95 version and re-compile cactid. I see now that the nft_popen.c you were referring to was part of the cactid source....

Will let you know.
Top
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Post by drose12 »

Hmm, latest SVN doesn't quite compile:

gcc -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -L/usr/local/ssl/lib -L/usr/lib -L/usr/lib/mysql -o cactid sql.o cactid.o util.o snmp.o locks.o poller.o nft_popen.o php.o -lnetsnmp -lcrypto -lmysqlclient -lmysqlclient -lz -lpthread -lm -lnsl
poller.o(.text+0x13e8): In function `poll_host':
/tmp/c/cacti/trunk/cactid/poller.c:174: undefined reference to `ping_host'
poller.o(.text+0x1418):/tmp/c/cacti/trunk/cactid/poller.c:178: undefined reference to `update_host_status'
poller.o(.text+0x1808):/tmp/c/cacti/trunk/cactid/poller.c:167: undefined reference to `update_host_status'
poller.o(.text+0x186a):/tmp/c/cacti/trunk/cactid/poller.c:178: undefined reference to `update_host_status'
collect2: ld returned 1 exit status
make: *** [cactid] Error 1


Looks like when it's linking, it doesn't reference the ping.o which has the ping_host function in it. The ping.c also doesn't seem to be getting compiled, and when trying to compile it, I see that:

ping.c: In function `ping_host':
ping.c:70: error: too few arguments to function `cacti_log'
cacti_log("DEBUG: Falling back to UDP Ping due to inability to set asroot permissions\n");

I changed it to:

cacti_log("DEBUG: Falling back to UDP Ping due to inability to set asroot permissions\n",SEV_DEBUG,
0);

and added ping.o to the am_cactid_OBJECTS line in the Makefile, since for whatever reason it isn't being added...and was able to compile. Sorry for being a hack.

I tried the 0.9-dev version and it doesn't seem to work....

more of the:

11/02/2005 09:15:59 AM - CACTID: Poller[0] Host[8] DS[2143] WARNING: Could not call script server
11/02/2005 09:15:59 AM - CACTID: Poller[0] Host[8] ERROR: The function was interrupted before any of the selected events occurred and before the timeout interval expired.
11/02/2005 09:15:59 AM - CACTID: Poller[0] Host[8] ERROR: The POPEN timed out
11/02/2005 09:54:59 AM - POLLER: Poller[0] Maximum runtime of 296 seconds exceeded. Exiting.
Top
User avatar
TheWitness
Developer
Posts: 17047
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

You need to grab the SVN from BRANCH_0_8_6. The "Trunk" version is "0.9" which will NOT work with 086. It is very broken at the moment. Ian refers to 0.9 as the "Dark Branch" knowing my sons love of Star Wars. It's actually going to be a great upgrade.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
drose12
Posts: 31
Joined: Thu Mar 10, 2005 11:02 am

Post by drose12 »

ok, tried 0.8.6g from SVN, its still not happy for me:

CACTID: Using cactid config file [cactid.conf]
CACTID: Version 0.8.6g starting
CACTID: Host[1] ERROR: Empty result [127.0.0.1]: 'perl /var/www/html/cacti/scripts/linux_memory.pl MemFree:'
CACTID: Host[1] DS[3] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[1] ERROR: Empty result [127.0.0.1]: 'perl /var/www/html/cacti/scripts/linux_memory.pl SwapFree:'
CACTID: Host[1] DS[4] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[1] ERROR: Empty result [127.0.0.1]: 'perl /var/www/html/cacti/scripts/loadavg_multi.pl'
CACTID: Host[1] DS[5] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[1] ERROR: Empty result [127.0.0.1]: 'perl /var/www/html/cacti/scripts/mem.pl'
CACTID: Host[1] DS[365] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[1] ERROR: Empty result [127.0.0.1]: 'perl /var/www/html/cacti/scripts/unix_processes.pl'
CACTID: Host[1] DS[7] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[1] ERROR: Empty result [127.0.0.1]: 'perl /var/www/html/cacti/scripts/unix_users.pl'
CACTID: Host[1] DS[6] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[25] ERROR: Empty result [hpov1.ceridian.ca]: 'perl /var/www/html/cacti/scripts/loadavg_multi.pl'
CACTID: Host[25] DS[339] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[25] ERROR: Empty result [hpov1.ceridian.ca]: 'perl /var/www/html/cacti/scripts/unix_processes.pl'
CACTID: Host[25] DS[342] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[25] ERROR: Empty result [hpov1.ceridian.ca]: 'perl /var/www/html/cacti/scripts/unix_users.pl'
CACTID: Host[25] DS[340] WARNING: Result from SCRIPT not valid. Partial Result: ...
CACTID: Host[8] ERROR: Empty result [calsvr2k.ceridian.ca]: 'perl /var/www/html/cacti/scripts/ping-latency.pl 170.153.42.2'
CACTID: Host[8] DS[6876] WARNING: Result from SCRIPT not valid. Partial Result: ...


CTL-C

Staying with 0.8.6f with snmp only for host detection
Top
Post Reply

Return to “Help: Linux/Unix Specific”

Who is online

Users browsing this forum: No registered users and 0 guests