Problems after upgrading to 08.6f

andrew2 · Post by **andrew2** » Tue Jul 05, 2005 2:15 pm

Earlier today I upgraded from 0.8.6c to 0.8.6f to catch the recent security fixes. The upgrade seemed to go without a hitch (i.e. everything works

) but I notice now after a few hours that the load average on the system has skyrocketed. (Over 200 before I noticed, now down to 25 which is still ridiculously high.)

I'm also using cactid 0.8.6d. Is there any issue with Cacti 0.8.6f and Cactid 0.8.6d? Everything *seems* to work ok -- graphs are being updated without problem, but when I check out the process list, I see a bunch of processes like:

8995 ? R 8:40 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
8996 ? R 8:43 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
8997 ? R 8:36 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
8998 ? R 8:45 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
8999 ? R 8:31 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
16900 ? R 5:14 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
16901 ? R 5:09 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
16910 ? R 5:12 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
16914 ? R 5:09 /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0

and so on. The number of php processes increases by 5 every 5 minutes, which makes sense since I've got Cacti configured to run 5 poller processes. It just seems like they never end properly. When running poller.php manually from the commandline, it runs normally and returns normal results, taking about 30 seconds to complete a polling run.

Do I need to upgrade Cactid, or is there something else going on here?

Thanks,

Andrew

andrew2 · Post by **andrew2** » Tue Jul 05, 2005 2:33 pm

Hate to reply to my own post, but if anyone else runs across the same problem, I've added this to my crontab to help keep things under control until I find the actual problem:

*/5 * * * * kill -9 `ps ax | grep php | grep ? | cut -c0-5` > /dev/null 2>&1

It will terminate any defunct php processes every 5 minutes.

Andrew

Post by **TheWitness** » Tue Jul 05, 2005 3:33 pm

andrew,

could you please run "php <path_cacti>/cmd.php 0 x" from the command line in debug mode and post the STDOUT results "x" is some host_id > 0. Also, run the following from the command line:

"php <path_cacti>/script_server.php"
> "quit"

If you get any warnings/errors, please post them. You can also run cactid the same way. What version of Cactid are you running?

TheWitness

andrew2 · Post by **andrew2** » Wed Jul 06, 2005 8:49 am

Well this is interesting. Here's the output from both commands:

[root@monitor2 root]# php /root/cacti-0.8.5/cmd.php 0 1
#!/usr/bin/php -q

[root@monitor2 root]# php /root/cacti-0.8.5/script_server.php
PHP Script Server has Started - Parent is cmd

Both hang indefinitely after that.

I'm running cactid 0.8.6d. Running that from the commandline I get:

[root@monitor2 root]# /usr/bin/php /root/cacti-0.8.5/script_server.php cactid 0
PHP Script Server has Started - Parent is cactid

And it hangs there as well. Which seems very odd to me, because if I run poller.php from the commandline, cactid runs fine. (It kicks a few errors here and there, but those are just misconfigured check commands and the like which shouldn't be a problem.) And, like I mentioned before, everything appears to work ok -- graphs continue to be updated as normal, it's just the extra cactid processes that keep hanging around. Should I upgrade to a newer version of cactid?

Thanks,

Andrew

Post by **TheWitness** » Wed Jul 06, 2005 4:41 pm

Andrew,

When you manually start the script server, you should be able to type the command "quit" and then a carriage return to exit it. Does that not happen? If not, please run "php -v" and "php -m" and post. Also, you may want to e-mail me the results of the php_info() command.

TheWitness

andrew2 · Post by **andrew2** » Thu Jul 07, 2005 8:49 am

Yes, that seems to work fine:

[root@monitor2 root]# php /root/cacti-0.8.5/script_server.php
PHP Script Server has Started - Parent is cmd
quit
PHP Script Server Shutdown request received, exiting

I'm using PHP 4.3.8 with Apache 1.3.33, both pretty vanilla installations.

Also, one more bit of information -- a stack trace on the defunct script_server processes shows no system calls at all. They appear to be doing other than taking up system resources.

Thanks,

Andrew

Post by **TheWitness** » Thu Jul 07, 2005 9:39 pm

This is likely an issue with hosed up Data Sources. Please goto Data Sources and select host filter = None and delete your bogus Data Sources. What is likely happening is that the parent process cmd.php is crashing and not quitting the script server process.

TheWitness

p.s. We have a patch in the wings for this issue. Just been on Vacation for the last week.

andrew2 · Post by **andrew2** » Fri Jul 08, 2005 9:16 am

There don't appear to be any datasources not associated with a host. Just to see where the problem might be, I tried switching back to cmd.php instead of cactid. Interestingly, that didn't work at all -- here's the log snippet from the switchover:

07/08/2005 10:20:09 AM - CACTID: Poller[0] Time: 9.1122 s, Threads: 5, Hosts: 13
07/08/2005 10:20:10 AM - SYSTEM STATS: Time: 9.9091 s, Method: cactid, Processes: 5, Threads: 5, Hosts: 65, Hosts/Process: 13, Data Sources 507, RRDs Processed 272
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] ERROR: Either there are no items in the cache or polling is disabled
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Host[1] No host availability check possible for '127.0.0.1'.
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Host[1] DS[3] CMD: perl /root/cacti-0.8.5/scripts/linux_memory.pl MemFree:, output: 7834407/08/2005 10:25:01 AM - CMDPHP: Poller[0] Host[1] DS[4] CMD: perl /root/cacti-0.8.5/scripts/linux_memory.pl SwapFree:, output: 1993876
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Time: 0.0750 s, Theads: N/A, Hosts: 13
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Host[1] DS[5] CMD: perl /root/cacti-0.8.5/scripts/loadavg_multi.pl, output: 1min:6.74 5min:5.11 10min:4.69
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Host[1] DS[6] CMD: perl /root/cacti-0.8.5/scripts/unix_users.pl, output: 1
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Time: 0.1987 s, Theads: N/A, Hosts: 13
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Host[1] DS[7] CMD: perl /root/cacti-0.8.5/scripts/unix_processes.pl, output: 130
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Time: 0.4715 s, Theads: N/A, Hosts: 12
07/08/2005 10:25:01 AM - CMDPHP: Poller[0] Time: 0.3782 s, Theads: N/A, Hosts: 13
07/08/2005 10:25:01 AM - SYSTEM STATS: Time: 1.1268 s, Method: cmd.php, Processes: 5, Threads: N/A, Hosts: 65, Hosts/Process: 13, Data Sources 507, RRDs Processed 5

Of particular interest that I see is that cmd.php only processed 5 RRDs out of 272 it should have processed. I'm wondering if that has anything to do with "ERROR: Either there are no items in the cache or polling is disabled" Polling is definitely enabled (at least in the GUI...I'm not sure where to look in the database to be sure it's set correctly) and I'm not sure what the cache is. Does any of this help?

Thanks,

Andrew

Adze15021979 · Post by **Adze15021979** » Fri Jul 08, 2005 1:22 pm

Please try the following:

* Disable poller in cacti settings screen.
* Clear the poller_output and poller_item tables in mysql database (mysql -u cactiuser -p cactidb, delete from poller_output;, delete form poller_item;).
* Regenerate cache in cacti system utilities screen. (this can take a while...)
* Enable the poller and wait...

andrew2 · Post by **andrew2** » Fri Jul 08, 2005 2:46 pm

That doesn't seem to have made a difference. Using cmd.php as the poller still returns the same results as above, and using cactid as the poller works, but still leaves behind all the defunct processes chewing up system resources.

Thanks,

Andrew

Post by **TheWitness** » Sat Jul 09, 2005 8:07 am

Andrew,

Please dump your database and forward to me. Thanks,

TheWitness

Post by **TheWitness** » Tue Jul 12, 2005 9:41 pm

Andrew,

Please set your "Availability" type to "SNMP Reliable". You can find that option under "Settings->Poller". Let me know if that makes things better.

TheWitness

muh · Post by **muh** » Wed Jul 13, 2005 7:48 am

I'm seeing something very similar to this. Just upgrades from 0.8.6c to 0.8.6f and no data is being collected. Output from php poller.php is -

07/13/2005 01:38:37 PM - SYSTEM STATS: Time: 1.0968 s, Method: cmd.php, Processes: 1, Threads: N/A, Hosts: 9, Hosts/Process: 9, Data Sources 476, RRDs Processed 0

Had a bit of a prod around, everything is enabled/active, theres just no data coming in.

Any help would be greatly appreciated.

Post by **TheWitness** » Wed Jul 13, 2005 9:03 am

muh,

Although you hijacked Andrew's thread, run 1 pass in DEBUG and post your output to a new topic stating your original question and I will assist. The output is quite unusual.

TheWitness

andrew2 · Post by **andrew2** » Wed Jul 13, 2005 9:06 am

TheWitness wrote:Andrew,

Please set your "Availability" type to "SNMP Reliable". You can find that option under "Settings->Poller". Let me know if that makes things better.

TheWitness

Unfortunately that hasn't seemed to make any difference. Could this have anything to do with the version of SNMP utilities I'm using? I'm still on an pretty old version: ucd-snmp-4.2.5-8.73.1

I'm loathe to upgrade that since other programs on the server depend on snmpwalk and snmpget as well and if the syntax has changed any moving to net-snmp, they will have to be tweaked accordingly.

Andrew

Problems after upgrading to 08.6f

Problems after upgrading to 08.6f

Who is online