terminated with signal '15' in file:'ss_host_disk.php

[mikecarel]

Hi to all;

I already upgraded from Cacti 0.8.7e to Cacti 0.8.7g. My upgrade was ok except for 1 error that doesn't exist from my previous Cacti version:

PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk'.

I used to delete temporary the graph that is using this script to avoid this error.

Any reasons or other things i need to do to avoid this error? I'm running Redhat5.5, php ver5.1.6, spine ver 0.8.7g

Thanks for any feedback and help.

[julian924s]

Hi - getting the same warnings:

08/16/2010 03:50:15 PM - PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk', params:'10.8.0.10 320 2:161:8000:1:10:xxxxxxx:cacti:xxxxxx:MD5::DES: get used 6'
08/16/2010 03:50:15 PM - PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk', params:'10.8.0.10 320 2:161:8000:1:10:xxxxxxxx:cacti:xxxx:MD5::DES: get total 7'
08/16/2010 03:50:15 PM - PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk', params:'10.8.0.10 320 2:161:8000:1:10:xxxxxx:cacti:xxxxx:MD5::DES: get used 7'


Gone from 0.8.7d to 0.8.7g, never seem these before. Got the latest version of ss_host_disk.php out of svn but I was already using this version.

Cacti Version - 0.8.7g
Plugin Architecture - 2.8
Poller Type - Cactid v
Server Info - Linux 2.6.18-194.8.1.el5
Web Server - Apache/2.2.3 (CentOS)
PHP - 5.1.6
PHP Extensions - libxml, xml, wddx, tokenizer, sysvshm, sysvsem, sysvmsg, standard, SimpleXML, sockets, SPL, shmop, session, Reflection, pspell, posix, mime_magic, iconv, hash, gmp, gettext, ftp, exif, date, curl, ctype, calendar, bz2, zlib, pcre, openssl, apache2handler, dbase, gd, memcache, mssql, mysql, mysqli, PDO, pdo_mysql, pdo_sqlite, snmp
MySQL - 5.0.77-log
RRDTool - 1.4.4
SNMP - 5.3.2.2
Plugins

• Device Monitoring (monitor - v0.8.2)
  Syslog Monitoring (syslog - v1.04)
  Host Info (hostinfo - v0.2)
  status

Thanks.

[TheWitness]

This is the result of your poller either timing out or your script_server timing out. The script server has been enhanced in the 0.8.7g release to provide meaning messages when it's killed.

In your case a Signal 15 tells me that you are likely running Linux and that someone sent a 'sigterm' to your script_server process. This could be the result of some automated script run by you, by someone issuing a kill from the command line, or the parent process (cmd.php/spine) being killed/terminated, or lastly a script server timeout (which would indicate you are using spine). The latter is what I suspect.

TheWitness

[julian924s]

Ah right ok, I am indeed using spine and the script & server timeout value was 12secs - I don't remember changing this but could have done many months ago. I've changed it to 20secs to see how it goes.


Thanks.

[jackie]

I just upgraded from cacti 0.8.7b to 0.8.7g and I'm also seeing these Script Server terminated message. We have just setup a new Linux box on which 0.8.7g is installed.

I checked my Script Server timeout value, and it is set to 35 seconds. From the errors I'm seeing (below), it looks like the process is getting killed 11 seconds into the poll (07:27:11). Is the 35 seconds timeout not being reached for some reason? The max execution time in /etc/php.ini is set to 30 seconds. I don't believe anything else on the system would be killing the script server.

Code: Select all

10/18/2010 07:27:11 AM - PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk', params:'gunter2.acns.colostate.edu 47 2:161:10000:2:5:public:admin:2admincacti:MD5::DES: get total 20'
10/18/2010 07:27:11 AM - PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk', params:'gunter2.acns.colostate.edu 47 2:161:10000:2:5:public:admin:2admincacti:MD5::DES: get used 20'

We are setup with:
Concurrent Poller Processes: 2
Max Theads/Process: 4
No. of PHP Script Servers: 6
Script & Script Server Timeout Value: 35
Spine 0.8.7g - 6121

Thanks for your help with this mystery.
Jackie

[TheWitness]

Are your pollers over-running one another? That would explain it, or is there some 'cleanup' process running. Spine does forcibly kill a thread that does not end properly for sure. How often is this happening? Is polling affected?

TheWitness

[jackie]

Thanks for your response.

I don't believe my pollers are overrunning each other. Spine completes in 9-16 seconds according to the log file. So, it seems like I have lots of time per polling cycle.

This is a "brand new" Linux box, so I don't think there is any cleanup process running, unless Linux does something strange.

I'm seeing an error every 12-30 minutes or so. It's not a repeatable pattern though. And it is affecting several different devices.

My graphs do look different on this new server vs. the old server. I can see one graph that is missing data, but it is an snmp query, not a server script graph. So, I do think these timeouts are affecting the polling.

I turned on verbose logging in the cacti log file, but I don't see any additional messages, just the Server Script terminated messages. Is there a way to get more info from the Server Scripts when they execute?

Thanks again,
Jackie

[jackie]

Here's a sample of the our cacti log with the warning messages:

Code: Select all

10/21/2010 08:11:43 AM - EXPORT STATS: ExportDate:2010-10-21_8:11:43 ExportDuration:95.1111 TotalGraphsExported:784
10/21/2010 08:11:11 AM - PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk', params:'xxxx.acns.colostate.edu 46 2:161:10000:2:5:xxxxxx:xxxxxx:xxxxxx:MD5::DES: get used 20'
10/21/2010 08:11:11 AM - SYSTEM STATS: Time:9.4446 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:89 RRDsProcessed:83
10/21/2010 08:10:07 AM - SYSTEM STATS: Time:6.3504 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:92 RRDsProcessed:84
10/21/2010 08:09:17 AM - SYSTEM STATS: Time:15.3193 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:90 RRDsProcessed:82
10/21/2010 08:08:15 AM - SYSTEM STATS: Time:13.3932 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:89 RRDsProcessed:82
10/21/2010 08:05:14 AM - SYSTEM STATS: Time:11.9106 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:87 RRDsProcessed:82
10/21/2010 08:03:54 AM - EXPORT STATS: ExportDate:2010-10-21_8:03:54 ExportDuration:99.4059 TotalGraphsExported:784
10/21/2010 08:03:22 AM - SYSTEM STATS: Time:20.5696 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:89 RRDsProcessed:83
10/21/2010 08:02:14 AM - SYSTEM STATS: Time:12.3541 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:92 RRDsProcessed:84
10/21/2010 08:01:10 AM - SYSTEM STATS: Time:8.6901 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:90 RRDsProcessed:82
10/21/2010 08:00:16 AM - SYSTEM STATS: Time:15.3234 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:89 RRDsProcessed:82
10/21/2010 07:59:12 AM - SYSTEM STATS: Time:10.3044 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:87 RRDsProcessed:82
10/21/2010 07:58:55 AM - EXPORT STATS: ExportDate:2010-10-21_7:58:55 ExportDuration:95.5566 TotalGraphsExported:784
10/21/2010 07:58:15 AM - PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'', function:'', params:''
10/21/2010 07:58:15 AM - SYSTEM STATS: Time:13.3848 Method:spine Processes:2 Threads:4 Hosts:36 HostsPerProcess:18 DataSources:89 RRDsProcessed:83 

The time between errors seems to vary widely. I've seen over an hour between errors, and as little as 5 minutes. I'd sure like to figure this out. I've heard from another user that adjusting the Script Server timeout didn't help them either. Please let me know what I can do to debug this.

Thanks,
Jackie

[jackie]

I'm still seeing these errors. One interesting thing I noticed, is that if I reduce the number of threads, thus increasing the polling time used by spine, I see fewer warnings. Looks like the script is getting killed when spine finishes and it is not observing the Server Script timeout value set in the cacti Settings panel.

Would love to hear anyone thoughts on this.....thanks. Jackie

[rmcdonald]

I am seeing these errors as well. I think Spine or Poller is over running the time or each other. Anyone find a solution for this?

[TheWitness]

PHPSVR: Poller[0] WARNING: Script Server terminated with signal '15' in file:'ss_host_disk.php', function:'ss_host_disk', params:'xxxx.acns.colostate.edu 46 2:1612:5:xxxxxx:xxxxxx:xxxxxx:MD5::DES: get used 20'

This error means that the Script/Script Server timeout is not high enough to allow the Host Disk polling to complete. So, you need to increase timeouts. Bottom line. There is a plugin option as well for these Host Resource MIB issues. It's pretty unpublished, but will be released real soon.

TheWitness

[rmcdonald]

What is the plugin?

[TheWitness]

hmib. However, the documentation is light (README). I think that Trunk is ready to release. Any changes beyond this date will not break the existing functionality. Let me know how it goes:

svn://svn.cacti.net/cacti_plugins/hmib/trunk

TheWitness

[rmcdonald]

One other question is I think I have a device that is causing polling to run for more than 60 seconds how do I go about finding that device?

[TheWitness]

That's likely the one. Using spine 0.8.7h at -V 3 will show total polling times per host. However, hmib will likely solve this issue.