Help debug/troubleshoot spine timeout issues

Rael · Post by **Rael** » Sun Feb 24, 2008 10:26 am

Here's the entry in my cacti logs:

02/24/2008 03:14:55 PM - SYSTEM STATS: Time:294.4949 Method:spine Processes:4 Threads:40 Hosts:458 HostsPerProcess:115 DataSources:8469 RRDsProcessed:7904
02/24/2008 03:14:55 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal

I'd like to troubleshoot why this process takes so long so I can at least explain/correct the issue or justify buying more powerful hardware perhaps.

Box config is: 2x4 running SLES9SP3 @ Kernel: 2.6.5-7.286-bigsmp

CACTI CONFIG:

Code: Select all

Technical Support 
 
General Information 
Date Sun, 24 Feb 2008 15:19:57 +0000 
Cacti Version 0.8.7b 
Cacti OS unix 
SNMP Version net-snmp 
RRDTool Version RRDTool 1.2.x  
Hosts 457 
Graphs 3535 
Data Sources Script/Command: 814
SNMP: 6806
SNMP Query: 420
Script Query - Script Server: 5
Total: 8045 
Poller Information 
Interval 300 
Type spine 
Items Action[0]: 7645
Action[1]: 814
Action[2]: 10
Total: 8469 
Concurrent Processes 4 
Max Threads 40 
PHP Servers 2 
Script Timeout 295 
Max OID 30 
Last Run Statistics Time:294.4949 Method:spine Processes:4 Threads:40 Hosts:458 HostsPerProcess:115 DataSources:8469 RRDsProcessed:7904 
PHP Information 
PHP Version 4.4.4 
PHP OS Linux 
PHP uname Linux eccws587 2.6.5-7.286-bigsmp #1 SMP Thu May 31 10:12:58 UTC 2007 i686 
PHP SNMP Not Installed 
max_execution_time 600 
memory_limit 
It is highly suggested that you alter you php.ini memory_limit to 248M or higher. This suggested memory value is calculated based on the number of data source present and is only to be used as a suggestion, actual values may vary system to system based on requirements.
 
MySQL Table Information 
Name Rows Engine Collation Check Status 
cdef 16 MyISAM latin1_swedish_ci OK 
cdef_items 48 MyISAM latin1_swedish_ci OK 
colors 104 MyISAM latin1_swedish_ci OK 
data_input 19 MyISAM latin1_swedish_ci OK 
data_input_data 83942 MyISAM latin1_swedish_ci OK 
data_input_fields 93 MyISAM latin1_swedish_ci OK 
data_local 8045 MyISAM latin1_swedish_ci OK 
data_template 76 MyISAM latin1_swedish_ci OK 
data_template_data 8121 MyISAM latin1_swedish_ci OK 
data_template_data_rra 32681 MyISAM latin1_swedish_ci OK 
data_template_rrd 12070 MyISAM latin1_swedish_ci OK 
graph_local 3535 MyISAM latin1_swedish_ci OK 
graph_template_input 127 MyISAM latin1_swedish_ci OK 
graph_template_input_defs 526 MyISAM latin1_swedish_ci OK 
graph_templates 72 MyISAM latin1_swedish_ci OK 
graph_templates_gprint 3 MyISAM latin1_swedish_ci OK 
graph_templates_graph 3607 MyISAM latin1_swedish_ci OK 
graph_templates_item 44765 MyISAM latin1_swedish_ci OK 
graph_tree 3 MyISAM latin1_swedish_ci OK 
graph_tree_items 560 MyISAM latin1_swedish_ci OK 
host 457 MyISAM latin1_swedish_ci OK 
host_graph 3367 MyISAM latin1_swedish_ci OK 
host_snmp_cache 89711 MyISAM latin1_swedish_ci OK 
host_snmp_query 1324 MyISAM latin1_swedish_ci OK 
host_template 3 MyISAM latin1_swedish_ci OK 
host_template_graph 50 MyISAM latin1_swedish_ci OK 
host_template_snmp_query 10 MyISAM latin1_swedish_ci OK 
plugin_config 0 MyISAM latin1_swedish_ci OK 
plugin_db_changes 0 MyISAM latin1_swedish_ci OK 
plugin_hooks 2 MyISAM latin1_swedish_ci OK 
plugin_realms 1 MyISAM latin1_swedish_ci OK 
plugin_thold_contacts 2 MyISAM latin1_swedish_ci OK 
plugin_thold_log 0 MyISAM latin1_swedish_ci OK 
plugin_thold_template_contact 0 MyISAM latin1_swedish_ci OK 
plugin_thold_threshold_contact 0 MyISAM latin1_swedish_ci OK 
poller 0 MyISAM latin1_swedish_ci OK 
poller_command 0 MyISAM latin1_swedish_ci OK 
poller_item 8469 MyISAM latin1_swedish_ci OK 
poller_output 0 MyISAM latin1_swedish_ci OK 
poller_reindex 1326 MyISAM latin1_swedish_ci OK 
poller_time 4 MyISAM latin1_swedish_ci OK 
reportit_measurands 0 MyISAM latin1_swedish_ci OK 
reportit_reports 0 MyISAM latin1_swedish_ci OK 
reportit_rvars 0 MyISAM latin1_swedish_ci OK 
reportit_templates 0 MyISAM latin1_swedish_ci OK 
reportit_variables 0 MyISAM latin1_swedish_ci OK 
rra 5 MyISAM latin1_swedish_ci OK 
rra_cf 18 MyISAM latin1_swedish_ci OK 
settings 123 MyISAM latin1_swedish_ci OK 
settings_graphs 58 MyISAM latin1_swedish_ci OK 
settings_tree 0 MyISAM latin1_swedish_ci OK 
snmp_query 11 MyISAM latin1_swedish_ci OK 
snmp_query_graph 24 MyISAM latin1_swedish_ci OK 
snmp_query_graph_rrd 53 MyISAM latin1_swedish_ci OK 
snmp_query_graph_rrd_sv 50 MyISAM latin1_swedish_ci OK 
snmp_query_graph_sv 50 MyISAM latin1_swedish_ci OK 
thold_data 0 MyISAM latin1_swedish_ci OK 
thold_template 2 MyISAM latin1_swedish_ci OK 
user_auth 6 MyISAM latin1_swedish_ci OK 
user_auth_perms 0 MyISAM latin1_swedish_ci OK 
user_auth_realm 66 MyISAM latin1_swedish_ci OK 
user_log 12 MyISAM latin1_swedish_ci OK 
version 1 MyISAM latin1_swedish_ci OK 
weathermap_auth 0 MyISAM latin1_swedish_ci OK 
weathermap_maps 0 MyISAM latin1_swedish_ci OK 
 
PHP Module Information 
apache2handler
--------------------------------------------------------------------------------

Apache Version  IBM_HTTP_Server/6.1.0.5 Apache/2.0.47  
Apache API Version  20020903  
Server Administrator  XXXXXXXX
Hostname:Port  XXXXXXXXX:443  
User/Group  cacti(1531181)/1531181  
Max Requests  Per Child: 2000 - Keep Alive: on - Max Per Connection: 100  
Timeouts  Connection: 300 - Keep-Alive: 15  
Virtual Server  Yes  
Server Root  /XXXX/unix/cen/ewwwadmin/site-cactidev  
Loaded Modules  core worker http_core mod_suexec mod_so mod_access mod_log_config mod_mime mod_status mod_dir mod_alias mod_rewrite mod_deflate sapi_apache2 mod_ibm_ssl mod_auth_wsl  

Directive Local Value Master Value 
engine 1 1 
last_modified 0 0 
xbithack 0 0 


Apache Environment
Variable Value 
HTTPS  ON  
HTTPS_CIPHER  SSL_RSA_WITH_RC4_128_SHA  
SSL_CIPHER  SSL_RSA_WITH_RC4_128_SHA  
SSL_PROTOCOL_VERSION  TLSV1  
HTTPS_KEYSIZE  128  
HTTPS_SECRETKEYSIZE  128  
SSL_CLIENT_SESSIONID  eBsAAMKq51+xoYBfWFkdegp64qlYWFhYFovBRwYAAAA=  
SSL_CLIENT_NEWSESSIONID  TRUE  
SSL_SERVER_DN  CN=cXXXXXXX
SSL_SERVER_CN  XXXXXXXX
SSL_SERVER_L  XXXXXX
SSL_SERVER_ST  XXXXXX
SSL_SERVER_C  XXXX
SSL_SERVER_O  XXXXXX
SSL_SERVER_OU  Terms of use at www.verisign.com/rpa (c)00, Quality & Process Leadership  
auth-user  XXXXX
REMOTE_USER  XXXXXX
auth-type  basic  
HTTP_ACCEPT  image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/x-shockwave-flash, */*  
HTTP_REFERER  https://XXXXXXXXX/hosting/utilities.php  
HTTP_ACCEPT_LANGUAGE  en-us  
HTTP_ACCEPT_ENCODING  gzip, deflate  
HTTP_USER_AGENT  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; F-6.0SP2-20041109; InfoPath.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)  
HTTP_HOST  XXXXXXXXXXXX
HTTP_CONNECTION  Keep-Alive  
HTTP_COOKIE  WSL-credential=R77Cuke/T1oBAOAABHVzZXJpZD1ybXVzc2VsbABpcGFkZHI9MTkuODYuODguMTkyAGFjaWdyb3VwPU5PTk9WVk0AZGVwdD01MDAxSjE1NzcAb3JnY29kZT0/AGVtcGNvZGU9QQBtcnJvbGU9TgBvcmc9SVQAY29tcGFueT1DSUJFUiwuSU5DLgBkaXZhYmJyPT8Ac2l0ZWNvZGU9NTA1MQBjaXR5PURFQVJCT1JOAHN0YXRlPU1JAGNvdW50cnk9VVNBAHN1YmplY3RpZD1ybXVzc2VsbEBmb3JkLmNvbQAAAENOPXdzbHY0LWludGVybmFsANedDPjJK0R6jOf1fsZTqHMXg+ZPKtRbuJhKK/STwjxm/uzuWFXjHjCdKY/gbuQ3qDGmjYKxV9jGtQI+7Oik1bXwhPEBC05DkApgNLEmeSuRgO7unedvmtFfCAe5pXCpQQYFMoG5Jp9aMXv8yUJ3TMv4Cb3QYL6Mu3A4t0E4JR2o; Ford-WSL=R77Cuke/T1pwcm9mc3B3MkB3ZWJmYXJtLmRlYXJib3JuLmZvcmQuY29tAHJtdXNzZWxsADE5Ljg2Ljg4LjE5MgBOT05PVlZNADUwMDFKMTU3NwA/AEEATgBJVABDSUJFUiwuSU5DLgA/ADUwNTEAREVBUkJPUk4ATUkAVVNBAE5VTEwuY2VudG9rcwAAQW1+0cphQzVcxBVLIKS+cQMU+oI2RlYXkKOIK+75BDf9uP8AisRgCqePRDo6pTMll+sDZ5q3hLXL0x61Wi0N2g==; PC-Cookie=aXNzdWVUaW1lPVN1biwgMjQgRmViIDIwMDggMTQ6MDk6MjQgR01UOmV4cGlyZVRpbWU9TW9uLCAyNSBGZWIgMjAwOCAxNDowOToyNCBHTVQ6dXNlcklkPUZPUkROQTFccm11c3NlbGw%3D; CDSID=XXXXXX; RLF=cm11c3NlbGw6WQAAQ049d3NsdjQtaW50ZXJuYWwAFkSKCMOWke1m3VHVEyENVNCt8D0D+0j3MA/Cv9XZtAKRB3AnqcZVhhA6PkMj8s7gdaRcRvADJ6cgstdL506CcUzduLAQem5m5+I9gc+FTdFNxszWLCik9KPzVc0D4NahBM5x2AyUmSa/1EZnBTVodvH+y+UaO31kK2hKttu6m8E=; SITESERVER=ID=293ddeaf9513e621ac5906690ad37358; Cacti=5fa61173b0860d1569488201bcf92bb4; clickedFoldert2=1%5E14%5E15%5E20%5E18%5E63%5E65%5E19%5E; highlightedTreeviewLinkt2=19  
PATH  /usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin  
SERVER_SIGNATURE  <address>IBM_HTTP_Server/6.1.0.5 Apache/2.0.47 Server at XXXXXXXX Port 443</address>  
SERVER_SOFTWARE  IBM_HTTP_Server/6.1.0.5 Apache/2.0.47  
SERVER_NAME  XXXXXXXXX  
SERVER_ADDR  XXXXXXX
SERVER_PORT  443  
REMOTE_ADDR  XXXXXXXXXX
DOCUMENT_ROOT  /proj/netops/cacti/cacti-0.8.6j  
SERVER_ADMIN  XXXXXXXX  
SCRIPT_FILENAME  /proj/netops/cacti/cacti-0.8.6j/hosting/utilities.php  
REMOTE_PORT  1156  
GATEWAY_INTERFACE  CGI/1.1  
SERVER_PROTOCOL  HTTP/1.1  
REQUEST_METHOD  GET  
QUERY_STRING  action=view_tech  
REQUEST_URI  /hosting/utilities.php?action=view_tech  
SCRIPT_NAME  /hosting/utilities.php  


HTTP Headers Information
HTTP Request Headers 
HTTP Request  GET /hosting/utilities.php?action=view_tech HTTP/1.1  
Accept  image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/x-shockwave-flash, */*  
Referer  https://XXXXXXXX/hosting/utilities.php  
Accept-Language  en-us  
Accept-Encoding  gzip, deflate  
User-Agent  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; F-6.0SP2-20041109; InfoPath.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)  
Host  XXXXXXXXX
Connection  Keep-Alive  
Cookie  WSL-credential=R77Cuke/T1oBAOAABHVzZXJpZD1ybXVzc2VsbABpcGFkZHI9MTkuODYuODguMTkyAGFjaWdyb3VwPU5PTk9WVk0AZGVwdD01MDAxSjE1NzcAb3JnY29kZT0/AGVtcGNvZGU9QQBtcnJvbGU9TgBvcmc9SVQAY29tcGFueT1DSUJFUiwuSU5DLgBkaXZhYmJyPT8Ac2l0ZWNvZGU9NTA1MQBjaXR5PURFQVJCT1JOAHN0YXRlPU1JAGNvdW50cnk9VVNBAHN1YmplY3RpZD1ybXVzc2VsbEBmb3JkLmNvbQAAAENOPXdzbHY0LWludGVybmFsANedDPjJK0R6jOf1fsZTqHMXg+ZPKtRbuJhKK/STwjxm/uzuWFXjHjCdKY/gbuQ3qDGmjYKxV9jGtQI+7Oik1bXwhPEBC05DkApgNLEmeSuRgO7unedvmtFfCAe5pXCpQQYFMoG5Jp9aMXv8yUJ3TMv4Cb3QYL6Mu3A4t0E4JR2o; Ford-WSL=R77Cuke/T1pwcm9mc3B3MkB3ZWJmYXJtLmRlYXJib3JuLmZvcmQuY29tAHJtdXNzZWxsADE5Ljg2Ljg4LjE5MgBOT05PVlZNADUwMDFKMTU3NwA/AEEATgBJVABDSUJFUiwuSU5DLgA/ADUwNTEAREVBUkJPUk4ATUkAVVNBAE5VTEwuY2VudG9rcwAAQW1+0cphQzVcxBVLIKS+cQMU+oI2RlYXkKOIK+75BDf9uP8AisRgCqePRDo6pTMll+sDZ5q3hLXL0x61Wi0N2g==; PC-Cookie=aXNzdWVUaW1lPVN1biwgMjQgRmViIDIwMDggMTQ6MDk6MjQgR01UOmV4cGlyZVRpbWU9TW9uLCAyNSBGZWIgMjAwOCAxNDowOToyNCBHTVQ6dXNlcklkPUZPUkROQTFccm11c3NlbGw%3D; CDSID=XXXXXX; RLF=cm11c3NlbGw6WQAAQ049d3NsdjQtaW50ZXJuYWwAFkSKCMOWke1m3VHVEyENVNCt8D0D+0j3MA/Cv9XZtAKRB3AnqcZVhhA6PkMj8s7gdaRcRvADJ6cgstdL506CcUzduLAQem5m5+I9gc+FTdFNxszWLCik9KPzVc0D4NahBM5x2AyUmSa/1EZnBTVodvH+y+UaO31kK2hKttu6m8E=; SITESERVER=ID=293ddeaf9513e621ac5906690ad37358; Cacti=5fa61173b0860d1569488201bcf92bb4; clickedFoldert2=1%5E14%5E15%5E20%5E18%5E63%5E65%5E19%5E; highlightedTreeviewLinkt2=19  
HTTP Response Headers 
X-Powered-By  PHP/4.4.4  
Expires  Thu, 19 Nov 1981 08:52:00 GMT  
Last-Modified  Sun, 24 Feb 2008 15:19:56 GMT  
Cache-Control  no-store, no-cache, must-revalidate, post-check=0, pre-check=0  
Pragma  no-cache  


ctype
--------------------------------------------------------------------------------

ctype functions  enabled  


eAccelerator
--------------------------------------------------------------------------------

eAccelerator support enabled 
Version  0.9.5  
Caching Enabled  true  
Optimizer Enabled  true  
Memory Size  16,777,180 Bytes  
Memory Available  12,733,784 Bytes  
Memory Allocated  4,043,396 Bytes  
Cached Scripts  46  
Removed Scripts  0  
Cached Keys  0  

Directive Local Value Master Value 
eaccelerator.allowed_admin_path no value no value 
eaccelerator.cache_dir /proj/netops/cacti/eaccelerator /proj/netops/cacti/eaccelerator 
eaccelerator.check_mtime 0 0 
eaccelerator.compress 1 1 
eaccelerator.compress_level 9 9 
eaccelerator.debug 0 0 
eaccelerator.enable 1 1 
eaccelerator.filter no value no value 
eaccelerator.log_file no value no value 
eaccelerator.name_space no value no value 
eaccelerator.optimizer 1 1 
eaccelerator.shm_max 0 0 
eaccelerator.shm_only 0 0 
eaccelerator.shm_prune_period 0 0 
eaccelerator.shm_size 16 16 
eaccelerator.shm_ttl 0 0 


gd
--------------------------------------------------------------------------------

GD Support  enabled  
GD Version  bundled (2.0.28 compatible)  
FreeType Support  enabled  
FreeType Linkage  with freetype  
GIF Read Support  enabled  
GIF Create Support  enabled  
JPG Support  enabled  
PNG Support  enabled  
WBMP Support  enabled  
XBM Support  enabled  


mysql
--------------------------------------------------------------------------------

MySQL Support enabled 
Active Persistent Links  1  
Active Links  1  
Client API version  5.0.27  
MYSQL_MODULE_TYPE  external  
MYSQL_SOCKET  /tmp/mysql.sock  
MYSQL_INCLUDE  -I/proj/netops/cacti/mysql/include/mysql  
MYSQL_LIBS  -L/proj/netops/cacti/mysql/lib/mysql -lmysqlclient  

Directive Local Value Master Value 
mysql.allow_persistent On On 
mysql.connect_timeout 60 60 
mysql.default_host no value no value 
mysql.default_password no value no value 
mysql.default_port no value no value 
mysql.default_socket no value no value 
mysql.default_user no value no value 
mysql.max_links Unlimited Unlimited 
mysql.max_persistent Unlimited Unlimited 
mysql.trace_mode Off Off 


overload
--------------------------------------------------------------------------------

User-Space Object Overloading Support  enabled  


pcre
--------------------------------------------------------------------------------

PCRE (Perl Compatible Regular Expressions) Support  enabled  
PCRE Library Version  6.6 06-Feb-2006  


posix
--------------------------------------------------------------------------------

Revision  $Revision: 1.51.2.4.2.1 $  


rrdtool
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

rrdtool Version 1.2.x extension 

rrdtool support  enabled  


session
--------------------------------------------------------------------------------

Session Support  enabled  
Registered save handlers  files user  

Directive Local Value Master Value 
session.auto_start Off Off 
session.bug_compat_42 Off Off 
session.bug_compat_warn On On 
session.cache_expire 180 180 
session.cache_limiter nocache nocache 
session.cookie_domain no value no value 
session.cookie_lifetime 0 0 
session.cookie_path / / 
session.cookie_secure Off Off 
session.entropy_file no value no value 
session.entropy_length 0 0 
session.gc_divisor 1000 1000 
session.gc_maxlifetime 1440 1440 
session.gc_probability 1 1 
session.name Cacti PHPSESSID 
session.referer_check no value no value 
session.save_handler files files 
session.save_path /tmp /tmp 
session.serialize_handler php php 
session.use_cookies On On 
session.use_only_cookies Off Off 
session.use_trans_sid Off Off 


sockets
--------------------------------------------------------------------------------

Sockets Support  enabled  


standard
--------------------------------------------------------------------------------

Regex Library  Bundled library enabled  
Dynamic Library Support  enabled  
Path to sendmail  /usr/sbin/sendmail -t -i  

Directive Local Value Master Value 
assert.active 1 1 
assert.bail 0 0 
assert.callback no value no value 
assert.quiet_eval 0 0 
assert.warning 1 1 
auto_detect_line_endings 0 0 
default_socket_timeout 60 60 
safe_mode_allowed_env_vars PHP_ PHP_ 
safe_mode_protected_env_vars LD_LIBRARY_PATH LD_LIBRARY_PATH 
url_rewriter.tags a=href,area=href,frame=src,input=src,form=fakeentry a=href,area=href,frame=src,input=src,form=fakeentry 
user_agent no value no value 


tokenizer
--------------------------------------------------------------------------------

Tokenizer Support  enabled  


xml
--------------------------------------------------------------------------------

XML Support  active  
XML Namespace Support  active  
EXPAT Version  1.95.6  


zlib
--------------------------------------------------------------------------------

ZLib Support  enabled  
Compiled Version  1.2.3  
Linked Version  1.2.3  

Directive Local Value Master Value 
zlib.output_compression Off Off 
zlib.output_compression_level -1 -1 
zlib.output_handler no value no value 


Additional Modules
Module Name

fmangeant · Post by **fmangeant** » Mon Feb 25, 2008 3:42 am

Hi

which version of RRDtool are you using ?

Rael · Post by **Rael** » Mon Feb 25, 2008 8:54 am

1.2.26

kyosanim · Post by **kyosanim** » Mon Feb 25, 2008 11:00 am

you should higher the memory_limit to from 8mb to 32mb.

which "availability method" do you use?
i had problems with "icmp" ping method - perhaps you can try "snmp".

Rael · Post by **Rael** » Mon Feb 25, 2008 1:21 pm

I'm sorry, what memory limit should I up?

I'm using UDP ping but for down'd host detection I'm using snmp.

fmangeant · Post by **fmangeant** » Tue Feb 26, 2008 2:35 am

memory_limit is a PHP setting (in php.ini).

Rael · Post by **Rael** » Tue Feb 26, 2008 10:09 am

I believe this has already been changed

memory_limit = 256M ; Maximum amount of memory a script may consume (8MB)

Is this what you're talking about?

Rael · Post by **Rael** » Tue Mar 04, 2008 9:07 am

No other ideas ehh?

Post by **TheWitness** » Tue Mar 04, 2008 9:54 pm

Use the SVN version of Spine. It corrects some segmentation faults due to a Linux kernel issue. You can run "./spine -V 5" to verify proper operations fom the command line.

TheWitness

Help debug/troubleshoot spine timeout issues

Help debug/troubleshoot spine timeout issues

Who is online