Issue with weathermap and boost.

[jmarshall28@gmail.com]

I am having an issue with weathermap ver 0.97a and Cacti/spine 0.8.7e. Cacti works fine in boost mode as long as weathermap the plugin is disabled. However when I enable weathermap and issue the SET command in my weathermaps for them to use poller_output I have issues. The time taken to generate weathermaps steadily increaes, all the while increasing CPU/Memory in the MYSQL process. Evenrtually either mysqld runs out of connections or PHP runs out of memory and the system becomes unresponsive requiring httpd/mysqld restarts. It only takes about an hour for this to occur. The server is a beast of a machine. 16 Xeon X5560 2.8ghz cores, 32GB of RAM, fairly dencent I/O as well. When you click console->weathermaps-> along the bottom it gives you data about weathermap runtime. Without boost my server is around 9 seconds, with boost it increases each run till around 1900 seconds, then the server fails. I first saw this behavior on another server running CactiEZ 0.6 that had been updated to cacti/spine 0.87e,weathermap 0.97a. I thought it was just an issue with CactiEZ. I am currently running CENTOS 5.4 Kernel 2.6.18-164-el5 64bit on the mew 16 core server that I installed cacti on. Here is the tech support page for cacti.

[jmarshall28@gmail.com]

Server information.

Date Tue, 06 Jul 2010 14:46:59 -0400
Cacti Version 0.8.7e
Cacti OS unix
SNMP Version NET-SNMP version: 5.3.2.2
RRDTool Version RRDTool 1.2.x
Hosts 53
Graphs 1219
Data Sources Script/Command: 5
SNMP: 256
SNMP Query: 1257
Script - Script Server (PHP): 1
Total: 1519
Poller Information
Interval 60
Type spine
Items Action[0]: 2524
Action[1]: 5
Action[2]: 1
Total: 2530
Concurrent Processes 2
Max Threads 10
PHP Servers 1
Script Timeout 25
Max OID 10
Last Run Statistics Time:4.7798 Method:spine Processes:2 Threads:10 Hosts:54 HostsPerProcess:27 DataSources:2444 RRDsProcessed:1429
PHP Information
PHP Version 5.1.6
PHP OS Linux
PHP uname Linux cacti2.localdomain 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64
PHP SNMP Installed
max_execution_time 30
memory_limit 768M
MySQL Table Information
Name Rows Engine Collation Check Status
cdef 13 MyISAM latin1_swedish_ci OK
cdef_items 36 MyISAM latin1_swedish_ci OK
colors 101 MyISAM latin1_swedish_ci OK
data_input 17 MyISAM latin1_swedish_ci OK
data_input_data 13742 MyISAM latin1_swedish_ci OK
data_input_fields 57 MyISAM latin1_swedish_ci OK
data_local 1519 MyISAM latin1_swedish_ci OK
data_source_stats_daily 2527 MyISAM utf8_general_ci OK
data_source_stats_hourly 2217 MyISAM utf8_general_ci OK
data_source_stats_hourly_cache 0 MEMORY utf8_general_ci The storage engine for the table doesn't support check
data_source_stats_hourly_last 0 MEMORY utf8_general_ci The storage engine for the table doesn't support check
data_source_stats_monthly 2520 MyISAM utf8_general_ci OK
data_source_stats_weekly 2520 MyISAM utf8_general_ci OK
data_source_stats_yearly 2520 MyISAM utf8_general_ci OK
data_template 90 MyISAM latin1_swedish_ci OK
data_template_data 1609 MyISAM latin1_swedish_ci OK
data_template_data_rra 7886 MyISAM latin1_swedish_ci OK
data_template_rrd 2656 MyISAM latin1_swedish_ci OK
graph_local 1219 MyISAM latin1_swedish_ci OK
graph_template_input 138 MyISAM latin1_swedish_ci OK
graph_template_input_defs 467 MyISAM latin1_swedish_ci OK
graph_templates 58 MyISAM latin1_swedish_ci OK
graph_templates_gprint 4 MyISAM latin1_swedish_ci OK
graph_templates_graph 1277 MyISAM latin1_swedish_ci OK
graph_templates_item 10405 MyISAM latin1_swedish_ci OK
graph_tree 2 MyISAM latin1_swedish_ci OK
graph_tree_items 196 MyISAM latin1_swedish_ci OK
host 53 MyISAM latin1_swedish_ci OK
host_graph 342 MyISAM latin1_swedish_ci OK
host_snmp_cache 102953 MyISAM latin1_swedish_ci OK
host_snmp_query 55 MyISAM latin1_swedish_ci OK
host_template 9 MyISAM latin1_swedish_ci OK
host_template_graph 31 MyISAM latin1_swedish_ci OK
host_template_snmp_query 16 MyISAM latin1_swedish_ci OK
mac_track_approved_macs 0 MyISAM latin1_swedish_ci OK
mac_track_device_types 0 MyISAM latin1_swedish_ci OK
mac_track_devices 0 MyISAM latin1_swedish_ci OK
mac_track_ip_ranges 0 MyISAM latin1_swedish_ci OK
mac_track_ips 0 MEMORY latin1_swedish_ci The storage engine for the table doesn't support check
mac_track_macwatch 0 MyISAM latin1_swedish_ci OK
mac_track_oui_database 0 MyISAM latin1_swedish_ci OK
mac_track_ports 0 MyISAM latin1_swedish_ci OK
mac_track_processes 0 MyISAM latin1_swedish_ci OK
mac_track_scan_dates 0 MyISAM latin1_swedish_ci OK
mac_track_scanning_functions 13 MyISAM latin1_swedish_ci OK
mac_track_sites 0 MyISAM latin1_swedish_ci OK
mac_track_temp_ports 0 MEMORY latin1_swedish_ci The storage engine for the table doesn't support check
plugin_aggregate_color_template_items 44 MyISAM latin1_swedish_ci OK
plugin_aggregate_color_templates 4 MyISAM latin1_swedish_ci OK
plugin_autom8_graph_rule_items 6 MyISAM latin1_swedish_ci OK
plugin_autom8_graph_rules 3 MyISAM latin1_swedish_ci OK
plugin_autom8_match_rule_items 7 MyISAM latin1_swedish_ci OK
plugin_autom8_tree_rule_items 4 MyISAM latin1_swedish_ci OK
plugin_autom8_tree_rules 2 MyISAM latin1_swedish_ci OK
plugin_config 13 MyISAM latin1_swedish_ci OK
plugin_db_changes 367 MyISAM latin1_swedish_ci OK
plugin_discover_hosts 0 MyISAM latin1_swedish_ci OK
plugin_discover_template 1 MyISAM latin1_swedish_ci OK
plugin_flowview_devices 1 MyISAM latin1_swedish_ci OK
plugin_flowview_dnscache 0 MEMORY latin1_swedish_ci The storage engine for the table doesn't support check
plugin_flowview_queries 0 MyISAM latin1_swedish_ci OK
plugin_flowview_schedules 0 MyISAM latin1_swedish_ci OK
plugin_hooks 106 MyISAM latin1_swedish_ci OK
plugin_nectar 0 MyISAM latin1_swedish_ci OK
plugin_nectar_items 0 MyISAM latin1_swedish_ci OK
plugin_realms 16 MyISAM latin1_swedish_ci OK
plugin_routerconfigs_accounts 0 MyISAM latin1_swedish_ci OK
plugin_routerconfigs_backups 0 MyISAM latin1_swedish_ci OK
plugin_routerconfigs_devices 0 MyISAM latin1_swedish_ci OK
plugin_routerconfigs_devicetypes 2 MyISAM latin1_swedish_ci OK
plugin_rrdclean 223 MyISAM latin1_swedish_ci OK
plugin_rrdclean_action 0 MyISAM latin1_swedish_ci OK
plugin_thold_contacts 14 MyISAM latin1_swedish_ci OK
plugin_thold_log 0 MyISAM latin1_swedish_ci OK
plugin_thold_template_contact 0 MyISAM latin1_swedish_ci OK
plugin_thold_threshold_contact 0 MyISAM latin1_swedish_ci OK
plugin_update_info 19 MyISAM latin1_swedish_ci OK
plugin_wmi_accounts 0 MyISAM latin1_swedish_ci OK
plugin_wmi_queries 1 MyISAM latin1_swedish_ci OK
poller 0 MyISAM latin1_swedish_ci OK
poller_command 0 MyISAM latin1_swedish_ci OK
poller_item 2530 MyISAM latin1_swedish_ci OK
poller_output 0 MEMORY latin1_swedish_ci The storage engine for the table doesn't support check
poller_output_boost 0 MEMORY latin1_swedish_ci The storage engine for the table doesn't support check
poller_output_boost_processes 0 MEMORY latin1_swedish_ci The storage engine for the table doesn't support check
poller_output_rt 0 MyISAM latin1_swedish_ci OK
poller_reindex 54 MyISAM latin1_swedish_ci OK
poller_time 2 MyISAM latin1_swedish_ci OK
reportit_cache_measurands 0 MyISAM latin1_swedish_ci OK
reportit_cache_reports 0 MyISAM latin1_swedish_ci OK
reportit_cache_variables 0 MyISAM latin1_swedish_ci OK
reportit_data_items 442 MyISAM latin1_swedish_ci OK
reportit_data_source_items 3 MyISAM latin1_swedish_ci OK
reportit_measurands 5 MyISAM latin1_swedish_ci OK
reportit_presets 5 MyISAM latin1_swedish_ci OK
reportit_recipients 0 MyISAM latin1_swedish_ci OK
reportit_reports 6 MyISAM latin1_swedish_ci OK
reportit_results_1 57 MyISAM latin1_swedish_ci OK
reportit_results_10 38 MyISAM latin1_swedish_ci OK
reportit_results_5 58 MyISAM latin1_swedish_ci OK
reportit_results_6 110 MyISAM latin1_swedish_ci OK
reportit_results_8 69 MyISAM latin1_swedish_ci OK
reportit_results_9 110 MyISAM latin1_swedish_ci OK
reportit_rvars 5 MyISAM latin1_swedish_ci OK
reportit_templates 1 MyISAM latin1_swedish_ci OK
reportit_variables 1 MyISAM latin1_swedish_ci OK
rra 5 MyISAM latin1_swedish_ci OK
rra_cf 20 MyISAM latin1_swedish_ci OK
settings 218 MyISAM latin1_swedish_ci OK
settings_graphs 49 MyISAM latin1_swedish_ci OK
settings_tree 0 MyISAM latin1_swedish_ci OK
snmp_query 11 MyISAM latin1_swedish_ci OK
snmp_query_graph 25 MyISAM latin1_swedish_ci OK
snmp_query_graph_rrd 46 MyISAM latin1_swedish_ci OK
snmp_query_graph_rrd_sv 63 MyISAM latin1_swedish_ci OK
snmp_query_graph_sv 45 MyISAM latin1_swedish_ci OK
thold_data 10 MyISAM latin1_swedish_ci OK
thold_template 3 MyISAM latin1_swedish_ci OK
user_auth 23 MyISAM latin1_swedish_ci OK
user_auth_perms 2 MyISAM latin1_swedish_ci OK
user_auth_realm 228 MyISAM latin1_swedish_ci OK
user_log 674 MyISAM latin1_swedish_ci OK
version 1 MyISAM latin1_swedish_ci OK
weathermap_auth 27 MyISAM latin1_swedish_ci OK
weathermap_data 621 MyISAM latin1_swedish_ci OK
weathermap_groups 5 MyISAM latin1_swedish_ci OK
weathermap_maps 19 MyISAM latin1_swedish_ci OK
weathermap_settings 4 MyISAM latin1_swedish_ci OK
The Following Tables were Skipped Due to being INNODB
mac_track_macauth

PHP Module Information
apache2handler
--------------------------------------------------------------------------------

Apache Version Apache/2.2.3 (CentOS)
Apache API Version 20051115
Server Administrator root@localhost
Hostname:Port cacti2.localdomain:0
User/Group apache(48)/48
Max Requests Per Child: 4000 - Keep Alive: off - Max Per Connection: 100
Timeouts Connection: 120 - Keep-Alive: 15
Virtual Server No
Server Root /etc/httpd
Loaded Modules core prefork http_core mod_so mod_auth_basic mod_auth_digest mod_authn_file mod_authn_alias mod_authn_anon mod_authn_dbm mod_authn_default mod_authz_host mod_authz_user mod_authz_owner mod_authz_groupfile mod_authz_dbm mod_authz_default util_ldap mod_authnz_ldap mod_include mod_log_config mod_logio mod_env mod_ext_filter mod_mime_magic mod_expires mod_deflate mod_headers mod_usertrack mod_setenvif mod_mime mod_dav mod_status mod_autoindex mod_info mod_dav_fs mod_vhost_alias mod_negotiation mod_dir mod_actions mod_speling mod_userdir mod_alias mod_rewrite mod_proxy mod_proxy_balancer mod_proxy_ftp mod_proxy_http mod_proxy_connect mod_cache mod_suexec mod_disk_cache mod_file_cache mod_mem_cache mod_cgi mod_version mod_auth_mysql mod_perl mod_php5 mod_proxy_ajp mod_python mod_ssl

Directive Local Value Master Value
engine 1 1
last_modified 0 0
xbithack 0 0


Apache Environment
Variable Value
HTTP_ACCEPT application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
HTTP_REFERER http://172.25.95.15/utilities.php
HTTP_ACCEPT_LANGUAGE en-US
HTTP_USER_AGENT Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2)
HTTP_ACCEPT_ENCODING gzip, deflate
HTTP_HOST 172.25.95.15
HTTP_CONNECTION Keep-Alive
HTTP_COOKIE cacti=9r47ds8hehac48jbde91ufn626
PATH /sbin:/usr/sbin:/bin:/usr/bin
SERVER_SIGNATURE <address>Apache/2.2.3 (CentOS) Server at 172.25.95.15 Port 80</address>
SERVER_SOFTWARE Apache/2.2.3 (CentOS)
SERVER_NAME 172.25.95.15
SERVER_ADDR 172.25.95.15
SERVER_PORT 80
REMOTE_ADDR 172.25.95.200
DOCUMENT_ROOT /var/www/html
SERVER_ADMIN root@localhost
SCRIPT_FILENAME /var/www/html/utilities.php
REMOTE_PORT 52869
GATEWAY_INTERFACE CGI/1.1
SERVER_PROTOCOL HTTP/1.1
REQUEST_METHOD GET
QUERY_STRING action=view_tech
REQUEST_URI /utilities.php?action=view_tech
SCRIPT_NAME /utilities.php


HTTP Headers Information
HTTP Request Headers
HTTP Request GET /utilities.php?action=view_tech HTTP/1.1
Accept application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Referer http://172.25.95.15/utilities.php
Accept-Language en-US
User-Agent Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2)
Accept-Encoding gzip, deflate
Host 172.25.95.15
Connection Keep-Alive
Cookie cacti=9r47ds8hehac48jbde91ufn626
HTTP Response Headers
X-Powered-By PHP/5.1.6
Expires Thu, 19 Nov 1981 08:52:00 GMT
Last-Modified Tue, 06 Jul 2010 18:46:58 GMT
Cache-Control no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma no-cache
P3P CP="CAO PSA OUR"


bz2
--------------------------------------------------------------------------------

BZip2 Support Enabled
Stream Wrapper support compress.bz2://
Stream Filter support bzip2.decompress, bzip2.compress
BZip2 Version 1.0.3, 15-Feb-2005


calendar
--------------------------------------------------------------------------------

Calendar support enabled


ctype
--------------------------------------------------------------------------------

ctype functions enabled


curl
--------------------------------------------------------------------------------

CURL support enabled
CURL Information libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5


date
--------------------------------------------------------------------------------

date/time support enabled
Timezone Database Version 0.system
Timezone Database internal
Default timezone America/New_York

Directive Local Value Master Value
date.default_latitude 31.7667 31.7667
date.default_longitude 35.2333 35.2333
date.sunrise_zenith 90.583333 90.583333
date.sunset_zenith 90.583333 90.583333
date.timezone no value no value


eAccelerator
--------------------------------------------------------------------------------

eAccelerator support enabled
Version 0.9.5.2
Caching Enabled true
Optimizer Enabled true
Memory Size 33,554,368 Bytes
Memory Available 25,493,888 Bytes
Memory Allocated 8,060,480 Bytes
Cached Scripts 52
Removed Scripts 0
Cached Keys 0

Directive Local Value Master Value
eaccelerator.allowed_admin_path no value no value
eaccelerator.cache_dir /var/cache/php-eaccelerator /var/cache/php-eaccelerator
eaccelerator.check_mtime 1 1
eaccelerator.compress 1 1
eaccelerator.compress_level 9 9
eaccelerator.content shm_and_disk shm_and_disk
eaccelerator.debug 0 0
eaccelerator.enable 1 1
eaccelerator.filter no value no value
eaccelerator.keys shm_and_disk shm_and_disk
eaccelerator.log_file /var/log/httpd/eaccelerator_log /var/log/httpd/eaccelerator_log
eaccelerator.name_space no value no value
eaccelerator.optimizer 1 1
eaccelerator.sessions shm_and_disk shm_and_disk
eaccelerator.shm_max 0 0
eaccelerator.shm_only 0 0
eaccelerator.shm_prune_period 0 0
eaccelerator.shm_size 0 0
eaccelerator.shm_ttl 3600 3600


exif
--------------------------------------------------------------------------------

EXIF Support enabled
EXIF Version 1.4 $Id: exif.c,v 1.173.2.5 2006/04/10 18:23:24 helly Exp $
Supported EXIF Version 0220
Supported filetypes JPEG,TIFF


ftp
--------------------------------------------------------------------------------

FTP support enabled


gd
--------------------------------------------------------------------------------

GD Support enabled
GD Version bundled (2.0.28 compatible)
FreeType Support enabled
FreeType Linkage with freetype
FreeType Version 2.2.1
GIF Read Support enabled
GIF Create Support enabled
JPG Support enabled
PNG Support enabled
WBMP Support enabled
XBM Support enabled


gettext
--------------------------------------------------------------------------------

GetText Support enabled


gmp
--------------------------------------------------------------------------------

gmp support enabled


hash
--------------------------------------------------------------------------------

hash support enabled
Hashing Engines md4 md5 sha1 sha256 sha384 sha512 ripemd128 ripemd160 whirlpool tiger128,3 tiger160,3 tiger192,3 tiger128,4 tiger160,4 tiger192,4 snefru gost adler32 crc32 crc32b haval128,3 haval160,3 haval192,3 haval224,3 haval256,3 haval128,4 haval160,4 haval192,4 haval224,4 haval256,4 haval128,5 haval160,5 haval192,5 haval224,5 haval256,5


iconv
--------------------------------------------------------------------------------

iconv support enabled
iconv implementation glibc
iconv library version 2.5

Directive Local Value Master Value
iconv.input_encoding ISO-8859-1 ISO-8859-1
iconv.internal_encoding ISO-8859-1 ISO-8859-1
iconv.output_encoding ISO-8859-1 ISO-8859-1


ldap
--------------------------------------------------------------------------------

LDAP Support enabled
RCS Version $Id: ldap.c,v 1.161.2.3 2006/01/01 12:50:08 sniper Exp $
Total Links 0/unlimited
API Version 3001
Vendor Name OpenLDAP
Vendor Version 20343
SASL Support Enabled


libxml
--------------------------------------------------------------------------------

libXML support active
libXML Version 2.6.26
libXML streams enabled


mime_magic
--------------------------------------------------------------------------------

mime_magic support invalid magic file, disabled

Directive Local Value Master Value
mime_magic.debug Off Off
mime_magic.magicfile /usr/share/file/magic.mime /usr/share/file/magic.mime


mysql
--------------------------------------------------------------------------------

MySQL Support enabled
Active Persistent Links 1
Active Links 1
Client API version 5.0.77
MYSQL_MODULE_TYPE external
MYSQL_SOCKET /var/lib/mysql/mysql.sock
MYSQL_INCLUDE -I/usr/include/mysql
MYSQL_LIBS -L/usr/lib64/mysql -lmysqlclient

Directive Local Value Master Value
mysql.allow_persistent On On
mysql.connect_timeout 60 60
mysql.default_host no value no value
mysql.default_password no value no value
mysql.default_port no value no value
mysql.default_socket no value no value
mysql.default_user no value no value
mysql.max_links Unlimited Unlimited
mysql.max_persistent Unlimited Unlimited
mysql.trace_mode Off Off


mysqli
--------------------------------------------------------------------------------

MysqlI Support enabled
Client API library version 5.0.77
Client API header version 5.0.77
MYSQLI_SOCKET /var/lib/mysql/mysql.sock

Directive Local Value Master Value
mysqli.default_host no value no value
mysqli.default_port 3306 3306
mysqli.default_pw no value no value
mysqli.default_socket no value no value
mysqli.default_user no value no value
mysqli.max_links Unlimited Unlimited
mysqli.reconnect Off Off


openssl
--------------------------------------------------------------------------------

OpenSSL support enabled
OpenSSL Version OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008


pcre
--------------------------------------------------------------------------------

PCRE (Perl Compatible Regular Expressions) Support enabled
PCRE Library Version 6.6 06-Feb-2006


PDO
--------------------------------------------------------------------------------

PDO support enabled
PDO drivers mysql, sqlite


pdo_mysql
--------------------------------------------------------------------------------

PDO Driver for MySQL, client library version 5.0.77


pdo_sqlite
--------------------------------------------------------------------------------

PDO Driver for SQLite 3.x enabled
PECL Module version 1.0.1 $Id: pdo_sqlite.c,v 1.10.2.6 2006/01/01 12:50:12 sniper Exp $
SQLite Library 3.3.6


posix
--------------------------------------------------------------------------------

Revision $Revision: 1.70.2.3 $


pspell
--------------------------------------------------------------------------------

PSpell Support enabled


Reflection
--------------------------------------------------------------------------------

Reflection enabled
Version $Id: php_reflection.c,v 1.164.2.33 2006/03/29 14:28:42 tony2001 Exp $


session
--------------------------------------------------------------------------------

Session Support enabled
Registered save handlers files user eaccelerator
Registered serializer handlers php php_binary wddx

Directive Local Value Master Value
session.auto_start Off Off
session.bug_compat_42 Off Off
session.bug_compat_warn On On
session.cache_expire 180 180
session.cache_limiter nocache nocache
session.cookie_domain no value no value
session.cookie_lifetime 0 0
session.cookie_path / /
session.cookie_secure Off Off
session.entropy_file no value no value
session.entropy_length 0 0
session.gc_divisor 1000 1000
session.gc_maxlifetime 1440 1440
session.gc_probability 1 1
session.hash_bits_per_character 5 5
session.hash_function 0 0
session.name cacti PHPSESSID
session.referer_check no value no value
session.save_handler files files
session.save_path /var/lib/php/session /var/lib/php/session
session.serialize_handler php php
session.use_cookies On On
session.use_only_cookies Off Off
session.use_trans_sid 0 0


shmop
--------------------------------------------------------------------------------

shmop support enabled


SimpleXML
--------------------------------------------------------------------------------

Simplexml support enabled
Revision $Revision: 1.151.2.22 $
Schema support enabled


snmp
--------------------------------------------------------------------------------

NET-SNMP Support enabled
NET-SNMP Version 5.3.2.2


sockets
--------------------------------------------------------------------------------

Sockets Support enabled


SPL
--------------------------------------------------------------------------------

SPL support enabled
Interfaces Countable, OuterIterator, RecursiveIterator, SeekableIterator, SplObserver, SplSubject
Classes AppendIterator, ArrayIterator, ArrayObject, BadFunctionCallException, BadMethodCallException, CachingIterator, DirectoryIterator, DomainException, EmptyIterator, FilterIterator, InfiniteIterator, InvalidArgumentException, IteratorIterator, LengthException, LimitIterator, LogicException, NoRewindIterator, OutOfBoundsException, OutOfRangeException, OverflowException, ParentIterator, RangeException, RecursiveArrayIterator, RecursiveCachingIterator, RecursiveDirectoryIterator, RecursiveFilterIterator, RecursiveIteratorIterator, RuntimeException, SimpleXMLIterator, SplFileInfo, SplFileObject, SplObjectStorage, SplTempFileObject, UnderflowException, UnexpectedValueException


standard
--------------------------------------------------------------------------------

Regex Library Bundled library enabled
Dynamic Library Support enabled
Path to sendmail /usr/sbin/sendmail -t -i

Directive Local Value Master Value
assert.active 1 1
assert.bail 0 0
assert.callback no value no value
assert.quiet_eval 0 0
assert.warning 1 1
auto_detect_line_endings 0 0
default_socket_timeout 60 60
safe_mode_allowed_env_vars PHP_ PHP_
safe_mode_protected_env_vars LD_LIBRARY_PATH LD_LIBRARY_PATH
url_rewriter.tags a=href,area=href,frame=src,input=src,form=fakeentry a=href,area=href,frame=src,input=src,form=fakeentry
user_agent no value no value


sysvmsg
--------------------------------------------------------------------------------

sysvmsg support enabled
Revision $Revision: 1.20.2.3 $


tokenizer
--------------------------------------------------------------------------------

Tokenizer Support enabled


wddx
--------------------------------------------------------------------------------

WDDX Support enabled
WDDX Session Serializer enabled


xml
--------------------------------------------------------------------------------

XML Support active
XML Namespace Support active
libxml2 Version 2.6.26


zlib
--------------------------------------------------------------------------------

ZLib Support enabled
Stream Wrapper support compress.zlib://
Stream Filter support zlib.inflate, zlib.deflate
Compiled Version 1.2.3
Linked Version 1.2.3

Directive Local Value Master Value
zlib.output_compression Off Off
zlib.output_compression_level -1 -1
zlib.output_handler no value no value


Additional Modules
Module Name
dbase
sysvsem
sysvshm

[dionvdc]

Perhaps Howie or someone else has more an exact answer, but i think it could be caused by the tables filling up.
What's your boost interval and does the time weathermaps take decrease after boost has run?

[Howie]

It does sound like tables filling up. I don't see what might cause that....

Is your rra folder a symlink to somewhere else? Weathermap will be trying to map an rrd filename to a ds ID in the Cacti database, and that might get confused if the rra_root isn't the 'real' filename for the file.

How many rows are there in the weathermap_data table? There should be one for each target in your maps. Each one should have a local_data_id in there too.

[jmarshall28@gmail.com]

I do not use a symlink, the path is real and not symbolic.
Boost is set to update every 30minutes or 100,000 records.

There are 827 rows in the weathmap-data table and the local ID is present in each row, and it matches the rrd-file number as well.

id rrdfile data_source_name last_time last_value last_calc sequence local_data_id
101 path_rra/fhr00tal_traffic_in_301.rrd traffic_in 1278515524 235782956410611 63044914.5 161050 301


Weathermap data Before boost run at 11:05 a.m. today

Last Completed Run: Wed, 07 Jul 2010 11:04:37 EDT: 19 maps were run in 1510 seconds with 0 warnings.

After Boost run.
Last Completed Run: Wed, 07 Jul 2010 11:11:22 EDT: 19 maps were run in 1855 seconds with 0 warnings.

[jmarshall28@gmail.com]

Log Outputs

Before Boost update:

Current Boost Status
Boost On Demand Updating: Idle
Total Data Sources: 2737
Total Boost Records: 67358
Boost Storage Statistics
Database Engine: MEMORY
Current Boost Table Size: 10 MBytes
Avg Bytes/Record: 152 Bytes
Max Record Length: 108 Bytes
Max Allowed Boost Table Size: 2 GBytes
Estimated Maximum Records: 11184810 Records
Runtime Statistics
Last Start Time: 2010-07-07 10:35:27
Last Run Duration: 32 seconds (2% of update frequency)
RRD Updates: 12376
Peak Poller Memory: 16.86 MBytes
Detailed Runtime Timers: RRDUpdates:12376 TotalTime:32.2096 range_local_data_id:0.01 rcaston_add:0.02 get_records:0.12 results_cycle:27.09 rrd_path:0.3 rrd_template:0.49 rrd_lastupdate:11.23 rrd_field_names:0.06 rrdupdate:14.87 delete:0.03
Max Poller Memory Allowed: 1024 MBytes
Run Time Configuration
Update Frequency: 30 Minutes
Next Start Time: 2010-07-07 11:05:27
Maximum Records: 100000 Records
Maximum Allowed Runtime: 20 Minutes
Boost Server Details
Server Config Status: Enabled
Multiprocess Server: Single Process
Update Timeout: 2 Seconds
Server/Port: localhost@9050
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdtool
Image Caching
Image Cacing Status: Enabled
Cache Directory: /var/www/html/plugins/boost/cache/
Cached Files: 32 Files
Cached Files Size: 1 MBytes

After Boost Update

Current Boost Status
Boost On Demand Updating: Idle
Total Data Sources: 2737
Total Boost Records: 1293
Boost Storage Statistics
Database Engine: MEMORY
Current Boost Table Size: 11 MBytes
Avg Bytes/Record: 152 Bytes
Max Record Length: 108 Bytes
Max Allowed Boost Table Size: 2 GBytes
Estimated Maximum Records: 11184810 Records
Runtime Statistics
Last Start Time: 2010-07-07 11:06:22
Last Run Duration: 6 seconds (0% of update frequency)
RRD Updates: 72644
Peak Poller Memory: 58.56 MBytes
Detailed Runtime Timers: RRDUpdates:72644 TotalTime:6.5115 range_local_data_id:0.03 rcaston_add:0.16 get_records:0.72 results_cycle:5.29 rrd_path:0.25 rrd_template:0.53 rrd_lastupdate:0.26 rrd_field_names:0.37 rrdupdate:3.23 delete:0.19
Max Poller Memory Allowed: 1024 MBytes
Run Time Configuration
Update Frequency: 30 Minutes
Next Start Time: 2010-07-07 11:36:22
Maximum Records: 100000 Records
Maximum Allowed Runtime: 20 Minutes
Boost Server Details
Server Config Status: Enabled
Multiprocess Server: Single Process
Update Timeout: 2 Seconds
Server/Port: localhost@9050
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdtool
Image Caching
Image Cacing Status: Enabled
Cache Directory: /var/www/html/plugins/boost/cache/
Cached Files: 32 Files
Cached Files Size: 1 MBytes

Log before during and after boost update.

07/07/2010 11:08:22 AM - SYSTEM THOLD STATS: Time:0.0011 Tholds:0 Hosts:0
07/07/2010 11:08:22 AM - NECTAR: Poller[0] nectar_poller_bottom, reports found: 0
07/07/2010 11:08:22 AM - SYSTEM STATS: Time:20.8429 Method:spine Processes:1 Threads:1 Hosts:55 HostsPerProcess:55 DataSources:2648 RRDsProcessed:0
07/07/2010 11:08:01 AM - SPINE: Poller[0] Host[1] Description[Localhost] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
07/07/2010 11:07:24 AM - SYSTEM THOLD STATS: Time:0.0011 Tholds:0 Hosts:0
07/07/2010 11:07:24 AM - NECTAR: Poller[0] nectar_poller_bottom, reports found: 0
07/07/2010 11:07:24 AM - SYSTEM STATS: Time:22.7270 Method:spine Processes:1 Threads:1 Hosts:55 HostsPerProcess:55 DataSources:2649 RRDsProcessed:0
07/07/2010 11:07:03 AM - SPINE: Poller[0] Host[1] Description[Localhost] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
07/07/2010 11:06:47 AM - SYSTEM DSSTATS STATS: Type:DAILY, Time:25.1113
07/07/2010 11:06:42 AM - DSSTATS: Poller[0] WARNING: File '' Does not exist
07/07/2010 11:06:41 AM - DSSTATS: Poller[0] WARNING: File '/var/www/html/rra/mfnorlorld75c001_cisco_procmem_1152.rrd' Does not exist
07/07/2010 11:06:41 AM - DSSTATS: Poller[0] WARNING: File '/var/www/html/rra/mfnorlorld75c001_cisco_iomem_1151.rrd' Does not exist
07/07/2010 11:06:41 AM - DSSTATS: Poller[0] WARNING: File '/var/www/html/rra/mfnorlorld75c001_procmem_1150.rrd' Does not exist
07/07/2010 11:06:41 AM - DSSTATS: Poller[0] WARNING: File '/var/www/html/rra/mfnorlorld75c001_5min_cpu_1149.rrd' Does not exist
07/07/2010 11:06:41 AM - DSSTATS: Poller[0] WARNING: File '/var/www/html/rra/mfnorlorld75c001_traffic_in_1142.rrd' Does not exist
07/07/2010 11:06:41 AM - DSSTATS: Poller[0] WARNING: File '/var/www/html/rra/mfnorlorld75c001_traffic_in_1141.rrd' Does not exist
07/07/2010 11:06:28 AM - SYSTEM BOOST STATS: Time:6.5100 RRDUpdates:72644
07/07/2010 11:06:22 AM - SYSTEM THOLD STATS: Time:0.0011 Tholds:0 Hosts:0
07/07/2010 11:06:22 AM - NECTAR: Poller[0] nectar_poller_bottom, reports found: 0
07/07/2010 11:06:22 AM - SYSTEM STATS: Time:20.8507 Method:spine Processes:1 Threads:1 Hosts:55 HostsPerProcess:55 DataSources:2648 RRDsProcessed:0
07/07/2010 11:06:01 AM - SPINE: Poller[0] Host[1] Description[Localhost] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
07/07/2010 11:05:23 AM - SYSTEM THOLD STATS: Time:0.0011 Tholds:0 Hosts:0
07/07/2010 11:05:23 AM - NECTAR: Poller[0] nectar_poller_bottom, reports found: 0
07/07/2010 11:05:23 AM - SYSTEM STATS: Time:21.9062 Method:spine Processes:1 Threads:1 Hosts:55 HostsPerProcess:55 DataSources:2648 RRDsProcessed:0
07/07/2010 11:05:01 AM - SPINE: Poller[0] Host[1] Description[Localhost] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
07/07/2010 11:04:23 AM - SYSTEM THOLD STATS: Time:0.0011 Tholds:0 Hosts:0
07/07/2010 11:04:23 AM - NECTAR: Poller[0] nectar_poller_bottom, reports found: 0
07/07/2010 11:04:23 AM - SYSTEM STATS: Time:21.8757 Method:spine Processes:1 Threads:1 Hosts:55 HostsPerProcess:55 DataSources:2648 RRDsProcessed:0
07/07/2010 11:04:01 AM - SPINE: Poller[0] Host[1] Description[Localhost] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'
07/07/2010 11:03:22 AM - SYSTEM THOLD STATS: Time:0.0013 Tholds:0 Hosts:0
07/07/2010 11:03:22 AM - NECTAR: Poller[0] nectar_poller_bottom, reports found: 0
07/07/2010 11:03:22 AM - SYSTEM STATS: Time:20.8490 Method:spine Processes:1 Threads:1 Hosts:55 HostsPerProcess:55 DataSources:2648 RRDsProcessed:0
07/07/2010 11:03:01 AM - SPINE: Poller[0] Host[1] Description[Localhost] ERROR: Empty result [127.0.0.1]: '/usr/bin/php -q /var/www/html/plugins/syslog/syslog_counter.php'

[Howie]

Last Completed Run: Wed, 07 Jul 2010 11:04:37 EDT: 19 maps were run in 1510 seconds with 0 warnings.

After Boost run.
Last Completed Run: Wed, 07 Jul 2010 11:11:22 EDT: 19 maps were run in 1855 seconds with 0 warnings.

Are these maps especially big (number of targets) or use non-rrd targets like snmp? Those are terrible runtimes... I would expect more like 45 seconds

If they are normal-sized maps, then I guess it's something like a missing index in a SQL table.

[jmarshall28@gmail.com]

Also something to note. The machine is configured for one poller thread and no forks. However the machine consumes more and memory and CPU over time, and i get more and more poller.php threads that exist over time. eventually either the database server runs out of connections or php runs out of memory. Usually it is the database server runs out of connections and a restart of the database server and killing all the php poller threads gets it going again. Of course you loose the data in those threads. For example a current top and ps snapshot

Cpu(s): 2.7%us, 14.8%sy, 0.0%ni, 82.3%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32949488k total, 3030420k used, 29919068k free, 43128k buffers
Swap: 34996216k total, 0k used, 34996216k free, 698524k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4589 mysql 15 0 421m 66m 4496 S 212.6 0.2 173:16.35 mysqld

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
6845 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
6846 ? S 0:08 php /var/www/html/poller.php
6993 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
6994 ? S 0:07 php /var/www/html/poller.php
7143 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
7144 ? S 0:06 php /var/www/html/poller.php
7296 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
7297 ? S 0:05 php /var/www/html/poller.php
7351 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
7352 ? S 0:04 php /var/www/html/poller.php
7508 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
7509 ? S 0:04 php /var/www/html/poller.php
7660 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
7661 ? S 0:03 php /var/www/html/poller.php
7823 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
7824 ? S 0:03 php /var/www/html/poller.php
7987 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
7988 ? S 0:03 php /var/www/html/poller.php
8150 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
8151 ? S 0:03 php /var/www/html/poller.php
8345 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
8346 ? R 0:03 php /var/www/html/poller.php
8502 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
8503 ? S 0:03 php /var/www/html/poller.php
8660 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
8661 ? S 0:02 php /var/www/html/poller.php
8822 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
8823 ? S 0:02 php /var/www/html/poller.php
8970 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
8971 ? S 0:02 php /var/www/html/poller.php
9123 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
9124 ? S 0:02 php /var/www/html/poller.php
9282 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
9283 ? S 0:02 php /var/www/html/poller.php
9434 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
9435 ? S 0:01 php /var/www/html/poller.php
9589 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
9590 ? S 0:01 php /var/www/html/poller.php
9737 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
9738 ? S 0:01 php /var/www/html/poller.php
9885 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
9886 ? S 0:01 php /var/www/html/poller.php
10033 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
10034 ? S 0:01 php /var/www/html/poller.php
10181 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
10182 ? S 0:01 php /var/www/html/poller.php
10334 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
10335 ? S 0:01 php /var/www/html/poller.php
10482 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
10483 ? S 0:01 php /var/www/html/poller.php
10630 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
10631 ? S 0:01 php /var/www/html/poller.php
10926 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
10927 ? S 0:01 php /var/www/html/poller.php
11079 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
11080 ? S 0:01 php /var/www/html/poller.php
11230 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
11231 ? S 0:01 php /var/www/html/poller.php
11398 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
11399 ? S 0:01 php /var/www/html/poller.php
11546 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
11547 ? S 0:01 php /var/www/html/poller.php
11694 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
11695 ? S 0:01 php /var/www/html/poller.php
11840 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
11841 ? S 0:01 php /var/www/html/poller.php
11992 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
11993 ? S 0:01 php /var/www/html/poller.php
12140 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
12141 ? S 0:00 php /var/www/html/poller.php
12288 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
12289 ? S 0:00 php /var/www/html/poller.php
12438 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
12439 ? S 0:00 php /var/www/html/poller.php
12584 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
12585 ? S 0:00 php /var/www/html/poller.php
12736 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
12737 ? S 0:00 php /var/www/html/poller.php
12886 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
12887 ? S 0:00 php /var/www/html/poller.php
12935 pts/1 S+ 0:00 grep poller

[jmarshall28@gmail.com]

howie check pms. I sent you a couple of map configs

[jmarshall28@gmail.com]

I do not use snmp targets, just cacti rrd targets like so

SET rrd_use_poller_output 1

LINK router1-router2
INFOURL /graph.php?rra_id=all&local_graph_id=64
OVERLIBGRAPH /graph_image.php?local_graph_id=64&rra_id=0&graph_nolegend=true&graph
_height=100&graph_width=300
TARGET /var/www/html/rra/router1_traffic_in_71.rrd
NODES router1 router2

[Howie]

Weathermap runs on the end of the poller, so if it takes 30 minutes to run, then that's probably why you have all those poller processes, especially with 1 minute polling. That is also where all the memory is going - lots of copies of the poller.

If you take out the rrd_use_poller_output then what is the weathermap runtime? I know that you'll get zeroes in the map, but it at least points the finger clearly at the ReadData code if the time goes back down, because the poller_output data collection will still happen if there are still rows in weathermap_data. So if the time goes down, it's not the poller_output collection, but the reading from weathermap_data that is slow. If it doesn't go down, then it's probably the collection side. I would expect the collection side to be the problem, if anything.

[jmarshall28@gmail.com]

It does not. Change the files at 11:40 to remove the SET command.


Last 11:45 data.


Last Completed Run: Wed, 07 Jul 2010 11:45:30 EDT: 19 maps were run in 3188 seconds with 2 warnings.

The 2 warnings concern a dead router on one of the maps.

[Howie]

It does not. Change the files at 11:40 to remove the SET command.


Last 11:45 data.


Last Completed Run: Wed, 07 Jul 2010 11:45:30 EDT: 19 maps were run in 3188 seconds with 2 warnings.

The 2 warnings concern a dead router on one of the maps.

OK - and now clear the weathermap_data table? (DELETE FROM weathermap_data)

What does that do for the time?

[jmarshall28@gmail.com]

I spoke to soon howie. I did not wait long enough i suppose.
All my weathermaps are now blank and the times have gotten much smaller from run to run. Also mysql is back to normal, and the poller threads are back to one. Here are the stats:

Last Completed Run: Wed, 07 Jul 2010 13:06:48 EDT: 19 maps were run in 26 seconds with 590 warnings.

[root@cacti2 configs]# ps -ax | grep poller
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
31765 ? Ss 0:00 /bin/sh -c php /var/www/html/poller.php > /dev/null 2>&1
31766 ? S 0:00 php /var/www/html/poller.php
31883 pts/1 S+ 0:00 grep poller
[root@cacti2 configs]#

top - 13:08:41 up 2:49, 2 users, load average: 0.70, 0.60, 0.97
Tasks: 253 total, 1 running, 252 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.3%us, 9.1%sy, 0.0%ni, 88.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32949488k total, 1200600k used, 31748888k free, 57592k buffers
Swap: 34996216k total, 0k used, 34996216k free, 753044k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4589 mysql 15 0 425m 66m 4496 S 101.1 0.2 279:10.31 mysqld

[jmarshall28@gmail.com]

I enabled SET rrd_use_poller_output 1 on one map out of 19 and my run times started increasing again. Of course that one map is not blank either.

Last Completed Run: Wed, 07 Jul 2010 13:48:04 EDT: 19 maps were run in 102 seconds with 2 warnings.