Cact box all of a sudden running extremely slow..

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Developers, Moderators

Post Reply
xminos
Cacti User
Posts: 63
Joined: Wed Apr 26, 2006 8:10 am

Cact box all of a sudden running extremely slow..

Post by xminos »

Cact box all of a sudden running extremely slow..

I would greatly appreciate any input on how to troubleshoot this..

Background info: This box has been running without issues for the last 6 months. Nothing has changed, no scripts added no packages updated..

Last week some users started complaining of slowness on the cacti front end. Taking a look at the server everything looks good, CPU is low, lots and lots of free memory and iowait time is well within reason. Cacti is just very very slow to load and sometimes even times out. I have verfied via packet captures that there is no packet loss to the server itself as to rule out the network.

The back end seems fine. Hosts pull no problem, scripts run without issues. The CLI feels responsive as it always has been.

I have bounced apache and mysqld. I have checked mysql for errors and even rebooted the box itself. No change.

Host info:

[cactiuser@cacti01 ~]$ uname -a
Linux cacti01 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux
Cacti Version - 0.8.7d
Plugin Architecture - 2.4
Poller Type - Cactid v
Server Info - Linux 2.6.18-128.el5
Web Server - Apache/2.2.3 (Red Hat)
PHP - 5.1.6
PHP Extensions - libxml, xml, wddx, tokenizer, sysvshm, sysvsem, sysvmsg, standard, SimpleXML, sockets, SPL, shmop, session, Reflection, pspell, posix, mime_magic, iconv, hash, gmp, gettext, ftp, exif, date, curl, ctype, calendar, bz2, zlib, pcre, openssl, apache2handler, dbase, gd, mysql, mysqli, odbc, PDO, pdo_mysql, PDO_ODBC, pdo_sqlite, snmp
MySQL - 5.0.45
RRDTool - 1.2.23
SNMP - 5.3.2.2
Plugins
  • Global Plugin Settings (settings - v0.5)
    Host Info (hostinfo - v0.2)
    Network Tools (tools - v0.3)
    Cacti-ReportIt (reportit - v0.6.1)
    Report Creator (reports - v0.3)
    Create Aggregate Graphs (aggregate - v0.70 Beta 1)
Using spine, poller runtime has not changed
08/26/2009 04:35:54 AM - SYSTEM STATS: Time:51.8682 Method:spine Processes:15 Threads:40 Hosts:655 HostsPerProcess:44 DataSources:26308 RRDsProcessed:14982
08/26/2009 04:40:49 AM - SYSTEM STATS: Time:48.5362 Method:spine Processes:15 Threads:40 Hosts:655 HostsPerProcess:44 DataSources:26305 RRDsProcessed:14981
08/26/2009 04:46:03 AM - SYSTEM STATS: Time:61.1654 Method:spine Processes:15 Threads:40 Hosts:655 HostsPerProcess:44 DataSources:26299 RRDsProcessed:14999
08/26/2009 04:50:50 AM - SYSTEM STATS: Time:48.2487 Method:spine Processes:15 Threads:40 Hosts:655 HostsPerProcess:44 DataSources:26308 RRDsProcessed:15003
08/26/2009 04:55:50 AM - SYSTEM STATS: Time:48.1015 Method:spine Processes:15 Threads:40 Hosts:655 HostsPerProcess:44 DataSources:26305 RRDsProcessed:15017
08/26/2009 05:00:50 AM - SYSTEM STATS: Time:48.8416 Method:spine Processes:15 Threads:40 Hosts:655 HostsPerProcess:44 DataSources:26299 RRDsProcessed:14993
08/27/2009 04:56:05 PM - SYSTEM STATS: Time:63.5703 Method:spine Processes:15 Threads:40 Hosts:655 HostsPerProcess:44 DataSources:26305 RRDsProcessed:15007




Technical Support
General Information
Date Thu, 27 Aug 2009 21:42:46 +0000
Cacti Version 0.8.7d
Cacti OS unix
SNMP Version NET-SNMP version: 5.3.2.2
RRDTool Version RRDTool 1.2.x
Hosts 658
Graphs 13560
Data Sources Script/Command: 1189
SNMP: 5214
SNMP Query: 10089
Script Query: 315
Script - Script Server (PHP): 5
Total: 16812
Poller Information
Interval 300
Type spine
Items Action[0]: 24864
Action[1]: 1949
Action[2]: 5
Total: 26818
Concurrent Processes 15
Max Threads 40
PHP Servers 8
Script Timeout 600
Max OID 60
Last Run Statistics Time:122.3269 Method:spine Processes:15 Threads:40 Hosts:655 HostsPerProcess:44 DataSources:26305 RRDsProcessed:15017
PHP Information
PHP Version 5.1.6
PHP OS Linux
PHP uname Linux cacti01 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64
PHP SNMP Installed
max_execution_time 30
memory_limit 512M
MySQL Table Information
Name Rows Engine Collation Check Status
cdef 24 MyISAM latin1_swedish_ci OK
cdef_items 55 MyISAM latin1_swedish_ci OK
colors 132 MyISAM latin1_swedish_ci OK
data_input 68 MyISAM latin1_swedish_ci OK
data_input_data 167657 MyISAM latin1_swedish_ci OK
data_input_fields 386 MyISAM latin1_swedish_ci OK
data_local 16812 MyISAM latin1_swedish_ci OK
data_template 361 MyISAM latin1_swedish_ci OK
data_template_data 17173 MyISAM latin1_swedish_ci OK
data_template_data_rra 93161 MyISAM latin1_swedish_ci OK
data_template_rrd 30987 MyISAM latin1_swedish_ci OK
graph_local 13560 MyISAM latin1_swedish_ci OK
graph_template_input 761 MyISAM latin1_swedish_ci OK
graph_template_input_defs 1767 MyISAM latin1_swedish_ci OK
graph_templates 264 MyISAM latin1_swedish_ci OK
graph_templates_gprint 4 MyISAM latin1_swedish_ci OK
graph_templates_graph 13824 MyISAM latin1_swedish_ci OK
graph_templates_item 102252 MyISAM latin1_swedish_ci OK
graph_tree 25 MyISAM latin1_swedish_ci OK
graph_tree_items 2295 MyISAM latin1_swedish_ci OK
host 658 MyISAM latin1_swedish_ci OK
host_graph 3927 MyISAM latin1_swedish_ci OK
host_snmp_cache 272308 MyISAM latin1_swedish_ci OK
host_snmp_query 966 MyISAM latin1_swedish_ci OK
host_template 24 MyISAM latin1_swedish_ci OK
host_template_graph 160 MyISAM latin1_swedish_ci OK
host_template_snmp_query 22 MyISAM latin1_swedish_ci OK
plugin_aggregate_color_template_items 44 MyISAM latin1_swedish_ci OK
plugin_aggregate_color_templates 4 MyISAM latin1_swedish_ci OK
plugin_config 4 MyISAM latin1_swedish_ci OK
plugin_db_changes 48 MyISAM latin1_swedish_ci OK
plugin_hooks 44 MyISAM latin1_swedish_ci OK
plugin_realms 7 MyISAM latin1_swedish_ci OK
plugin_snmptt 0 MyISAM latin1_swedish_ci OK
plugin_snmptt_alert 0 MyISAM latin1_swedish_ci OK
plugin_snmptt_statistics 0 MyISAM latin1_swedish_ci OK
plugin_snmptt_tree 0 MyISAM latin1_swedish_ci OK
plugin_snmptt_unknown 0 MyISAM latin1_swedish_ci OK
plugin_thold_contacts 39 MyISAM latin1_swedish_ci OK
plugin_thold_log 8688 MyISAM latin1_swedish_ci OK
plugin_thold_template_contact 0 MyISAM latin1_swedish_ci OK
plugin_thold_threshold_contact 0 MyISAM latin1_swedish_ci OK
plugin_update_info 13 MyISAM latin1_swedish_ci OK
poller 0 MyISAM latin1_swedish_ci OK
poller_command 0 MyISAM latin1_swedish_ci OK
poller_item 26818 MyISAM latin1_swedish_ci OK
poller_output 1 MyISAM latin1_swedish_ci OK
poller_output_boost 0 MEMORY latin1_swedish_ci The storage engine for the table doesn't support check
poller_output_boost_processes 0 MEMORY latin1_swedish_ci The storage engine for the table doesn't support check
poller_output_rt 0 MyISAM latin1_swedish_ci OK
poller_reindex 961 MyISAM latin1_swedish_ci OK
poller_time 13 MyISAM latin1_swedish_ci OK
reportit_cache_measurands 0 MyISAM latin1_swedish_ci OK
reportit_cache_reports 0 MyISAM latin1_swedish_ci OK
reportit_cache_variables 0 MyISAM latin1_swedish_ci OK
reportit_measurands 4 MyISAM latin1_swedish_ci OK
reportit_presets 2 MyISAM latin1_swedish_ci OK
reportit_recipients 0 MyISAM latin1_swedish_ci OK
reportit_reports 3 MyISAM latin1_swedish_ci OK
reportit_results_1 219 MyISAM latin1_swedish_ci OK
reportit_results_2 165 MyISAM latin1_swedish_ci OK
reportit_results_7 79 MyISAM latin1_swedish_ci OK
reportit_rrdlist_1 219 MyISAM latin1_swedish_ci OK
reportit_rrdlist_2 165 MyISAM latin1_swedish_ci OK
reportit_rrdlist_7 79 MyISAM latin1_swedish_ci OK
reportit_rvars 3 MyISAM latin1_swedish_ci OK
reportit_templates 4 MyISAM latin1_swedish_ci OK
reportit_variables 4 MyISAM latin1_swedish_ci OK
reports 24 MyISAM latin1_swedish_ci OK
reports_data 817 MyISAM latin1_swedish_ci OK
rra 9 MyISAM latin1_swedish_ci OK
rra_cf 26 MyISAM latin1_swedish_ci OK
settings 187 MyISAM latin1_swedish_ci OK
settings_graphs 1337 MyISAM latin1_swedish_ci OK
settings_tree 0 MyISAM latin1_swedish_ci OK
snmp_query 21 MyISAM latin1_swedish_ci OK
snmp_query_graph 43 MyISAM latin1_swedish_ci OK
snmp_query_graph_rrd 78 MyISAM latin1_swedish_ci OK
snmp_query_graph_rrd_sv 73 MyISAM latin1_swedish_ci OK
snmp_query_graph_sv 51 MyISAM latin1_swedish_ci OK
thold 1 MyISAM latin1_swedish_ci OK
thold_data 1039 MyISAM latin1_swedish_ci OK
thold_template 36 MyISAM latin1_swedish_ci OK
tholdset 1 MyISAM latin1_swedish_ci OK
user_auth 45 MyISAM latin1_swedish_ci OK
user_auth_perms 566 MyISAM latin1_swedish_ci OK
user_auth_realm 436 MyISAM latin1_swedish_ci OK
user_log 11040 MyISAM latin1_swedish_ci OK
version 1 MyISAM latin1_swedish_ci OK
weathermap_auth 2 MyISAM latin1_swedish_ci OK
weathermap_maps 1 MyISAM latin1_swedish_ci OK
PHP Module Information
apache2handler
Apache Version Apache/2.2.3 (Red Hat)
Apache API Version 20051115
Server Administrator root@ cacti01.nmcc. spectrum.com
Hostname:Port cacti01.nmcc. spectrum.com:80
User/Group apache(48)/48
Max Requests Per Child: 4000 - Keep Alive: off - Max Per Connection: 100
Timeouts Connection: 120 - Keep-Alive: 15
Virtual Server Yes
Server Root /etc/httpd
Loaded Modules core prefork http_core mod_so mod_auth_basic mod_auth_digest mod_authn_file mod_authn_alias mod_authn_anon mod_authn_dbm mod_authn_default mod_authz_host mod_authz_user mod_authz_owner mod_authz_groupfile mod_authz_dbm mod_authz_default util_ldap mod_authnz_ldap mod_include mod_log_config mod_logio mod_env mod_ext_filter mod_mime_magic mod_expires mod_deflate mod_headers mod_usertrack mod_setenvif mod_mime mod_dav mod_status mod_autoindex mod_info mod_dav_fs mod_vhost_alias mod_negotiation mod_dir mod_actions mod_speling mod_userdir mod_alias mod_rewrite mod_proxy mod_proxy_balancer mod_proxy_ftp mod_proxy_http mod_proxy_connect mod_cache mod_suexec mod_disk_cache mod_file_cache mod_mem_cache mod_cgi mod_version mod_php5 mod_proxy_ajp

Directive Local Value Master Value
engine 1 1
last_modified 0 0
xbithack 0 0

Apache Environment
Variable Value
HTTP_HOST 208.4.96.10
HTTP_USER_AGENT Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13
HTTP_ACCEPT text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP_ACCEPT_LANGUAGE en-us,en;q=0.5
HTTP_ACCEPT_ENCODING gzip,deflate
HTTP_ACCEPT_CHARSET ISO-8859-1,utf-8;q=0.7,*;q=0.7
HTTP_KEEP_ALIVE 300
HTTP_CONNECTION keep-alive
HTTP_REFERER http://208.4.96.10/cacti/utilities.php
HTTP_COOKIE Cacti=d47kee4a8837vh62a9koq3rv03; PHPSESSID=2gdd0nsufop681tuv44q4baiu4
PATH /sbin:/usr/sbin:/bin:/usr/bin
SERVER_SIGNATURE <address>Apache/2.2.3 (Red Hat) Server at 208.4.96.10 Port 80</address>
SERVER_SOFTWARE Apache/2.2.3 (Red Hat)
SERVER_NAME 208.4.96.10
SERVER_ADDR 208.4.96.10
SERVER_PORT 80
REMOTE_ADDR 10.22.199.246
DOCUMENT_ROOT /apps/cactiweb
SERVER_ADMIN root@ cacti01.nmcc. spectrum.com
SCRIPT_FILENAME /apps/cactiweb/cacti/utilities.php
REMOTE_PORT 7899
GATEWAY_INTERFACE CGI/1.1
SERVER_PROTOCOL HTTP/1.1
REQUEST_METHOD GET
QUERY_STRING action=view_tech
REQUEST_URI /cacti/utilities.php?action=view_tech
SCRIPT_NAME /cacti/utilities.php

HTTP Headers Information
HTTP Request Headers
HTTP Request GET /cacti/utilities.php?action=view_tech HTTP/1.1
Host 208.4.96.10
User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 300
Connection keep-alive
Referer http://208.4.96.10/cacti/utilities.php
Cookie Cacti=d47kee4a8837vh62a9koq3rv03; PHPSESSID=2gdd0nsufop681tuv44q4baiu4
HTTP Response Headers
X-Powered-By PHP/5.1.6
Expires Thu, 19 Nov 1981 08:52:00 GMT
Last-Modified Thu, 27 Aug 2009 21:42:45 GMT
Cache-Control no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma no-cache
P3P CP="CAO PSA OUR"

bz2
BZip2 Support Enabled
Stream Wrapper support compress.bz2://
Stream Filter support bzip2.decompress, bzip2.compress
BZip2 Version 1.0.3, 15-Feb-2005

calendar
Calendar support enabled

ctype
ctype functions enabled

curl
CURL support enabled
CURL Information libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

date
date/time support enabled
Timezone Database Version 0.system
Timezone Database internal
Default timezone UTC

Directive Local Value Master Value
date.default_latitude 31.7667 31.7667
date.default_longitude 35.2333 35.2333
date.sunrise_zenith 90.583333 90.583333
date.sunset_zenith 90.583333 90.583333
date.timezone no value no value

exif
EXIF Support enabled
EXIF Version 1.4 $Id: exif.c,v 1.173.2.5 2006/04/10 18:23:24 helly Exp $
Supported EXIF Version 0220
Supported filetypes JPEG,TIFF

ftp
FTP support enabled

gd
GD Support enabled
GD Version bundled (2.0.28 compatible)
FreeType Support enabled
FreeType Linkage with freetype
FreeType Version 2.2.1
GIF Read Support enabled
GIF Create Support enabled
JPG Support enabled
PNG Support enabled
WBMP Support enabled
XBM Support enabled

gettext
GetText Support enabled

gmp
gmp support enabled

hash
hash support enabled
Hashing Engines md4 md5 sha1 sha256 sha384 sha512 ripemd128 ripemd160 whirlpool tiger128,3 tiger160,3 tiger192,3 tiger128,4 tiger160,4 tiger192,4 snefru gost adler32 crc32 crc32b haval128,3 haval160,3 haval192,3 haval224,3 haval256,3 haval128,4 haval160,4 haval192,4 haval224,4 haval256,4 haval128,5 haval160,5 haval192,5 haval224,5 haval256,5

iconv
iconv support enabled
iconv implementation glibc
iconv library version 2.5

Directive Local Value Master Value
iconv.input_encoding ISO-8859-1 ISO-8859-1
iconv.internal_encoding ISO-8859-1 ISO-8859-1
iconv.output_encoding ISO-8859-1 ISO-8859-1

libxml
libXML support active
libXML Version 2.6.26
libXML streams enabled

mime_magic
mime_magic support invalid magic file, disabled

Directive Local Value Master Value
mime_magic.debug Off Off
mime_magic.magicfile /usr/share/file/magic.mime /usr/share/file/magic.mime

mysql
MySQL Support enabled
Active Persistent Links 1
Active Links 1
Client API version 5.0.45
MYSQL_MODULE_TYPE external
MYSQL_SOCKET /var/lib/mysql/mysql.sock
MYSQL_INCLUDE -I/usr/include/mysql
MYSQL_LIBS -L/usr/lib64/mysql -lmysqlclient

Directive Local Value Master Value
mysql.allow_persistent On On
mysql.connect_timeout 60 60
mysql.default_host no value no value
mysql.default_password no value no value
mysql.default_port no value no value
mysql.default_socket no value no value
mysql.default_user no value no value
mysql.max_links Unlimited Unlimited
mysql.max_persistent Unlimited Unlimited
mysql.trace_mode Off Off

mysqli
MysqlI Support enabled
Client API library version 5.0.45
Client API header version 5.0.45
MYSQLI_SOCKET /var/lib/mysql/mysql.sock

Directive Local Value Master Value
mysqli.default_host no value no value
mysqli.default_port 3306 3306
mysqli.default_pw no value no value
mysqli.default_socket no value no value
mysqli.default_user no value no value
mysqli.max_links Unlimited Unlimited
mysqli.reconnect Off Off

odbc
ODBC Support enabled
Active Persistent Links 0
Active Links 0
ODBC library unixODBC
ODBC_INCLUDE -I/usr/include
ODBC_LFLAGS -L/usr/lib64
ODBC_LIBS -lodbc

Directive Local Value Master Value
odbc.allow_persistent On On
odbc.check_persistent On On
odbc.default_db no value no value
odbc.default_pw no value no value
odbc.default_user no value no value
odbc.defaultbinmode return as is return as is
odbc.defaultlrl return up to 4096 bytes return up to 4096 bytes
odbc.max_links Unlimited Unlimited
odbc.max_persistent Unlimited Unlimited

openssl
OpenSSL support enabled
OpenSSL Version OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

pcre
PCRE (Perl Compatible Regular Expressions) Support enabled
PCRE Library Version 6.6 06-Feb-2006

PDO
PDO support enabled
PDO drivers mysql, odbc, sqlite

pdo_mysql
PDO Driver for MySQL, client library version 5.0.45

PDO_ODBC
PDO Driver for ODBC (unixODBC) enabled
ODBC Connection Pooling Enabled, strict matching

pdo_sqlite
PDO Driver for SQLite 3.x enabled
PECL Module version 1.0.1 $Id: pdo_sqlite.c,v 1.10.2.6 2006/01/01 12:50:12 sniper Exp $
SQLite Library 3.3.6

posix
Revision $Revision: 1.70.2.3 $

pspell
PSpell Support enabled

Reflection
Reflection enabled
Version $Id: php_reflection.c,v 1.164.2.33 2006/03/29 14:28:42 tony2001 Exp $

session
Session Support enabled
Registered save handlers files user
Registered serializer handlers php php_binary wddx

Directive Local Value Master Value
session.auto_start Off Off
session.bug_compat_42 Off Off
session.bug_compat_warn On On
session.cache_expire 180 180
session.cache_limiter nocache nocache
session.cookie_domain no value no value
session.cookie_lifetime 0 0
session.cookie_path / /
session.cookie_secure Off Off
session.entropy_file no value no value
session.entropy_length 0 0
session.gc_divisor 1000 1000
session.gc_maxlifetime 1440 1440
session.gc_probability 1 1
session.hash_bits_per_character 5 5
session.hash_function 0 0
session.name Cacti PHPSESSID
session.referer_check no value no value
session.save_handler files files
session.save_path /var/lib/php/session /var/lib/php/session
session.serialize_handler php php
session.use_cookies On On
session.use_only_cookies Off Off
session.use_trans_sid 0 0

shmop
shmop support enabled

SimpleXML
Simplexml support enabled
Revision $Revision: 1.151.2.22 $
Schema support enabled

snmp
NET-SNMP Support enabled
NET-SNMP Version 5.3.2.2

sockets
Sockets Support enabled

SPL
SPL support enabled
Interfaces Countable, OuterIterator, RecursiveIterator, SeekableIterator, SplObserver, SplSubject
Classes AppendIterator, ArrayIterator, ArrayObject, BadFunctionCallException, BadMethodCallException, CachingIterator, DirectoryIterator, DomainException, EmptyIterator, FilterIterator, InfiniteIterator, InvalidArgumentException, IteratorIterator, LengthException, LimitIterator, LogicException, NoRewindIterator, OutOfBoundsException, OutOfRangeException, OverflowException, ParentIterator, RangeException, RecursiveArrayIterator, RecursiveCachingIterator, RecursiveDirectoryIterator, RecursiveFilterIterator, RecursiveIteratorIterator, RuntimeException, SimpleXMLIterator, SplFileInfo, SplFileObject, SplObjectStorage, SplTempFileObject, UnderflowException, UnexpectedValueException

standard
Regex Library Bundled library enabled
Dynamic Library Support enabled
Path to sendmail /usr/sbin/sendmail -t -i

Directive Local Value Master Value
assert.active 1 1
assert.bail 0 0
assert.callback no value no value
assert.quiet_eval 0 0
assert.warning 1 1
auto_detect_line_endings 0 0
default_socket_timeout 60 60
safe_mode_allowed_env_vars PHP_ PHP_
safe_mode_protected_env_vars LD_LIBRARY_PATH LD_LIBRARY_PATH
url_rewriter.tags a=href,area=href,frame=src,input=src,form=fakeentry a=href,area=href,frame=src,input=src,form=fakeentry
user_agent no value no value

sysvmsg
sysvmsg support enabled
Revision $Revision: 1.20.2.3 $

tokenizer
Tokenizer Support enabled

wddx
WDDX Support enabled
WDDX Session Serializer enabled

xml
XML Support active
XML Namespace Support active
libxml2 Version 2.6.26

zlib
ZLib Support enabled
Stream Wrapper support compress.zlib://
Stream Filter support zlib.inflate, zlib.deflate
Compiled Version 1.2.3
Linked Version 1.2.3

Directive Local Value Master Value
zlib.output_compression Off Off
zlib.output_compression_level -1 -1
zlib.output_handler no value no value

Additional Modules
Module Name
dbase
sysvsem
sysvshm


Server Performance info:

07:00:01 PM CPU %user %nice %system %iowait %steal %idle
07:10:01 PM all 2.59 0.00 1.37 4.93 0.00 91.10
07:20:01 PM all 2.60 0.00 1.39 4.00 0.00 92.00
07:30:01 PM all 2.72 0.00 1.38 4.11 0.00 91.79
07:40:01 PM all 2.60 0.04 1.40 4.74 0.00 91.21
07:50:01 PM all 2.63 0.00 1.39 4.02 0.00 91.96
08:00:01 PM all 2.63 0.00 1.39 3.86 0.00 92.12
08:10:01 PM all 2.66 0.00 1.39 5.90 0.00 90.05
08:20:01 PM all 2.77 0.00 1.42 3.87 0.00 91.95
08:30:01 PM all 2.63 0.00 1.36 4.24 0.00 91.76
08:40:01 PM all 2.63 0.04 1.39 5.04 0.00 90.89
08:50:01 PM all 2.62 0.00 1.37 3.88 0.00 92.14
09:00:01 PM all 2.68 0.00 1.40 4.02 0.00 91.91
09:10:01 PM all 2.64 0.00 1.36 5.11 0.00 90.90
09:20:01 PM all 2.59 0.00 1.37 3.89 0.00 92.15
09:30:01 PM all 2.60 0.00 1.36 3.81 0.00 92.24
Average: all 2.66 0.01 1.39 4.42 0.00 91.52


[cactiuser@cacti01 ~]$ cat /proc/meminfo
MemTotal: 8174168 kB
MemFree: 1119288 kB
Buffers: 406804 kB
Cached: 5753364 kB
SwapCached: 416 kB
Active: 2469232 kB
Inactive: 4201504 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 8174168 kB
LowFree: 1119288 kB
SwapTotal: 4192956 kB
SwapFree: 4192540 kB
Dirty: 24 kB
Writeback: 0 kB
AnonPages: 510124 kB
Mapped: 46468 kB
Slab: 334428 kB
PageTables: 16912 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 8280040 kB
Committed_AS: 1607064 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 265020 kB
VmallocChunk: 34359472759 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB


[cactiuser@ cacti01 ~]$ mpstat
Linux 2.6.18-128.el5 ( cacti01) 08/27/2009

09:48:38 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
09:48:38 PM all 2.86 0.01 1.27 4.43 0.03 0.10 0.00 91.31 2855.42


[cactiuser@ cacti01 ~]$ mpstat -P ALL
Linux 2.6.18-128.el5 ( cacti01) 08/27/2009

09:49:23 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
09:49:23 PM all 2.86 0.01 1.27 4.43 0.03 0.10 0.00 91.31 2855.41
09:49:23 PM 0 2.68 0.00 1.20 1.15 0.01 0.03 0.00 94.93 1146.60
09:49:23 PM 1 2.61 0.00 1.16 1.19 0.00 0.01 0.00 95.03 2.95
09:49:23 PM 2 3.18 0.00 1.30 0.88 0.00 0.00 0.00 94.63 0.00
09:49:23 PM 3 3.16 0.05 1.30 1.38 0.15 0.52 0.00 93.44 1383.08
09:49:23 PM 4 2.44 0.00 1.40 23.73 0.06 0.14 0.00 72.24 167.85
09:49:23 PM 5 2.61 0.00 1.15 4.73 0.01 0.02 0.00 91.47 20.65
09:49:23 PM 6 3.04 0.00 1.30 1.26 0.00 0.01 0.00 94.38 20.58
09:49:23 PM 7 3.13 0.01 1.34 1.10 0.01 0.05 0.00 94.37 113.70



[cactiuser@ cacti01 ~]$ /usr/bin/iostat
Linux 2.6.18-128.el5 ( cacti01) 08/27/2009

avg-cpu: %user %nice %system %iowait %steal %idle
2.86 0.01 1.40 4.43 0.00 91.31

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 8.68 69.10 132.62 37152375 71306628
sda1 0.00 0.11 0.00 56983 468
sda2 0.00 0.01 0.00 3822 832
sda3 8.68 68.98 132.62 37090666 71305328
sdb 401.68 235.36 3765.45 126549674 2024590808
dm-0 1.58 3.76 11.31 2020530 6079632
dm-1 0.95 0.03 7.61 15122 4092544
dm-2 2.16 6.68 15.07 3589922 8104400
dm-3 12.50 12.54 97.52 6741906 52432592
dm-4 0.10 0.62 0.71 333394 383320
dm-5 0.17 13.93 0.01 7491458 7368
dm-6 1.06 31.42 0.38 16895746 205472
dm-7 476.58 235.35 3765.45 126540986 2024591720
xminos
Cacti User
Posts: 63
Joined: Wed Apr 26, 2006 8:10 am

Post by xminos »

some stats from the cacti box. For what it's worth it took me 7-8 minutes to get these graphs to load.. jeeesus this thing is a dog right now..

Saving the png's took a long time and multiple retries.


Nothing abnormal..

Image

Image

Image

Image

Image

Image

Interface stats:

2 Link encap:Ethernet HWaddr 00:15:17:4A:BC:AA
inet addr:10.0.96.10 Bcast 10.0.96.255 Mask:255.255.255.0
inet6 addr: fe80::215:17ff:fe4a:bcaa/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:770047999 errors:0 dropped:0 overruns:0 frame:0
TX packets:107149542 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:250836526446 (233.6 GiB) TX bytes:20712331932 (19.2 GiB)
Memory:fc5e0000-fc600000
User avatar
Linegod
Developer
Posts: 1626
Joined: Thu Feb 20, 2003 10:16 am
Location: Canada
Contact:

Post by Linegod »

And there is nothing _external_ that could be causing an issue? Firewall? Proxy?

And I know you are saying 'nothing abnormal', but are the load and processes what you would expect?
--
Live fast, die young
You're sucking up my bandwidth.

J.P. Pasnak,CD
CCNA, LPIC-1
http://www.warpedsystems.sk.ca
xminos
Cacti User
Posts: 63
Joined: Wed Apr 26, 2006 8:10 am

Post by xminos »

Linegod wrote:And there is nothing _external_ that could be causing an issue? Firewall? Proxy?

And I know you are saying 'nothing abnormal', but are the load and processes what you would expect?
No firewall or proxy. It appears to be on the box itself sadly..

Load and processes look normal when comparing it historically. Same amount of processes, same load on the box which is why it's so odd.

Any suggestions for checking the mysql database for problems?
User avatar
Linegod
Developer
Posts: 1626
Joined: Thu Feb 20, 2003 10:16 am
Location: Canada
Contact:

Post by Linegod »

You could take a look at the tables to see if there is any large overhead, and do an OPTIMIZE TABLE ....

It just doesn't seem like a DB issue to me from looking at your info, but if you could attach the output of 'show processlist;' from a mysql prompt when this issue is happening, it may help.

Average file size on your RRD files would help as well....
--
Live fast, die young
You're sucking up my bandwidth.

J.P. Pasnak,CD
CCNA, LPIC-1
http://www.warpedsystems.sk.ca
xminos
Cacti User
Posts: 63
Joined: Wed Apr 26, 2006 8:10 am

Post by xminos »

It just doesn't seem like a DB issue to me from looking at your info, but if you could attach the output of 'show processlist;' from a mysql prompt when this issue is happening, it may help.

Code: Select all

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 850932
Server version: 5.0.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SHOW PROCESSLIST;
+--------+-----------+-----------+-------+---------+------+-------------+--------------------------------------------------------------------------------------------+
| Id     | User      | Host      | db    | Command | Time | State       | Info                                                                                       |
+--------+-----------+-----------+-------+---------+------+-------------+--------------------------------------------------------------------------------------------+
| 772270 | cactiuser | localhost | cacti | Sleep   |  306 |             | NULL                                                                                       |
| 778102 | cactiuser | localhost | cacti | Sleep   |  304 |             | NULL                                                                                       |
| 786123 | cactiuser | localhost | cacti | Sleep   |  266 |             | NULL                                                                                       |
| 789746 | cactiuser | localhost | cacti | Sleep   |  112 |             | NULL                                                                                       |
| 791228 | cactiuser | localhost | cacti | Sleep   |  317 |             | NULL                                                                                       |
| 795602 | cactiuser | localhost | cacti | Sleep   |  295 |             | NULL                                                                                       |
| 795603 | cactiuser | localhost | cacti | Sleep   |  287 |             | NULL                                                                                       |
| 796332 | cactiuser | localhost | cacti | Sleep   |  287 |             | NULL                                                                                       |
| 797064 | cactiuser | localhost | cacti | Sleep   |  112 |             | NULL                                                                                       |
| 797067 | cactiuser | localhost | cacti | Sleep   |  304 |             | NULL                                                                                       |
| 799984 | cactiuser | localhost | cacti | Sleep   |  288 |             | NULL                                                                                       |
| 802171 | cactiuser | localhost | cacti | Sleep   |  304 |             | NULL                                                                                       |
| 841461 | cactiuser | localhost | cacti | Sleep   |  317 |             | NULL                                                                                       |
| 841462 | cactiuser | localhost | cacti | Sleep   |  288 |             | NULL                                                                                       |
| 841463 | cactiuser | localhost | cacti | Sleep   |  295 |             | NULL                                                                                       |
| 841464 | cactiuser | localhost | cacti | Sleep   |  287 |             | NULL                                                                                       |
| 841467 | cactiuser | localhost | cacti | Sleep   |  125 |             | NULL                                                                                       |
| 841470 | cactiuser | localhost | cacti | Sleep   |  304 |             | NULL                                                                                       |
| 841471 | cactiuser | localhost | cacti | Sleep   |  304 |             | NULL                                                                                       |
| 841472 | cactiuser | localhost | cacti | Sleep   |  303 |             | NULL                                                                                       |
| 850932 | cactiuser | localhost | NULL  | Query   |    0 | NULL        | SHOW PROCESSLIST                                                                           |
| 850933 | cactiuser | localhost | cacti | Query   |    0 | cleaning up | delete from poller_output where local_da76' and rrd_name='traffic_in' and time='2009-08-31 |
| 850938 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850942 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850943 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850944 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850945 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850946 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850947 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850948 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850949 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850950 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850952 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850953 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850954 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850955 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850956 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850957 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850958 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850959 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850960 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850961 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850963 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850964 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850966 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850967 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850968 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850969 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850972 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850974 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850977 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850979 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850981 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850983 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850985 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 850987 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851001 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851009 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851011 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851015 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851017 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851032 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851033 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851035 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851036 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851040 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851044 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851045 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851046 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851047 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851048 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851049 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851050 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851051 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851052 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851053 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851055 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851056 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851058 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851059 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851060 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851061 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851062 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851063 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851064 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851065 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851066 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851067 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851068 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851070 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851071 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851072 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851073 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851074 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851075 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851076 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851078 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851079 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851081 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851082 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851083 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851084 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851085 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851086 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851087 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851088 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851089 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851090 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851091 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851092 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851093 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851095 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851096 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851097 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851098 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851099 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851100 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851102 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851105 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851106 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851107 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851108 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851110 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851111 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851113 | cactiuser | localhost | cacti | Sleep   |    5 |             | NULL                                                                                       |
| 851115 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851116 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851117 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851119 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851121 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851122 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851123 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851124 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851125 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851126 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851127 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851128 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851129 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851130 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851131 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851132 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851133 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851135 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851136 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851137 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851138 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851139 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851140 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851141 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851142 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851144 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851145 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851149 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851151 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851152 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851153 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851166 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851169 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851171 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851172 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851173 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851175 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851176 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851177 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851178 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851179 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851180 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851181 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851183 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851184 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851185 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851187 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851188 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851194 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851195 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851196 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851197 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851199 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851200 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851201 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851202 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851203 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851204 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851205 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851206 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851207 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851208 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851209 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851210 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851211 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851212 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851213 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851214 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851215 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851217 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851218 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851219 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851220 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851221 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851222 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851223 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851224 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851225 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851226 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851227 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851228 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851230 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851232 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851233 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                       |
| 851234 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851237 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851238 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851239 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851240 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851241 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851242 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851243 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851244 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851248 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851249 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851250 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851251 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851252 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851255 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851256 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851257 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851258 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                       |
| 851259 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851263 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851265 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851267 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851269 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851271 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851275 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851308 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851311 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851312 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851325 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851327 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851328 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851329 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851330 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851331 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851332 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851334 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851336 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851337 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851338 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851339 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                       |
| 851346 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851349 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851350 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851360 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851364 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851366 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851368 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851369 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851370 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851371 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851372 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851373 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851374 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851375 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851376 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851377 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851378 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851380 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851382 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851383 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851384 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851385 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851386 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851387 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851389 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851391 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851397 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851399 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851404 | cactiuser | localhost | cacti | Sleep   |    4 |             | NULL                                                                                       |
| 851411 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851413 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851414 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851416 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851417 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851418 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851419 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851420 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851421 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851422 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851423 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                       |
| 851424 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851425 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851426 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851427 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851428 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851429 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851430 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851431 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851432 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851433 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851434 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851435 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851437 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851438 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851440 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851446 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851463 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851464 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851467 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                       |
| 851470 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851472 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851485 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851505 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                       |
| 851506 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851507 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851516 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851518 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                       |
| 851525 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                                 |
| 851531 | cactiuser | localhost | cacti | Sleep   |    3 |             | NULL                                                                                                 |
| 851537 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851539 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851542 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851545 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851550 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851551 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851552 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851554 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851555 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851556 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851557 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851559 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851560 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851568 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851571 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851583 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851589 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851591 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851593 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851594 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851595 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851596 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851597 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851598 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851599 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851600 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851601 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851602 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851603 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851604 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851606 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851608 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851609 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851610 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851611 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851612 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851613 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851614 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851615 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851616 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851617 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851618 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
| 851619 | cactiuser | localhost | cacti | Sleep   |    2 |             | NULL                                                                                                 |
+--------+-----------+-----------+-------+---------+------+-------------+------------------------------------------------------------------------------------------------------+
361 rows in set (0.00 sec)



I've attempted to OPTIMIZE TABLE via phpmyadmin, not sure if that's the same..


Average file size on your RRD files would help as well....
Hopefully I did this alright..

Code: Select all

[cactiuser@cacti01 rra]$ ls -lh | gawk '{sum += $5; n++;} END {print sum/n;}'
412.377

I'll try to pull the process list a couple of more times when this is occuring to see if it gets any worse.


Thanks for all of your help Linegod. I appreciate it man.
User avatar
Linegod
Developer
Posts: 1626
Joined: Thu Feb 20, 2003 10:16 am
Location: Canada
Contact:

Post by Linegod »

It would look to me like some of the cacti mysql processes are hanging. How long do they stick around for? (ie: Process 772270 has been around for 300+ seconds)

Can you do a 'netstat -atunpo|grep 3306' as well?
--
Live fast, die young
You're sucking up my bandwidth.

J.P. Pasnak,CD
CCNA, LPIC-1
http://www.warpedsystems.sk.ca
xminos
Cacti User
Posts: 63
Joined: Wed Apr 26, 2006 8:10 am

Post by xminos »

Linegod wrote:It would look to me like some of the cacti mysql processes are hanging. How long do they stick around for? (ie: Process 772270 has been around for 300+ seconds)

Can you do a 'netstat -atunpo|grep 3306' as well?
Thanks for your help Linegod,

we continued to troubleshoot it and found there to be a hardware issue. Replaced and all is well.

Thanks again for your help.
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

I'd never noticed before, but that POS load average graph is labelled 'CPU percent' on the y axis which is even more wrong than the stack.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests