what is problem wiht my cacti

[cadoi]

hi all,

my cacti server has been problem for last two day. the

Code: Select all

 "Last Run Statistics	Time:298.7491 Method:spine Processes:16 Threads:15 Hosts:718 HostsPerProcess:45 DataSources:8401 RRDsProcessed:3900"

is higher Unusually.
o other day the last run statistics is about 20 to 80. but now it is 298

and my map has problem as in the picture


my server is IBM p5: 8 CPU, 8G RAM
the poller information and htop command display display my server and my poller setting

but the log show:

Code: Select all

PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
....
MDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1062', SQL:"INSERT INTO user_log (username,user_id,result,ip,time)
........................................

i still google and do following but didn't resolve.

[TheWitness]

You may be out of space in /var.

TheWitness

[cadoi]

Code: Select all

You may be out of space in /var. 

df -h

Code: Select all

]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1               24G  3.9G   19G  17% /
/dev/md5              102G  6.4G   90G   7% /data
/dev/md4              3.8G  102M  3.5G   3% /tmp
/dev/md2               99M   16M   78M  18% /boot
/dev/md0               67G  1.8G   62G   3% /var
tmpfs                 3.9G     0  3.9G   0% /dev/shm

[TheWitness]

Your problem is too many processes and threads. You are likely causing spine to crash due to lack of connections. You need to tone it down.

TheWitness

[cadoi]

hi

Code: Select all

Your problem is too many processes and threads. You are likely causing spine to crash due to lack of connections. You need to tone it down. 

i still change following step:

step1: fist i set the concurrent process: 8 (the second is 10) and modified other number
step2: max threads: 5 to 10
step3: php server:5 to 10

i did this and monitoring for two sequencys of poller time

but the problem not resolve.

what information you need to resolve my problem?

thanks for your support

[cadoi]

Hi TheWitness,

i reolve the problem of last runstatics: it is 20 to 70 now.
But the the log of weathermap still error as the picture show before.

i check my log:

Code: Select all

09/16/2010 09:20:20 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
09/16/2010 09:20:19 AM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating 

when i check with spine

Code: Select all

spine -V 5 1 845

it run only host id 240

Code: Select all

09/16/2010 09:25:30 AM - SPINE: Poller[0] Host[240] DS[1907] SCRIPT: perl /var/www/html/cacti/scripts/ping-latency.pl  x.x.x.x, output: min:3.113 avg:10.416 max:12.550 dev:2.567 loss:0
09/16/2010 09:25:30 AM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine thread)
You have new mail in /var/spool/mail/root
[root@SRV-CACTI-ACCESS lib]#

i searched on google.com with key word "SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine thread)" but can't resolve. can you help me


thanks

[cadoi]

Hi all,

I resolved my problem by: del host that has id 240 and add again.
I used spine to check host-id from 1 to 850 and check again when the error

Code: Select all

when i check with spine
Code:
spine -V 5 1 845


it run only host id 240

Code:
09/16/2010 09:25:30 AM - SPINE: Poller[0] Host[240] DS[1907] SCRIPT: perl /var/www/html/cacti/scripts/ping-latency.pl  x.x.x.x, output: min:3.113 avg:10.416 max:12.550 dev:2.567 loss:0
09/16/2010 09:25:30 AM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine thread)
You have new mail in /var/spool/mail/root
[root@SRV-CACTI-ACCESS lib]#

then i checked;

Code: Select all

spine -V 5 230 250 

it still error on host-id 240, then i deleted this host and tried again, it was Ok.

[TheWitness]

What version of spine? Tell me about the host.

TheWitness

[cadoi]

Code: Select all

What version of spine?

cacti 0.8.7e
spine 0.8.7 e

the host id 40 i deleted. then i readded again. the host-id is 849

Code: Select all

Tell me about the host

System:HUAWEI-3COM CORP. SNMP agent for QuidWay S6500

Code: Select all

trcert -U -p x.x.x.x

Code: Select all

tracert -U -p 161 x.x.x.x
traceroute to x.x.x.x 30 hops max, 40 byte packets
 1 x  0.169 ms  0.122 ms  0.088 ms
 2  y  0.565 ms  0.595 ms  0.677 ms
 3  z  0.258 ms  0.246 ms  0.235 ms
 4  t  ms  14.468 ms  14.641 ms
 5   m 1.638 ms  2.665 ms  3.643 ms
 6   n  16.300 ms  16.327 ms  16.311 ms

[TheWitness]

Please understand this:

max_connections (in mysql) > (threads + concurrent script servers + concurrent scripts) * processes + anything for the ui.

You must restart mysql after making the changes to the my.cnf file.

TheWitness

[cadoi]

Code: Select all

max_connections (in mysql) > (threads + concurrent script servers + concurrent scripts) * processes + anything for the ui

more /etc/my.cnf

Code: Select all

[mysqld]
max_connections = 12000
key_buffer = 16M
myisam_sort_buffer_size = 32M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1024
thread_cache_size = 286
interactive_timeout = 50
wait_timeout = 1000
..............................

mysql serve only for cacti and syslog-ng (now the syslog-ng receives from 5 hosts)

Code: Select all

You must restart mysql after making the changes to the my.cnf file.

after change anything i allready restart thier daemon server