Spine trouble?

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Developers, Moderators

Post Reply
User avatar
fmangeant
Cacti Guru User
Posts: 2345
Joined: Fri Sep 19, 2003 8:36 am
Location: Sophia-Antipolis, France
Contact:
Contact fmangeant

Post by fmangeant »

It seems you're using a 1 minute polling interval. Have you tried with a 5 minute one ?
[size=84]
[color=green]HOWTOs[/color] :
[list][*][url=http://forums.cacti.net/viewtopic.php?t=15353]Install and configure the Net-SNMP agent for Unix[/url]
[*][url=http://forums.cacti.net/viewtopic.php?t=26151]Install and configure the Net-SNMP agent for Windows[/url]
[*][url=http://forums.cacti.net/viewtopic.php?t=28175]Graph multiple servers using an SNMP proxy[/url][/list]
[color=green]Templates[/color] :
[list][*][url=http://forums.cacti.net/viewtopic.php?t=15412]Multiple CPU usage for Linux[/url]
[*][url=http://forums.cacti.net/viewtopic.php?p=125152]Memory & swap usage for Unix[/url][/list][/size]
Top
ame
Cacti User
Posts: 77
Joined: Fri Nov 25, 2005 6:15 am

Post by ame »

Hi,

it takes longer until the errors are appearing, but it is the same......

the ghost threads are increasing as well.....

micky:~# ps -ax |grep cacti
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
5397 ? S 0:00 /usr/bin/php -q /usr/share/cacti/site/script_server.php spine 0
5400 ? S 0:00 /usr/bin/php -q /usr/share/cacti/site/script_server.php spine 0
7242 ? S 0:00 /usr/bin/php -q /usr/share/cacti/site/script_server.php spine 0
7245 ? S 0:00 /usr/bin/php -q /usr/share/cacti/site/script_server.php spine 0
8131 pts/0 S+ 0:00 grep cacti

all 1minute graphs aren´t display anything....and all 20 minutes I get
01/24/2008 02:45:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.

Alex
Top
ame
Cacti User
Posts: 77
Joined: Fri Nov 25, 2005 6:15 am

Post by ame »

Witness, do you have any idea regarding this problem?

Alex
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Post your technical support page. What is your OS and version "uname -a".

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Also, when these threads get stuck, perform a truss on one of them and post the output.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
ame
Cacti User
Posts: 77
Joined: Fri Nov 25, 2005 6:15 am

Post by ame »

micky:~# uname -a
Linux micky 2.6.18-5-686 #1 SMP Mon Dec 24 16:41:07 UTC 2007 i686 GNU/Linux

Fresh install of debian etch with cacti .87a and fresh compiled spine.

what do you mean with:
"Also, when these threads get stuck, perform a truss on one of them and post the output."
What should i do?

Alex
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Do a "man truss" and you will get your instructions.

Thanks,

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
ame
Cacti User
Posts: 77
Joined: Fri Nov 25, 2005 6:15 am

Post by ame »

that package doesn´t exist for debian....cannot install "truss"....
I have searched in the package database of debain and found there nothing as well....

I am confused.....
Last edited by ame on Fri Jan 25, 2008 7:54 am, edited 1 time in total.
Top
User avatar
fmangeant
Cacti Guru User
Posts: 2345
Joined: Fri Sep 19, 2003 8:36 am
Location: Sophia-Antipolis, France
Contact:
Contact fmangeant

Post by fmangeant »

You can also use strace :

Code: Select all

# strace -p pid
[size=84]
[color=green]HOWTOs[/color] :
[list][*][url=http://forums.cacti.net/viewtopic.php?t=15353]Install and configure the Net-SNMP agent for Unix[/url]
[*][url=http://forums.cacti.net/viewtopic.php?t=26151]Install and configure the Net-SNMP agent for Windows[/url]
[*][url=http://forums.cacti.net/viewtopic.php?t=28175]Graph multiple servers using an SNMP proxy[/url][/list]
[color=green]Templates[/color] :
[list][*][url=http://forums.cacti.net/viewtopic.php?t=15412]Multiple CPU usage for Linux[/url]
[*][url=http://forums.cacti.net/viewtopic.php?p=125152]Memory & swap usage for Unix[/url][/list][/size]
Top
ame
Cacti User
Posts: 77
Joined: Fri Nov 25, 2005 6:15 am

Post by ame »

that´s the output.....it hangs..after 10 minutes I made ctrl-c


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
micky:~# ps -ax |grep cacti
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
8287 ? S 0:00 /usr/bin/php -q /usr/share/cacti/site/script_server.php spine 0
8292 ? S 0:00 /usr/bin/php -q /usr/share/cacti/site/script_server.php spine 0
20603 ? Ss 0:00 /bin/sh -c /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log
20604 ? S 0:01 /usr/bin/php -q /usr/share/cacti/site/poller.php
20841 pts/0 S+ 0:00 grep cacti

micky:~# strace -p 8292
Process 8287 attached - interrupt to quit
read(10, <unfinished ...>
Process 8287 detached
micky:~# strace -p 8292
Process 8292 attached - interrupt to quit
read(12, <unfinished ...>
Process 8292 detached
micky:~#

Alex
Top
ame
Cacti User
Posts: 77
Joined: Fri Nov 25, 2005 6:15 am

Post by ame »

Hi,

I have took a look when this happens. Everytime when I get these error messages in the cacti log:

02/20/2008 07:00:26 AM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1062', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (1581,'hdd_used','2008-02-20 07:00:24','22072528896'),(1581,'hdd_total','2008-02-20 07:00:24','48417910784'),(1582,'hdd_total','2008-02-20 07:00:24','2593718272'),(1582,'hdd_used','2008-02-20 07:00:24','408158208'),(1586,'traffic_in','2008-02-20 07:00:24','1150309732'),(1586,'traffic_out','2008-02-20 07:00:24','516726394')'

then the spine processes are dying and resides as zombies. After some days these zombies has allocated all of the memory of my cacti machine.....

Any ideas?

Alex
Top
ame
Cacti User
Posts: 77
Joined: Fri Nov 25, 2005 6:15 am

Post by ame »

Hi,

I found out what the problem is: when I leave the threads parameter at "1" all is running fine - at the point I increase threads to 2 and more I get massive error messages and many zombie threads.....

Only for your interest if anyone has got the same problems....

Alex
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Post by TheWitness »

Alex,

Please use the SVN version, and let me know if the problem goes away. There have been a few critical changes.

However, threading issues in spine for Linux have not been seen for quite some time.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
ben_c
Cacti User
Posts: 203
Joined: Mon May 14, 2007 8:12 pm
Location: Melbourne, Australia.

Post by ben_c »

Any update on this guys?

I am getting the same behavior, will run fine for about half an hour then spine will crash out

03/26/2008 04:15:04 PM - SPINE: Poller[0] FATAL: Connection Failed: Too many connections (Spine thread)
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:15:04 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
03/26/2008 04:20:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.

I tried bumping the mysql max conns up to 1000 but with no change. Spine 0.8.7a currently.

Linux xxx 2.6.9-67.0.1.ELsmp #1 SMP Fri Nov 30 11:57:43 EST 2007 x86_64 x86_64 x86_64 GNU/Linux
Top
User avatar
burn26
Posts: 3
Joined: Fri Dec 12, 2008 7:12 pm

help

Post by burn26 »

03/23/2009 03:51:01 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:03 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:03 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:03 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:51:14 PM - SYSTEM STATS: Time:12.5990 Method:spine Processes:2 Threads:50 Hosts:134 HostsPerProcess:67 DataSources:1404 RRDsProcessed:626
03/23/2009 03:52:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:02 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:03 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:03 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:03 PM - SPINE: Poller[0] ERROR: SS[999] Script Server did not start properly return message was: ''
03/23/2009 03:52:08 PM - SYSTEM STATS: Time:6.0390 Method:spine Processes:2 Threads:50 Hosts:134 HostsPerProcess:67 DataSources:1411 RRDsProcessed:622

Hostname: smtp - OS: Linux 2.6.18-92.1.22.el5.centos.plus/x86_64 - Distro: CentOS 5.2 - CPU: 2 x Intel Xeon (3800.276 MHz) - Processes: 107 - Uptime: 27d 2h 47m - Users: 1 - Load Average: 0.06 - Memory Usage: 1128.98MB/11954.46MB (9.44%) - Disk Usage: 10.81GB/69.65GB (15.52%)

Code: Select all

Technical Support
General Information
Date 	Mon, 23 Mar 2009 15:47:05 -0600
Cacti Version 	0.8.7d
Cacti OS 	unix
SNMP Version 	NET-SNMP version: 5.3.1
RRDTool Version 	RRDTool 1.2.x
Hosts 	150
Graphs 	2368
Data Sources 	Script/Command: 133
SNMP: 38
SNMP Query: 2209
Script Query: 2
Script Query - Script Server: 9
Total: 2391
Poller Information
Interval 	60
Type 	spine
Items 	Action[0]: 4612
Action[1]: 130
Action[2]: 12
Total: 4754
Concurrent Processes 	2
Max Threads 	50
PHP Servers 	5
Script Timeout 	45
Max OID 	10
Last Run Statistics 	Time:12.7683 Method:spine Processes:1 Threads:100 Hosts:134 HostsPerProcess:134 DataSources:1404 RRDsProcessed:626
PHP Information
PHP Version 	5.1.6
PHP OS 	Linux
PHP uname 	Linux smtp.tsagconnect.net 2.6.18-92.1.22.el5.centos.plus #1 SMP Wed Dec 17 10:49:19 EST 2008 x86_64
PHP SNMP 	Installed
max_execution_time 	60
memory_limit 	128M
Top
Post Reply

Return to “Help: Linux/Unix Specific”

Who is online

Users browsing this forum: No registered users and 1 guest