Random gaps in 0.8.7

[spirre]

Hello,

I am using cacti 0.8.7 with spine 0.8.7a, and it's working fine, except some seemingly random gaps i some of the graphs.
Every polling interval i detect in the cacti.log that 1-4 datasources that gets
11/14/2007 10:35:20 AM - SPINE: Poller[0] Host[25] DS[3559] WARNING: Result from SCRIPT not valid. Partial Result: ...

And this is causing the gaps. If I run the poller with cmd.php there are no gaps.
And if I run spine manually in the host with the failing DS, I don't get any errors.

11/14/2007 10:36:01 AM - SYSTEM STATS: Time:59.7531 Method:spine Processes:4 Threads:30 Hosts:103 HostsPerProcess:26 DataSources:3182 RRDsProcessed:2409


What can be causing this?

[BIGHard]

have the same problem, occasionally spine runs over 60 secs, sometimes up to over 200. :/

[TheWitness]

There is a bug in 0.8.7 that will be corrected in 0.8.7a. What you have to do is edit poller.php and set the following "$cron_interval = 300;". Elsewhere, where it is set, you have to remove that code.

TheWitness

[spirre]

I am still experiencing problems with the gaps. Hosts and DS still seem very random. They are all using the same script though.
I will try to investigate the script, but it works outside poller.php.

[spirre]

simplified the script, now it works great.

[BIGHard]

how about one minute polling?

[BIGHard]

latest gap:

Code: Select all

11/15/2007 11:56:22 PM - SYSTEM STATS: Time:21.0026 Method:spine Processes:2 Threads:80 Hosts:345 HostsPerProcess:173 DataSources:2817 RRDsProcessed:2451
11/15/2007 11:56:22 PM - SYSTEM STATS: Time:21.0848 Method:spine Processes:2 Threads:80 Hosts:345 HostsPerProcess:173 DataSources:2817 RRDsProcessed:2554
11/15/2007 11:57:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty.  Potential Data Source Issues for Data Sources: hdd_used(DS[231]), hdd_total(DS[234]), (DS[5420]), (DS[5421]), (DS[5422]), (DS[5424]), hdd_used(DS[5711])
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/16/2007 12:00:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty.  Potential Data Source Issues for Data Sources: hdd_total(DS[231]), hdd_used(DS[232])
11/16/2007 12:00:25 AM - SYSTEM STATS: Time:24.1262 Method:spine Processes:2 Threads:80 Hosts:345 HostsPerProcess:173 DataSources:2779 RRDsProcessed:1608
11/16/2007 12:00:25 AM - SYSTEM STATS: Time:203.9214 Method:spine Processes:2 Threads:80 Hosts:345 HostsPerProcess:173 DataSources:2817 RRDsProcessed:2102

poller runs every minute - it did start at 11:57:01, crashed at 11:57:12 and the next one didn't runn up to 12:00:01 - not cron's fault, spine issue. and look at the 12:00:25 stats time. WTF?

[TheWitness]

11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
11/15/2007 11:57:12 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating

Either spine or php is segfaulting. Issue the following command:

Code: Select all

locate mysqlclient_r

Then in that directory, do the following:

Code: Select all

ls -l mysqlclient*

Post the output. In addition, please post the summary portion of your technical support page. Give the spine user permission to write core files to the spine bin directory, check /tmp for core files and the cacti web root for them as well. Make sure the poller user can write core files into the cacti web root as well is the case it is php.

TheWitness

[TheWitness]

Once you locate the core file, we will use gdb to find out where spine is bombing.

TheWitness

[BIGHard]

Code: Select all

-rw-r--r--  1 root root 1646376 Apr  5  2007 libmysqlclient.a
-rw-r--r--  1 root root 1657746 Apr  5  2007 libmysqlclient_r.a
lrwxrwxrwx  1 root root      26 Jul 29 23:17 libmysqlclient_r.so -> libmysqlclient_r.so.14.0.0
lrwxrwxrwx  1 root root      26 Mar  3  2007 libmysqlclient_r.so.10 -> libmysqlclient_r.so.10.0.0
-rwxr-xr-x  1 root root  240648 Feb 18  2005 libmysqlclient_r.so.10.0.0
lrwxrwxrwx  1 root root      26 Jul 29 23:17 libmysqlclient_r.so.14 -> libmysqlclient_r.so.14.0.0
-rwxr-xr-x  1 root root 1354032 Apr  5  2007 libmysqlclient_r.so.14.0.0
lrwxrwxrwx  1 root root      24 Jul 29 23:17 libmysqlclient.so -> libmysqlclient.so.14.0.0
lrwxrwxrwx  1 root root      24 Mar  3  2007 libmysqlclient.so.10 -> libmysqlclient.so.10.0.0
-rwxr-xr-x  1 root root  235680 Feb 18  2005 libmysqlclient.so.10.0.0
lrwxrwxrwx  1 root root      24 Jul 29 23:17 libmysqlclient.so.14 -> libmysqlclient.so.14.0.0
-rwxr-xr-x  1 root root 1347016 Apr  5  2007 libmysqlclient.so.14.0.0

Code: Select all

Interval  	60
Type 	spine
Items 	Action[0]: 3785
Action[1]: 366
Action[2]: 2331
Total: 6482
Concurrent Processes 	2
Max Threads 	80
PHP Servers 	10
Script Timeout 	5
Max OID 	50
Last Run Statistics 	Time:24.0765 Method:spine Processes:2 Threads:80 Hosts:345 HostsPerProcess:173 DataSources:2586 RRDsProcessed:1675

[TheWitness]

Please reduce your thread count to something like 10 to 20. Let me know when you have that core file.

TheWitness

[BIGHard]

reducing didn't help, i've increased it to see if it would change anything.

still hunting for core

[TheWitness]

If you reduce the thread count to 1 does the problem go away?

TheWitness

[BIGHard]

with thread count 1 spine barely manages do poll all the hosts:

Code: Select all

 11/19/2007 07:34:55 AM - SYSTEM STATS: Time:54.0383 Method:spine Processes:2 Threads:1 Hosts:348 HostsPerProcess:174 DataSources:2836 RRDsProcessed:960
11/19/2007 07:34:55 AM - SYSTEM STATS: Time:53.3689 Method:spine Processes:2 Threads:1 Hosts:348 HostsPerProcess:174 DataSources:2460 RRDsProcessed:1770

and i can see errors like:

Code: Select all

11/19/2007 07:34:48 AM - CACTID: Poller[0] ERROR: Problem with MySQL: 'Duplicate entry '3357-traffic_in-2007-11-19 07:34:48' for key 1'

[BIGHard]

i also get:

Code: Select all

 11/19/2007 07:35:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
11/19/2007 07:35:56 AM - CACTID: Poller[0] ERROR: Cactid Timed Out While Processing Hosts Internal
11/19/2007 07:35:55 AM - PHPSVR: Poller[0] Maximum runtime of 52 seconds exceeded for the Script Server. Exiting.
11/19/2007 07:35:55 AM - PHPSVR: Poller[0] Maximum runtime of 52 seconds exceeded for the Script Server. Exiting.
11/19/2007 07:35:55 AM - PHPSVR: Poller[0] Maximum runtime of 52 seconds exceeded for the Script Server. Exiting.
11/19/2007 07:35:55 AM - PHPSVR: Poller[0] Maximum runtime of 52 seconds exceeded for the Script Server. Exiting.
11/19/2007 07:35:55 AM - PHPSVR: Poller[0] Maximum runtime of 52 seconds exceeded for the Script Server. Exiting.
11/19/2007 07:35:55 AM - PHPSVR: Poller[0] Maximum runtime of 52 seconds exceeded for the Script Server. Exiting.