Random local queries fail.. sometimes

shamsway · Post by **shamsway** » Wed Aug 26, 2009 10:54 am

I have an issue where the queries for my local server fail on a somewhat random basis.

- I am using spine and have tried multiple different versions. I am currently running the CVS.
- Running the scripts from the command line (e.g. '/usr/bin/perl /var/data/cacti/scripts/loadavg_multi.pl') always seems to return a result. the log is indicating taht there is an empty result.
- I have the full path to perl in all the scripts and other places.
- running spine with debugging doesn't seem to provide anything particularly helpful.

i'm looking for direction as far as additional troubleshooting steps. the scripts do regularly work.. just not every time.

examples:

Code: Select all

08/26/2009 11:50:20 AM - WEATHERMAP: Poller[0] [Map 3] SJHC-Large: ReadData: LINK wa2datacore2-jh1datacore1, target: /var/data/cacti/rra/wa2-data-core-2_traffic_in_1378.rrd on config line 223 had no valid data, according to WeatherMapDataSource_rrd
08/26/2009 11:50:20 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 104.158287824% to 100%
08/26/2009 11:50:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 104.158287824% to 100%
08/26/2009 11:50:19 AM - SYSTEM THOLD STATS: Time:0.0227 Tholds:6 Hosts:0
08/26/2009 11:50:19 AM - SYSTEM STATS: Time:18.0658 Method:spine Processes:1 Threads:10 Hosts:85 HostsPerProcess:85 DataSources:6468 RRDsProcessed:3429
08/26/2009 11:50:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/loadavg_multi.pl'
08/26/2009 11:50:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl SwapFree:'
08/26/2009 11:50:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl MemFree:'
08/26/2009 11:45:19 AM - WEATHERMAP: Poller[0] [Map 3] SJHC-Large: ReadData: LINK wa2datacore2-jh1datacore1, target: /var/data/cacti/rra/wa2-data-core-2_traffic_in_1378.rrd on config line 223 had no valid data, according to WeatherMapDataSource_rrd
08/26/2009 11:45:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 107.833647352% to 100%
08/26/2009 11:45:18 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 107.833647352% to 100%
08/26/2009 11:45:18 AM - SYSTEM THOLD STATS: Time:0.0230 Tholds:6 Hosts:0
08/26/2009 11:45:18 AM - SYSTEM STATS: Time:16.5551 Method:spine Processes:1 Threads:10 Hosts:85 HostsPerProcess:85 DataSources:6468 RRDsProcessed:3429
08/26/2009 11:45:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/loadavg_multi.pl'
08/26/2009 11:45:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl SwapFree:'
08/26/2009 11:45:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl MemFree:'
08/26/2009 11:40:20 AM - WEATHERMAP: Poller[0] [Map 3] SJHC-Large: ReadData: LINK wa2datacore2-jh1datacore1, target: /var/data/cacti/rra/wa2-data-core-2_traffic_in_1378.rrd on config line 223 had no valid data, according to WeatherMapDataSource_rrd
08/26/2009 11:40:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 107.493908656% to 100%
08/26/2009 11:40:18 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 107.493908656% to 100%
08/26/2009 11:40:18 AM - SYSTEM THOLD STATS: Time:0.0284 Tholds:6 Hosts:0
08/26/2009 11:40:18 AM - SYSTEM STATS: Time:17.0998 Method:spine Processes:1 Threads:10 Hosts:85 HostsPerProcess:85 DataSources:6468 RRDsProcessed:3429
08/26/2009 11:40:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/loadavg_multi.pl'
08/26/2009 11:40:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl SwapFree:'
08/26/2009 11:40:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl MemFree:'
08/26/2009 11:35:22 AM - WEATHERMAP: Poller[0] [Map 3] SJHC-Large: ReadData: LINK wa2datacore2-jh1datacore1, target: /var/data/cacti/rra/wa2-data-core-2_traffic_in_1378.rrd on config line 223 had no valid data, according to WeatherMapDataSource_rrd
08/26/2009 11:35:20 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 100.069477376% to 100%
08/26/2009 11:35:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 100.069477376% to 100%
08/26/2009 11:35:19 AM - SYSTEM THOLD STATS: Time:0.0180 Tholds:6 Hosts:0
08/26/2009 11:35:19 AM - SYSTEM STATS: Time:18.2257 Method:spine Processes:1 Threads:10 Hosts:85 HostsPerProcess:85 DataSources:6468 RRDsProcessed:3429
08/26/2009 11:35:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/query_unix_partitions.pl get used /dev/sdb1'
08/26/2009 11:35:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/unix_processes.pl'
08/26/2009 11:35:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/unix_users.pl '
08/26/2009 11:35:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/loadavg_multi.pl'
08/26/2009 11:35:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl SwapFree:'
08/26/2009 11:35:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl MemFree:'
08/26/2009 11:30:25 AM - WEATHERMAP: Poller[0] [Map 3] SJHC-Large: ReadData: LINK wa2datacore2-jh1datacore1, target: /var/data/cacti/rra/wa2-data-core-2_traffic_in_1378.rrd on config line 223 had no valid data, according to WeatherMapDataSource_rrd
08/26/2009 11:30:25 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wh0phonecore1-wa2datagstack6 101.265446384% to 100%
08/26/2009 11:30:24 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wh0phonecore1-wa2datagstack6 101.265446384% to 100%
08/26/2009 11:30:24 AM - SYSTEM THOLD STATS: Time:0.0190 Tholds:6 Hosts:0
08/26/2009 11:30:24 AM - SYSTEM STATS: Time:22.4369 Method:spine Processes:1 Threads:10 Hosts:85 HostsPerProcess:85 DataSources:6468 RRDsProcessed:3429
08/26/2009 11:30:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl SwapFree:'
08/26/2009 11:30:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl MemFree:'
08/26/2009 11:25:19 AM - WEATHERMAP: Poller[0] [Map 3] SJHC-Large: ReadData: LINK wa2datacore2-jh1datacore1, target: /var/data/cacti/rra/wa2-data-core-2_traffic_in_1378.rrd on config line 223 had no valid data, according to WeatherMapDataSource_rrd
08/26/2009 11:25:18 AM - SYSTEM THOLD STATS: Time:0.0215 Tholds:6 Hosts:0
08/26/2009 11:25:17 AM - SYSTEM STATS: Time:16.2887 Method:spine Processes:1 Threads:10 Hosts:85 HostsPerProcess:85 DataSources:6468 RRDsProcessed:3429
08/26/2009 11:25:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl SwapFree:'
08/26/2009 11:25:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl MemFree:'
08/26/2009 11:20:20 AM - WEATHERMAP: Poller[0] [Map 3] SJHC-Large: ReadData: LINK wa2datacore2-jh1datacore1, target: /var/data/cacti/rra/wa2-data-core-2_traffic_in_1378.rrd on config line 223 had no valid data, according to WeatherMapDataSource_rrd
08/26/2009 11:20:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wh0phonecore1-wa2datagstack6 101.077781432% to 100%
08/26/2009 11:20:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 100.030891696% to 100%
08/26/2009 11:20:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wh0phonecore1-wa2datagstack6 101.077781432% to 100%
08/26/2009 11:20:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 100.030891696% to 100%
08/26/2009 11:20:18 AM - SYSTEM THOLD STATS: Time:0.0184 Tholds:6 Hosts:0
08/26/2009 11:20:18 AM - SYSTEM STATS: Time:16.7246 Method:spine Processes:1 Threads:10 Hosts:85 HostsPerProcess:85 DataSources:6468 RRDsProcessed:3429
08/26/2009 11:15:20 AM - WEATHERMAP: Poller[0] [Map 3] SJHC-Large: ReadData: LINK wa2datacore2-jh1datacore1, target: /var/data/cacti/rra/wa2-data-core-2_traffic_in_1378.rrd on config line 223 had no valid data, according to WeatherMapDataSource_rrd
08/26/2009 11:15:19 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 100.722268928% to 100%
08/26/2009 11:15:18 AM - WEATHERMAP: Poller[0] [Map 4] SJH Lan: ColourFromPercent: Clipped wa2datacore1-wa2datagstack2 100.722268928% to 100%
08/26/2009 11:15:18 AM - SYSTEM THOLD STATS: Time:0.0175 Tholds:6 Hosts:0
08/26/2009 11:15:18 AM - SYSTEM STATS: Time:16.8962 Method:spine Processes:1 Threads:10 Hosts:85 HostsPerProcess:85 DataSources:6468 RRDsProcessed:3429
08/26/2009 11:15:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/loadavg_multi.pl'
08/26/2009 11:15:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl SwapFree:'
08/26/2009 11:15:03 AM - SPINE: Poller[0] Host[1] ERROR: Empty result [127.0.0.1]: '/usr/bin/perl /var/data/cacti/scripts/linux_memory.pl MemFree:'

HALP!

Thanks,
Shamsway

shamsway · Post by **shamsway** » Tue Sep 01, 2009 2:12 pm

I just disabled monitoring my localhost. i will probably switch it to using SNMP instead of the scripts. oh well...

i've got a larger problem that i need to make another post about anyways

Post by **Linegod** » Sat Sep 05, 2009 4:55 pm

When testing the command at the console, are you running it as the same user that executes the cronjob for poller.php?

shamsway · Post by **shamsway** » Tue Sep 08, 2009 9:45 am

yes... ? it's been a bit since i tried it.

i will try again as the cacti user, just to confirm, and post the results.

shamsway · Post by **shamsway** » Wed Sep 09, 2009 10:30 am

Here are the results from some recent failures in my log...

Code: Select all

root@cacti:/etc# su - www-data
$ /usr/bin/perl /var/data/cacti/scripts/query_unix_partitions.pl get available /dev/sda1
44885125
$ /usr/bin/perl /var/data/cacti/scripts/query_unix_partitions.pl get used /dev/sda1
74627437
$ /usr/bin/perl /var/data/cacti/scripts/linux_memory.pl MemFree:
174180
$ /usr/bin/perl /var/data/cacti/scripts/linux_memory.pl SwapFree:
416132
$ /usr/bin/perl /var/data/cacti/scripts/loadavg_multi.pl
1min:0.26 5min:0.40 10min:0.46
$ /usr/bin/perl /var/data/cacti/scripts/unix_users.pl
3
$ /usr/bin/perl /var/data/cacti/scripts/unix_processes.pl
123
$ /usr/bin/perl /var/data/cacti/scripts/query_unix_partitions.pl get available /dev/sdb1
7931496
$ /usr/bin/perl /var/data/cacti/scripts/query_unix_partitions.pl get used /dev/sdb1
1862220
$

they all seem to return results when i try manually...

Post by **Linegod** » Wed Sep 09, 2009 11:31 am

Perhaps the box is simply overloaded (post cpu graphs?).

You could try switching all the graphs to the SNMP version provided by ucd/net-snmp scripts, as they should require less process time.

If that doesn't help, post the output of 'Techinical Support'....

Cacti

Random local queries fail.. sometimes

Random local queries fail.. sometimes

Who is online