URGENT: 75% of Datasources/Graphs no longer graphed!

[Deviloper]

Hello,
I noticed that only a fraction of my graphs are getting data.
Most of my interface statistics are empty.

I check and crosscheck the problem on wire.
I am sure it is not an network error. (I sniffered and read the dumps by myself. )

A SNMP-Response takes 100ms - 200ms.
Timeout is set to 3500, 4000 and in some cases even 8000ms.

I tried:
Rebuilding the poller cache.
Disabling Boost.
Uninstalling Boost.
Increasing timeouts to 8000ms.
Setting Max. OID to 20.
Setting Max. OID to 0. (It looks like spine ignores this. It still uses getbulks after disabling bulkwalks. That is what TCP-Dump shows, not an idea of me.)
I tested "Verbose Query" -> This works fine.
I check snmpwalk and snmpget in the shell and it works.

I disabled spine and tried polling with cmd.php:
##################################################
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[197] DS[68337] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8899] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8899] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8898] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8898] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8898] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8898] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8897] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8897] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8897] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[169] DS[41990] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8897] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[169] DS[41990] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8896] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[169] DS[41990] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8896] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[169] DS[41990] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8896] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8896] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8895] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8895] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8895] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8895] WARNING: Result from SNMP not valid. Partial Result: U
02/03/2011 06:20:50 PM - CMDPHP: Poller[0] Host[70] DS[8894] WARNING: Result from SNMP not valid. Partial Result: U
############################################################################################

Some Graphs on a single Device, even of the same Graph- and Data-Template can be fine, but all others are empty!

See my other post on this :
http://forums.cacti.net/viewtopic.php?f ... 38#p206938

I would post you more information, but the forum is a little bit limited.
############################################################################################

Technical Support
General Information
Date Thu, 03 Feb 2011 18:56:15 +0100
Cacti Version 0.8.7g
Cacti OS unix
SNMP Version NET-SNMP version: 5.3.0.1
RRDTool Version RRDTool 1.2.x
Hosts 368
Graphs 89598
Data Sources SNMP: 328
SNMP Query: 84875
Script Query - Script Server: 4590
Total: 89793
Poller Information
Interval 300
Type spine
Items Action[0]: 183098
Action[2]: 4882
Total: 187980
Concurrent Processes 10
Max Threads 90
PHP Servers 9
Script Timeout 298
Max OID 100
Last Run Statistics Time:56.4320 Method:spine Processes:10 Threads:90 Hosts:311 HostsPerProcess:32 DataSources:187980 RRDsProcessed:40905
PHP Information
PHP Version 5.2.5
PHP OS Linux
PHP uname Linux BLABLABLA 2.6.16.60-0.67.1-smp #1 SMP Thu Aug 5 10:54:46 UTC 2010 x86_64
PHP SNMP Installed
max_execution_time 300
memory_limit 1024M
########################################################################################################

[Deviloper]

I also tried repairing the templates using repair_templates.php.

/usr/share/cacti/cli # php -q ./repair_templates.php --execute
NOTE: Repairing All Duplicated Templates
NOTE: Repairing Data Templates
NOTE: No Damaged Data Templates Found
NOTE: Repairing Graph Templates
NOTE: No Damaged Graph Templates Found

No damaged templates where found.

[Deviloper]

I tried setting my host_snmp_query method from 1 to 3.

select * from host_snmp_query where reindex_method = 1;
UPDATE host_snmp_query SET reindex_method = 3 WHERE reindex_method = 1;

than I tried to reindex one of the problematic devices:
php -q poller_reindex_hosts.php --id=153

finally rebuild the poller-cache.
php -q rebuild_poller_cache.php

Removing a problematic device by deleting it.
Recreated it with Console->Devices->ADD.

No Success.
Any Idea?

[Deviloper]

Can somebody please verify that the Spine-Debug-Output for "Insert poller_output" is truncated by default after 1000 Charakters.
##############################################################
DEVDBG: SQL:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (91294,'errors_in','2011-02-08 09:18:48','U'),(91294,'discards_in','2011-02-08 09:18:48','U'),(91294,'errors_out','2011-02-08 09:18:48','U'),(91294,'discards_out','2011-02-08 09:18:48','U'),(91293,'errors_in','2011-02-08 09:18:48','U'),(91293,'discards_in','2011-02-08 09:18:48','U'),(91293,'errors_out','2011-02-08 09:18:48','U'),(91293,'discards_out','2011-02-08 09:18:48','U'),(91292,'errors_in','2011-02-08 09:18:48','U'),(91292,'discards_in','2011-02-08 09:18:48','U'),(91292,'errors_out','2011-02-08 09:18:48','U'),(91292,'discards_out','2011-02-08 09:18:48','U'),(91291,'errors_in','2011-02-08 09:18:48','U'),(91291,'discards_in','2011-02-08 09:18:48','U'),(91291,'errors_out','2011-02-08 09:18:48','U'),(91291,'discards_out','2011-02-08 09:18:48','U'),(91290,'errors_in','2011-02-08 09:18:48','U'),(91290,'discards_in','2011-02-08 09:18:48','U'),(91290,'errors_out','2011-02-08 09:18:48','U'),(91290,'discards_out','2011-02-08 09:18:48','U'),
###############################################################

[Deviloper]

Developers and Moderators on Hollyday?