Gaps in graphs getting progressively worse

[macdaddy]

I'm having severe trouble with gaps in my graphs. I've been searching the forums and found several threads about it. Most, however, center around lengthening the timeout for SNMP. Not all of the gaps appear in graphs that utilize SNMP. For example the most recent problem is with graphs that are using the Advanced Ping script which doesn't use SNMP at all. I'm attaching example of gaps in I/O graphs and non-SNMP graphs.

Suggestions? The box isn't that loaded down. I only have 99 devices and not all are up. That amounts to 2189 graphs total. Do I need to switch to Spine by chance? I'm running 0.8.7d. I raised the timeout on the device associated with the attached I/O graph to 5000 and it's still giving me gaps. I'm seeing an assortment of errors in the log and I'm not sure where to go with them. Here's an example:

01/27/2010 11:06:48 AM - CMDPHP: Poller[0] Host[45] DS[2037] WARNING: Result from SNMP not valid. Partial Result: No Such Instance cur
01/27/2010 11:06:48 AM - CMDPHP: Poller[0] Host[46] DS[765] WARNING: Result from SNMP not valid. Partial Result: No Such Instance cur
01/27/2010 11:06:48 AM - CMDPHP: Poller[0] Host[46] DS[766] WARNING: Result from SNMP not valid. Partial Result: No Such Instance cur
01/27/2010 11:06:48 AM - CMDPHP: Poller[0] Host[84] DS[2102] WARNING: Result from SNMP not valid. Partial Result: No Such Object avail
01/27/2010 11:06:48 AM - CMDPHP: Poller[0] Host[84] DS[2102] WARNING: Result from SNMP not valid. Partial Result: No Such Object avail

I turned on debugging and got some more data:


01/27/2010 11:08:58 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'top_header'"
01/27/2010 11:08:58 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'page_title'"
01/27/2010 11:08:58 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'page_head'"
01/27/2010 11:08:58 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'body_style'"
01/27/2010 11:08:58 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'top_header_tabs'"
01/27/2010 11:08:58 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'draw_navigation_text'"
01/27/2010 11:08:58 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_arrays'"
01/27/2010 11:08:58 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'top_header'"
01/27/2010 11:09:16 AM - PHPSVR: Poller[0] Maximum runtime of 292 seconds exceeded for the Script Server. Exiting.
01/27/2010 11:09:16 AM - CMDPHP: Poller[0] Host[112] DS[2526] WARNING: Result from SERVER not valid. Partial Result: 01/27/2010 11:09:16
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_arrays'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_settings'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'valid_host_fields'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_form'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_insert'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'poller_top'"
01/27/2010 11:10:01 AM - POLLER: Poller[0] NOTE: Poller Int: '300', Cron Int: '300', Time Since Last: '300', Max Runtime '298', Poller Runs: '1'
01/27/2010 11:10:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 151, Data Sources: unicast_in(DS[2368]), unicast_out(DS[2368]), unicast_in(DS[2369]), unicast_out(DS[2369]), traffic_in(DS[2400]), traffic_out(DS[2400]), traffic_in(DS[2401]), traffic_out(DS[2401]), traffic_in(DS[2402]), traffic_out(DS[2402]), traffic_in(DS[2403]), traffic_out(DS[2403]), traffic_in(DS[2404]), traffic_out(DS[2404]), discards_in(DS[2405]), discards_out(DS[2405]), errors_in(DS[2405]), errors_out(DS[2405]), discards_in(DS[2406]), discards_out(DS[2406]), errors_in(DS[2406]), Additional Issues Remain. Only showing first 20
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'poller_command_args'"
01/27/2010 11:10:01 AM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q "/var/www/noc.domain.tld/html/cacti-0.8.7d/cmd.php" 0 46]
01/27/2010 11:10:01 AM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q "/var/www/noc.domain.tld/html/cacti-0.8.7d/cmd.php" 47 112]
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_arrays'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_settings'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'valid_host_fields'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_form'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_insert'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_arrays'"
01/27/2010 11:10:01 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_settings'"
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'valid_host_fields'"
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_form'"
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_insert'"
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_arrays'"
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_settings'"
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'valid_host_fields'"
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_form'"
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1146', SQL:"SELECT name, file, function FROM plugin_hooks WHERE status = 1 AND hook = 'config_insert'"
01/27/2010 11:10:02 AM - PHPSVR: Poller[0] DEBUG: SERVER: cmd
01/27/2010 11:10:02 AM - PHPSVR: Poller[0] DEBUG: GETCWD: /home/cacti
01/27/2010 11:10:02 AM - PHPSVR: Poller[0] DEBUG: DIRNAM: /var/www/noc.domain.tld/html/cacti-0.8.7d
01/27/2010 11:10:02 AM - PHPSVR: Poller[0] DEBUG: FILENM: /var/www/noc.domain.tld/html/cacti-0.8.7d/script_server.php
01/27/2010 11:10:02 AM - PHPSVR: Poller[0] PHP Script Server has Started - Parent is cmd
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] PHP Script Server Started Properly
01/27/2010 11:10:02 AM - POLLER: Poller[0] WARNING: Poller[0] Host[1] SNMP Time was not numeric
01/27/2010 11:10:02 AM - CMDPHP: Poller[0] Host[1] SNMP: Device does not require SNMP

Thanks

[gandalf]

The first graph seems to show clipping at 114 Mbps. Use 64 bit COUNTER graphs instead to 32 bit graphs to fix that.
For the second one, please use my 2nd link of my sig to analyze what is returned by the target
R.

[lsatkins]

Do you have any extremely slow links that you are monitoring? I was trying to monitor some remote sites that run on wireless air cards and every time I tried to graph them it would start putting gaps in all my other graphs. When I disabled the devices on the slow links the gaps went away. I tried upping SNMP timeouts but it didn't seem to help.

[NLCoredump]

I experienced similar problems in our case it happend when cacti wasn't able to do a full datasource gathering run within its polling cycle.

This happend on a badly tuned system when some off the hosts are down and you have a reasonable high time-out configured. This will increase polling times drasticly if you gather with just a on process with a few threads.
Like we did at first. we experienced similar gaps in our graphs.

With the new dedicated cacti system we now gather about 10K datasource in 15 seconds. We use a 500ms time-out in our network. There are only a couple hosts that are allowed a bit more. Make sure that all scripts that are used complete within a reasonable periode. I use 3 poller processes with an total of 90 threads to gather the data.

We use spine which helped us big time.

[macdaddy]

Gandalf, thanks for the info. You may be on to something with the 64 graphs. I'm pushing 119Mbps right now and noticed that this graph in particular is all kinds of messed up.

I've done some digging in the forums and looked at some of the docs and haven't figured out how to do 64 bit graphs. Do you have a handy link in your sig for that?

Thanks

[lmouly]

Hi

in console/Data sources, look for the datasource of your graph.

Edit it and look for the "output type id" field.
Select "In/out bits (64-bit counters)" instead of ""In/out bits"
Save

This will change the OID used to retrieve the data. Of course this will work only of your equipement answers to it.

[macdaddy]

Excellent. I've changed all the interface graphs on that particular router and they're still working. So far so good. It's a snow day so our utilization is higher than normal. I should know if it fixed that problem before COB today.

Could this somehow have affected my other non-SNMP graphs too? Ie, could it be the cause of the Advanced Ping gaps too? Thanks for the info