Traffic graphs not drawning above 80M.

[xinitrc]

Hello.

Sorry for my bad english.

I have a problem with network graphs.There is one frontend we have.Until it has less then 80M traffic on interfaces it keeps drawing the graph.
After it reaches it it draws nothing.No errors in log until traffic is 80M.After reaching this values errors apper:
07/02/2012 04:22:33 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'2006', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (1757, 'traffic_in', '2012-07-02 16:05:01', '5250241086898')'
07/02/2012 04:22:33 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'2006', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (1756, 'traffic_out', '2012-07-02 16:05:01', '4745285687468')' .

And then at night when traffic minimal ,errors dissaper and graph start drawing normally back.I just don't know where to dig else.

Cacti version:0.8.8a
SNMP Version:NET-SNMP version: 5.3.2.2
RRDTool Version RRDTool 1.4.x
Hosts 24
Graphs 902

Poller settings(cmd.php) :
Concurrent Processes 32
Max Threads 1
PHP Servers 10
Script Timeout 250
Max OID 30


System:CentoOS x86_64 5.8(Final)

php.ini:
max_execution_time = 600
max_input_time = 600
memory_limit = 1024M

mysql percona version:Percona 5.1.63

my.cnf:
symbolic-links=0
max_connections=400
log-slow-queries
slow_query_log_file=/var/log/mysql-slow.log
skip-name-resolve
wait_timeout=600
connect_timeout=300
max_allowed_packet=128M
tmp_table_size=128M
max_heap_table_size=128M
query_cache_limit=1G
innodb_flush_method=O_DIRECT
table_cache=2048
thread_cache_size=16
key_buffer_size=1G

P.S:snmp community: v2,graph was created with 64 bit counters.

Screenshot of graph:


Any help would be appreciated.

[websniper]

This looks like a 32-bit counter issue, yet you created it with 64-bit, or it could be a maximum value issue. I can think of no reason for the DB errors. Try creating a new graph using 64-bit and see if it graphs the higher amounts, provided that the device has 64-bit counters. If FAIL, go through the Data Source and Template as well as the Graph Template and check for a Max value close to 80 that can be increased.

I may be wrong, I am still hacking this stuff too.

Good Luck!

[xinitrc]

Let me explain more detailed whats happening.On frontend when traffic rises above 80M,stop to draw not only traffic graphs.But also http://www.percona.com/doc/percona-moni ... lates.html (interrupts,context,switches) and unicast packets.The data is gathered by script there ,which executes snmpwalk to extension MIB.Then it falls below 80M all graphics start to draw again.There is bond interfaces on every server.Connection to snmpd goes throw LAN.There are gigabit interfaces everythere.

Here is the log when errors appear:

Code: Select all

07/02/2012 09:20:00 PM - SYSTEM STATS: Time:298.8697 Method:cmd.php Processes:32 Threads:N/A Hosts:23 HostsPerProcess:1 DataSources:1128 RRDsProcessed:972
07/02/2012 09:20:01 PM - POLLER: Poller[0] WARNING: There are '3' detected as overrunning a polling process, please investigate
07/02/2012 09:20:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty.  Issues Found: 1, Data Sources: (DS[666])
07/02/2012 09:23:34 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'2006', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (1211, 'STAT_interrupts', '2012-07-02 21:00:01', 'gp:100122653790')'
07/02/2012 09:23:37 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'2006', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (1225, 'STAT_context_swtchs', '2012-07-02 20:50:01', 'gq:37176517696')'
07/02/2012 09:24:19 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'2006', SQL:"insert into poller_output (local_data_id, rrd_name, time, output) values (1210, 'STAT_forks', '2012-07-02 21:00:01', 'gr:1659272')'
07/02/2012 09:25:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
07/02/2012 09:25:00 PM - SYSTEM STATS: Time:298.8646 Method:cmd.php Processes:32 Threads:N/A Hosts:23 HostsPerProcess:1 DataSources:1128 RRDsProcessed:932

As we can see it gathers data succesfully.But why does mysql server has gone away errors ?I've already have high enough connect timeouts.
It gives errors in logfile on following graphs:context switches,interrupts,forks.The script is working fine i assure you.

Strange that no errors about traffic,unicast packets.So i assume it puts data in DB.But on some reason data is empty.It has high timeouts on snmpwalk.
I can fetch data by snmpwalk manually with no errors.

Hereis some screenshots:


#################################################

#################################################

#################################################


I've upgraded fom version 0.7.8i because same problem was there.If i use spine i see the same results.

Now traffic is below 80M all is draw as it should.After few hours(when traff jumps over 80M) i will enable debug and post results fetched data for (traffic,unicast packets,forks,interrupts and context switches).

P.S:every time poller runs it gives different amount of RRD's processed?Is this normal behavior?

[xinitrc]

Problem persists.Every time traffic on monitored host is above 80M i get in my cacti logs : mysql server has gone.

[gandalf]

This seems more like a ds MAXIMUM issue. So please rrdtool tune the rrd files in question to increase the upper limit
R.

[doublemarket]

I had exactly same problem and fixed by following command.

Code: Select all

$ rrdtool tune [.rrd file name] -a traffic_in:U
$ rrdtool tune [.rrd file name] -a traffic_out:U

Then you can confirm with rrdtune info command like this.

Code: Select all

$ rrdtool info [.rrd file name]

ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = NaN     <======
ds[traffic_in].last_ds = "8559057350587"
ds[traffic_in].value = NaN
ds[traffic_in].unknown_sec = 180
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = NaN     <======
ds[traffic_out].last_ds = "7413371704086"
ds[traffic_out].value = 1.4256435582e+09
ds[traffic_out].unknown_sec = 0
(snip)