Problems accurately graphing gigabit interfaces

[evilzardoz]

As per subject.

I thought it was odd how most of our network was operating just above 100Mbit/sec (about 120Mbit/sec is the max I am seeing). Looking at our other network management tool, I see 160-170Mbit/sec regulary.

I ran a test that pushed 67GB of data across the network which resulted in about 800Mbit/sec of data transfer as verified by our other network management tool (statseeker). Unfortunately the cacti graphs for that interface list a 109Mbit/sec maximum - and that's for a transfer that lasted over ten minutes. So it seems there is some kind of limit somewhere in the system.

I've increased the default upper limit and re-added one of my devices and that didn't solve the problem.

I've heard something about 64-bit counters -how do I use them? Will they work properly with my Cisco 6500 and 3750G switches? Is this a likely culprit of the problem?

Thanks

[TheWitness]

1) You will need to adjust your Max Value in the Data Templates for Interfaces.
2) You will need to utilize SNMP v2/v3
3) You need to select one of the 64bit counter graphs in Cacti. When creating graphs, you will see them on the drop down.

TheWitness

[evilzardoz]

1) You will need to adjust your Max Value in the Data Templates for Interfaces.
2) You will need to utilize SNMP v2/v3
3) You need to select one of the 64bit counter graphs in Cacti. When creating graphs, you will see them on the drop down.

TheWitness

1) Done.
2) Already using v2.
3) Is there a way to change existing graphs in a batch and have it apply to all graphs or do I need to recreate all my graphs?

I have recreated and changed some graphs and this appears to have fixed the problem - I just don't feel like editing several hundred interfaces one-by-one ;p

Thanks.

[TheWitness]

No. It's the labor of love. Take good notes.

[gandalf]

3) Is there a way to change existing graphs in a batch and have it apply to all graphs or do I need to recreate all my graphs?

Here are some hints. BUT THIS IS A RSIKY TASK. Well, it worked for me, but if your numbers differ from mine, this may have unwanted effects on your database.
All SQLs assume, that you perform your changes for all hosts related to a specific Host Template. You must know the Number of that Host Template for the queries to work. DO NOT USE MY HOST TEMPLATE NUMBER.

Task 1: Change Host SNMP Version for all Hosts of a given Host Template

Code: Select all

# do not forget to rebuild poller cache afterwards
#
UPDATE `host`
set host.snmp_version=2                                         # set new version
where
host.host_template_id=25 and                                    # give the Host Template, here: my Enterasys Switches
host.snmp_version = 1

Replace number 25 by your correct Host Template ID number. Do not forget to rebuild poller cache afterwards.

Task 2: change rrd max for Host Template

Code: Select all

# do not forget to rrdtune all files!
#
UPDATE `host`, `data_local`, `data_template_rrd`
set data_template_rrd.rrd_maximum=100000000000                  # set new maximum
where
host.host_template_id=25 and                                    # give the Host Template, here: my Enterasys Switches
data_local.host_id=host.id and 
data_local.data_template_id=41                                  # Interface Traffic Data Query
and data_template_rrd.local_data_id=data_local.id

Do not forget to rrdtune all files!

Task 3: change 32bit to 64bit OIDs for Host Template

Code: Select all

UPDATE `host`, `data_local`, `data_template_data`, `data_input_data`
set data_input_data.value=14                                    # set 64bit Counters
where
host.host_template_id=25 and                                    # give the Host Template, here: my Enterasys Switches
data_local.host_id=host.id and 
data_local.data_template_id=41                                  # Interface Traffic Data Query
and data_template_data.local_data_id=data_local.id and 
data_input_data.data_template_data_id=data_template_data.id and 
data_input_data.data_input_field_id=14 and                      # Output Type Id
data_input_data.value=20                                        # was 32bit Counter

It would be a good idea to check all that by querying the tables before changing them. I run MySqlAdmin for that.

Reinhard

[Eric 1013]

1) You will need to adjust your Max Value in the Data Templates for Interfaces.
TheWitness

Can you explain what value I need this adjusted to for Max Value? Thank you. When I select 64-bit counter, nothing displays in the graph! Thank you.

[gandalf]

Please see 2nd link of my sig for details
R.

[Eric 1013]

Thank you, gandalf. I reviewed your link. How would I tune the RRD ds for GB? Could you give me the command? I'm rather new to RRDTool commands.

Thank you once again.

Example:

Code: Select all

[root@ ~ rra]# rrdtool info dal6509-02_traffic_in_33.rrd | grep ds
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "577512029042494"
ds[traffic_in].value = 6.9750312943e+07
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "701145791666472"
ds[traffic_out].value = 3.2896981124e+07
ds[traffic_out].unknown_sec = 0

[gandalf]

rrdtool tune dal6509-02_traffic_in_33.rrd --maximum traffic_in:1000000000000
and the same using traffic_out. Repeat this for each rrd file that has the same issue
R.

[Eric 1013]

Thank you. They still show up with M on the graphs. Is that supposed to be? I changed all our RRD files. Here is an example of the changes:

Code: Select all

[root@ ~ rra]# rrdtool info dal6509-02_traffic_in_33.rrd | grep ds
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+12
ds[traffic_in].last_ds = "577751369011765"
ds[traffic_in].value = 1.2993563860e+07
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+12
ds[traffic_out].last_ds = "701276277315554"
ds[traffic_out].value = 7.7568973953e+06
ds[traffic_out].unknown_sec = 0

[/code]

[gandalf]

rrtool tune won't change the unit shown on the graphs. It only allows for storing higher values in the DB. If you want to override the unit default, there's a different way to go
R.

[Eric 1013]

Thank you.

[Eric 1013]

rrtool tune won't change the unit shown on the graphs. It only allows for storing higher values in the DB. If you want to override the unit default, there's a different way to go
R.

Hi gandalf, how would I override the unit? There are several graphs still displaying M instead of G. Thank you.

[gandalf]

man rrdgraph tells you

[-X|--units-exponent value]

This sets the 10**exponent scaling of the y-axis values. Normally, values will be scaled to the appropriate units (k, M, etc.). However,
you may wish to display units always in k (Kilo, 10e3) even if the data is in the M (Mega, 10e6) range, for instance. Value should be an
integer which is a multiple of 3 between -18 and 18 inclusively. It is the exponent on the units you wish to use. For example, use 3 to
display the y-axis values in k (Kilo, 10e3, thousands), use -6 to display the y-axis values in u (Micro, 10e-6, millionths). Use a value of
0 to prevent any scaling of the y-axis values.

This option is very effective at confusing the heck out of the default rrdtool autoscaler and grid painter. If rrdtool detects that it is
not successful in labeling the graph under the given circumstances, it will switch to the more robust --alt-y-grid mode.

This parameter is support by cacti graph template
R.

[rock]

Is there an updated version of this SQL for Cacti 0.8.7e? If I'm looking @ the DB correctly, it looks like the schema may have changed a bit and this SQL won't work anymore.

Thanks in advance.

-Rock

3) Is there a way to change existing graphs in a batch and have it apply to all graphs or do I need to recreate all my graphs?

Here are some hints. BUT THIS IS A RSIKY TASK. Well, it worked for me, but if your numbers differ from mine, this may have unwanted effects on your database.
All SQLs assume, that you perform your changes for all hosts related to a specific Host Template. You must know the Number of that Host Template for the queries to work. DO NOT USE MY HOST TEMPLATE NUMBER.

Task 1: Change Host SNMP Version for all Hosts of a given Host Template

Code: Select all

# do not forget to rebuild poller cache afterwards
#
UPDATE `host`
set host.snmp_version=2                                         # set new version
where
host.host_template_id=25 and                                    # give the Host Template, here: my Enterasys Switches
host.snmp_version = 1

Replace number 25 by your correct Host Template ID number. Do not forget to rebuild poller cache afterwards.

Task 2: change rrd max for Host Template

Code: Select all

# do not forget to rrdtune all files!
#
UPDATE `host`, `data_local`, `data_template_rrd`
set data_template_rrd.rrd_maximum=100000000000                  # set new maximum
where
host.host_template_id=25 and                                    # give the Host Template, here: my Enterasys Switches
data_local.host_id=host.id and 
data_local.data_template_id=41                                  # Interface Traffic Data Query
and data_template_rrd.local_data_id=data_local.id

Do not forget to rrdtune all files!

Task 3: change 32bit to 64bit OIDs for Host Template

Code: Select all

UPDATE `host`, `data_local`, `data_template_data`, `data_input_data`
set data_input_data.value=14                                    # set 64bit Counters
where
host.host_template_id=25 and                                    # give the Host Template, here: my Enterasys Switches
data_local.host_id=host.id and 
data_local.data_template_id=41                                  # Interface Traffic Data Query
and data_template_data.local_data_id=data_local.id and 
data_input_data.data_template_data_id=data_template_data.id and 
data_input_data.data_input_field_id=14 and                      # Output Type Id
data_input_data.value=20                                        # was 32bit Counter

It would be a good idea to check all that by querying the tables before changing them. I run MySqlAdmin for that.

Reinhard