Fix 64bit counters in Interface Traffic graphs

[coxxy]

I tried to make a 64bit graph with the same results, Im using a Dell Power edge server R220 running server 2012 with a 10 Gig Broadcomm NIC with a 3 Gbps Fiber connection

when it goes over 1Gbps the graph starts over so 1.2Gbps shows as 200Mbps

as far as i can tell its snmp v2

Fix64bit was intended to overcome problems with 32bit counters overflowing near 100Mbps. 1.2Gbps is something else. First of all, check RRD Maximum for you 64 bit graphs, set it to something more than 10^10. Then add some more zeroes at the end just to be sure.

As for timeouts, if the results are same, it could be something else. Try running spine in debug mode (-V=5 for verbosity, -R for read only, -H=yourhostid). see what will it say. Try also running tcpdump -vvv host youhostip for looking into other side.

Working on these steps now to see what happens, It weird how it only messes up at the 550 mb mark
i added an image ..

both these graphs are polled from identically configured servers - dell builds both running the same NIC drivers

the only difference the server having issues has a higher WAN fiber pipe

i have a bandwidth monitor ap running on the server so i can see the server is using over 1 Gbps but its not graphing

[coxxy]

I tried to make a 64bit graph with the same results, Im using a Dell Power edge server R220 running server 2012 with a 10 Gig Broadcomm NIC with a 3 Gbps Fiber connection

when it goes over 1Gbps the graph starts over so 1.2Gbps shows as 200Mbps

as far as i can tell its snmp v2

Fix64bit was intended to overcome problems with 32bit counters overflowing near 100Mbps. 1.2Gbps is something else. First of all, check RRD Maximum for you 64 bit graphs, set it to something more than 10^10. Then add some more zeroes at the end just to be sure.

As for timeouts, if the results are same, it could be something else. Try running spine in debug mode (-V=5 for verbosity, -R for read only, -H=yourhostid). see what will it say. Try also running tcpdump -vvv host youhostip for looking into other side.

Working on these steps now to see what happens, It weird how it only messes up at the 550 mb mark
i added an image ..

both these graphs are polled from identically configured servers - dell builds both running the same NIC drivers

the only difference the server having issues has a higher WAN fiber pipe

i have a bandwidth monitor ap running on the server so i can see the server is using over 1 Gbps but its not graphing

i tried to change the max values and now i get this in the log

03/25/2015 12:10:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 382, Data Sources: traffic_in(DS[47]), traffic_out(DS[47]), traffic_in(DS[48]), traffic_out(DS[48]), traffic_in(DS[49]), traffic_out(DS[49]),

i messed it up somewhere

[coxxy]

fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird

[shd]

fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird

What does the command "rrdtool info you-rrd-file-with-that-graph.rrd" say? May be you should make rrd_maximum higher?

[coxxy]

fixed the graphs but still only able to hit 600mb then the graph messes up ..so 800 down shows a 200 down ..its weird

What does the command "rrdtool info you-rrd-file-with-that-graph.rrd" say? May be you should make rrd_maximum higher?

im not 100% familliar with the CLI ...ill have to check what the command outputs and try the rrd max higher

[coxxy]

rrd_version = "0003"
step = 60
last_update = 1427478901
header_size = 1760
ds[traffic_in].index = 0
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 120
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "2426834031"
ds[traffic_in].value = 3.8675654746e+07
ds[traffic_in].unknown_sec = 0
ds[traffic_out].index = 1
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 120
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "2549141613"
ds[traffic_out].value = 3.6935149153e+06
ds[traffic_out].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 10080
rra[0].cur_row = 2924
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 25920
rra[1].cur_row = 15964
rra[1].pdp_per_row = 5
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = 0.0000000000e+00
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = 0.0000000000e+00
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "MAX"
rra[2].rows = 10080
rra[2].cur_row = 4455
rra[2].pdp_per_row = 1
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = NaN
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[3].cf = "MAX"
rra[3].rows = 25920
rra[3].cur_row = 3591
rra[3].pdp_per_row = 5
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = -inf
rra[3].cdp_prep[0].unknown_datapoints = 0
rra[3].cdp_prep[1].value = -inf
rra[3].cdp_prep[1].unknown_datapoints = 0
cacti@cacti:/usr/share/cacti/cli$

[shd]

rrd_version = "0003"
ds[traffic_in].index = 0
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "2426834031"
ds[traffic_in].value = 3.8675654746e+07
ds[traffic_in].unknown_sec = 0
ds[traffic_out].index = 1
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "2549141613"
ds[traffic_out].value = 3.6935149153e+06
cacti@cacti:/usr/share/cacti/cli$

Try changing rrdtool maximum:

Code: Select all

rrdtool tune your-rrd-file-with-that-graph.rrd -a traffic_in:1.0e+12
rrdtool tune your-rrd-file-with-that-graph.rrd -a traffic_out:1.0e+12

[coxxy]

Applied the change just gotta wait and see if it took it ...fingers crossed !!!

[coxxy]

Seems to still do it ...i restarted the server so will keep watching it see if it gets better .

[coxxy]

Seems to still do it ...i restarted the server so will keep watching it see if it gets better .

DEF:a='/var/lib/cacti/rra/*rrd-file*_traffic_in_484.rrd':'traffic_out':AVERAGE \
DEF:b='/var/lib/cacti/rra/*rrd-file*_traffic_in_484.rrd':'traffic_in':AVERAGE \


When i do a graph debug it shows the 484.RRD but the graph number is 485 ...i ran the command on the 484.rrd ...does that sound right?

[helpdeskdan]

I installed it, but graphs do nothing when I try to convert them, and the button does not show up. Logfile seems to indicate it tries to work. Odd.

[dpotts]

Awesome plugin.. Thanks!

[Yas]

Hello guys,

thanks to author for all the work and everybody else for contributing!

I'm planning to use this fine plugin to convert several hundreds cacti graphs on our production network, but before doing that, I need to be sure nothing will go wrong.
I will greatly appreciate quick and precise answers.

Q1: How much time it takes for this plugin to convert, lets say, one thousand graps? In order not to lose any data, conversion must take less than 5-minutes poll time.
Q2: If something went not as planned, is there a way to revert to old 32-bit graphs without losing any data? Should I somehow backup something beforehand, and what exactly?
Q3: Is there a way to find out all 32-bit graps and convert those in 1-click?

Thanks in advance.

[shd]

Q1: How much time it takes for this plugin to convert, lets say, one thousand graps? In order not to lose any data, conversion must take less than 5-minutes poll time.

Depends on your setup, but it's quite fast. Couple of SQL updates and rrdtool tune command, which doesn't recreate graph, just updates one parameter in rrd file. A couple thousand of fixes should be performed quite fast. I think of seconds, not minutes.

Q2: If something went not as planned, is there a way to revert to old 32-bit graphs without losing any data? Should I somehow backup something beforehand, and what exactly?

You should make database backup and rrd folder backup. The fixing process has some level of robustness, I haven't heard of any data corruption yet, but it's better to be on a safe side.

Q3: Is there a way to find out all 32-bit graps and convert those in 1-click?

Check that you have the plugin configured in "Settings", select graphs that are matching your 32-bit template, select "Fix 64 bit counters" action, click Go and it will show you which graphs it will be converting. Plugin performs some checks before adding graph to the fixing queue.

[Yas]

Thanks for clarification, conversion indeed went well without corrupting any data, we did not lose any statistics.
But for some reason we failed to batch-convert more than 1 interface graphs at once (cacti -> Graph Management -> select what we need and from pull-down menu choose "Fix 64bit counters" -> ok).
One-by-one conversion using a button to the right from a graph is perfectly fine, though.
Where should I start troubleshooting this inconvenience?
Cacti 0.8.7e, PA 2.6, fix64bit 0.4.

PS. No, we do not use thold plugin, and also cacti admin would rather not touch what is already working, so upgrading to newer version of cacti probably will not going to happen; is there any other way?