[Error] Threshold 100G Interface

[Jozuslog]

Congratulations for this fantastic software.

I have a Threshold applied on a 100Gbps interface, to alert me when the total traffic, reaches 15% and 17% of their capacity.

The threshold works fine until the traffic of the interface reaches a little more than 34Gbps, and return to normal values when the traffic return below 34Gbps.

To find the percentage of utilization a have the following RPN Expression: “|ds:traffic_in|,8,*,1000000,/,|query_ifHighSpeed|,/,100,*”

In the image attached is an example of the breach. The “Current value” in the image are in Bytes.

Before reaches 34Gbps: Current Value = 3 351 335 882.2 Bytes = 26.8107Gbps
After reaches 34Gbps: Current Value = 4 863 133 878.5 Bytes = 38.9051Gbps

Any help would be appreciated.

Operating System : Unix
Webserver : Apache
Cacti : 0.8.8g
Spine : 0.8.8f
MySQL : 5.5.47-0+deb7u1 (Debian)
PHP : 5.4.45-0+deb7u2
RRDTool : 1.5.5
Net-SNMP : 5.4.3
Plugin Architecture : 3.1
Plugins Instaled
Autom8 : 0.31
Aggregate : 1.01
Settings : 0.71
Discovery : 1.5
Weathermap : 0.97b
Realtime : 0.5
Spikekill : 1.3
Thold : 0.5

[ijimmeh]

I appear to have the same issue with monitoring interfaces that go above 34Gbps, both physical 100G interfaces and aggregates. Setting up baseline monitoring to alert on percentage changes things work fine at low bandwidth but the figure for the current value reported goes out of whack over 34Gbps.

The attached shows the odd figure for the current value as well as the accurate +/- percentage values that are to be alerted on, so I'm unsure as to why thold is accurately calculating the +/- % values with showing an inaccurate current value. Any help would be greatly appreciated.

[ijimmeh]

There's a section of the thold_get_currentval function in thold_functions.php that looks to be causing this for me -

/* assume counter reset if greater than max value */
if ($t_item['rrd_maximum'] > 0 && $currentval > $t_item['rrd_maximum']) {
$currentval = $item[$t_item['name']] / $polling_interval;
}elseif ($t_item['rrd_maximum'] == 0 && $currentval > 4.25E+9) {
$currentval = $item[$t_item['name']] / $polling_interval;
}

Commenting this section out has resolved my issue for the moment, although currently not sure if it's a valid long term solution.

[MOTTAMOURA]

Problem is counter interfaces 32 bits and interface counter 64 bits overflow add variable $icounter
Put the variable a to differentiate.

if ($item[$thold_data['name']] >= $thold_data['oldvalue']) {
// Everything is normal
$currentval = $item[$thold_data['name']] - $thold_data['oldvalue'];
} else {
// Possible overflow, see if its 32bit or 64bit
if ($thold_data['oldvalue'] > 4294967295) {
$currentval = (18446744073709551615 - $thold_data['oldvalue']) + $item[$thold_data['name']];
$icounter = 64; // is counter 64bits
} else {
$currentval = (4294967295 - $thold_data['oldvalue']) + $item[$thold_data['name']];
$icounter = 32; // is counter 32bits
}
}

$currentval = $currentval / $polling_interval;

plugin_log("NOTE COUNTER: '$currentval'", false, "THOLD");

/* assume counter reset if greater than max value */
if ($thold_data['rrd_maximum'] > 0 && $currentval > $thold_data['rrd_maximum']) {
$currentval = $item[$thold_data['name']] / $polling_interval;
}elseif ($thold_data['rrd_maximum'] == 0 && $currentval > 4.25E+9 && $icounter == 32) {
$currentval = $item[$thold_data['name']] / $polling_interval;
}

[Osiris]

Can you create a pull request for that into GitHub? Also, please insure that the max values in your RRDfiles are correct. You may be overflowing your RRDfile.

[Poslik]

I have had similar problem. The main issue was, that the condition in elsif >> }elseif ($thold_data['rrd_maximum'] == 0 && $currentval > 4.25E+9) { << was resolved like true >>
}elseif (|query_ifSpeed| == 0 && 4264205114.9701 > 4.25E+9) { <<. Yes, condition >> |query_ifSpeed| == 0 << is really true.

The variable $thold_data['rrd_maximum'] is filled from select (thold_functions.php line 671):
SELECT thold_data.id, thold_data.local_graph_id, thold_data.percent_ds, thold_data.expression, thold_data.data_type, thold_data.cdef, thold_data.local_data_id, thold_data.data_template_rrd_id, thold_data.lastread, UNIX_TIMESTAMP(thold_data.lasttime) AS lasttime, thold_data.oldvalue, data_template_rrd.data_source_name as name, data_template_rrd.data_source_type_id, data_template_data.rrd_step, data_template_rrd.rrd_maximum FROM thold_data LEFT JOIN data_template_rrd ON data_template_rrd.id = thold_data.data_template_rrd_id LEFT JOIN data_template_data ON data_template_data.local_data_id=thold_data.local_data_id limit 100;

where I have sometimes "|query_ifSpeed|" parameter and sometimes number parameter (depends on Data Source configuration).

I resoved it with this change 2x= on 3x= in condition >> }elseif ($thold_data['rrd_maximum'] === 0 && $currentval > 4.25E+9) { << and testing the data type also.

[netniV]

I presume this is within the thold_get_currentval function? Can you put a full copy of that function that you have in code blocks so I can take a look and compare to the current live one?

[Poslik]

Hello netniV, you are right. The mentioned function is below. (We use thold in version 1.0.3.)

Code: Select all

function thold_get_currentval(&$thold_data, &$rrd_reindexed, &$rrd_time_reindexed, &$item, &$currenttime) {
	/* adjust the polling interval by the last read, if applicable */
	$currenttime = $rrd_time_reindexed[$thold_data['local_data_id']];
	if ($thold_data['lasttime'] > 0) {
		$polling_interval = $currenttime - $thold_data['lasttime'];
	} else {
		$polling_interval = $thold_data['rrd_step'];
	}

	if (empty($polling_interval)) {
		$polling_interval = read_config_option('poller_interval');
	}

	$currentval = 0;

	if (isset($rrd_reindexed[$thold_data['local_data_id']])) {
		$item = $rrd_reindexed[$thold_data['local_data_id']];
		if (isset($item[$thold_data['name']])) {
			switch ($thold_data['data_source_type_id']) {
			case 2:	// COUNTER
				if ($thold_data['oldvalue'] != 0) {
					if ($item[$thold_data['name']] >= $thold_data['oldvalue']) {
						// Everything is normal
						$currentval = $item[$thold_data['name']] - $thold_data['oldvalue'];
					} else {
						// Possible overflow, see if its 32bit or 64bit
						if ($thold_data['oldvalue'] > 4294967295) {
							$currentval = (18446744073709551615 - $thold_data['oldvalue']) + $item[$thold_data['name']];
						} else {
							$currentval = (4294967295 - $thold_data['oldvalue']) + $item[$thold_data['name']];
						}
					}

					$currentval = $currentval / $polling_interval;

					/* assume counter reset if greater than max value */
					if ($thold_data['rrd_maximum'] > 0 && $currentval > $thold_data['rrd_maximum']) {
						$currentval = $item[$thold_data['name']] / $polling_interval;
					}elseif ($thold_data['rrd_maximum'] === 0 && $currentval > 4.25E+9) {
						$currentval = $item[$thold_data['name']] / $polling_interval;
					}
					
				} else {
					$currentval = 0;
				}
				break;
			case 3:	// DERIVE
				$currentval = ($item[$thold_data['name']] - $thold_data['oldvalue']) / $polling_interval;
				break;
			case 4:	// ABSOLUTE
				$currentval = $item[$thold_data['name']] / $polling_interval;
				break;
			case 1:	// GAUGE
			default:
				$currentval = $item[$thold_data['name']];
				break;
			}
		}
	}

	return $currentval;
}

[slakyari]

Hi, I am stuck same problem, How to resolve this.

What I have observed is that when in thresold edit mode in thold.php .. Current value is displayed properly. but some how in lastread column of table thold_data is not correct.

thold.php --------- this shows correct value.
// Data Source Item header
//------------------------
print " <tr>
<td colspan=2 bgcolor='#" . $colors["header"] . "' class='textHeaderDark'>
<strong>Data Source Item</strong> [" . (isset($template_rrd) ? $template_rrd["data_source_name"] : "") . "] " .
" - <strong>Current value: </strong>[" . get_current_value($rra, $ds, $thold_item_data_cdef) .
"]</td>
</tr>\n";

[slakyari]

ISSUE RESOLVED ... NO NEED To change any code.

Go to datasrouce, search your datasrouce and in edit mode just set to 12500000000 (100G = 12500000000 bytes) for traffic in and traffic out.

Data Source Item Fields [traffic_in]
Maximum Value to 12500000000

Data Source Item Fields [traffic_out]
Maximum Value to 12500000000

Then in after few minutes this change will give you correct value in thold.