False Positives - Negative Numbers for Current Data

[adityas]

Running Thold 0.4.4.

Every 2-3 polling cycles, thold returns a negative value for the host, which is obviously not correct. This is genarating a lot of false alarms.

06/21/2011 03:46:49 PM - THOLD: Threshold Breached ID: 1 DataTemplate: Juniper Qos Data DataSource: bytes Type: High/Low Enabled: on Current: -6538.1005 High: Low: 1024.00 Trigger: Never SentTo: <email>

06/21/2011 03:46:49 PM - SYSTEM THOLD STATS: Time:0.0724 Tholds:1 Hosts:0

06/21/2011 03:51:52 PM - THOLD: Threshold Restored ID: 1 DataTemplate: Juniper Qos Data DataSource: bytes Type: High/Low Enabled: on Current: 3454.2234 High: Low: 1024.00 Trigger: Never SentTo: <email>

06/21/2011 03:51:52 PM - SYSTEM THOLD STATS: Time:0.0693 Tholds:1 Hosts:0

06/21/2011 03:57:05 PM - SYSTEM THOLD STATS: Time:0.0058 Tholds:1 Hosts:0

06/21/2011 04:01:52 PM - THOLD: Threshold Breached ID: 1 DataTemplate: Juniper Qos Data DataSource: bytes Type: High/Low Enabled: on Current: -6996.8089 High: Low: 1024.00 Trigger: Never SentTo: <email>

06/21/2011 04:01:52 PM - SYSTEM THOLD STATS: Time:0.0640 Tholds:1 Hosts:0

The actual graph never has a negative value. And if I go into Thresholds -> <Click on Threshold Name> ------> the current value here is always the correct +ve value.

Anyone come across this before?

[adityas]

Digging into this further, it seems that every second poll is incorrect. Any idea why?

mysql> select threshold_value, current from plugin_thold_log where host_id = 3752;
+-----------------+----------------+
| threshold_value | current |
+-----------------+----------------+
| 1073741824 | 917085699.167 |
| 134217728.00 | -772338689.353 |
| | 387605445.023 |
| 1024.00 | -5896.9411 |
| | 2914.9297 |
| 1024.00 | -5883.8693 |
| | 2976.7318 |
| 1024.00 | -5917.5322 |
| | 3018.3839 |
| 1024.00 | -6070.3926 |
| | 3059.9365 |
| 1024.00 | -6211.6285 |
| | 3169.204 |
| 1024.00 | -6223.3237 |
| | 3385.0476 |
| 1024.00 | -6538.1005 |
| | 3454.2234 |
| 1024.00 | -6996.8089 |
| | 3579.1905 |
| 1024.00 | -7138.9477 |
| | 3650.8829 |
| 1024.00 | -7297.7495 |
| | 3721.5213 |
| 1024.00 | -4.0449 |
| | 4076.1388 |
| 1024.00 | -5949.635 |
| | 3023.837 |
| 1024.00 | -6039.2051 |
| | 2990.0864 |
| 1024.00 | -5938.25 |
| | 2902.8057 |
| 1024.00 | -5921.48 |
| | 3036.2792 |
| 1024.00 | -6184.9616 |
| | 3004.3685 |
| 1024.00 | -6040.559 |
| | 2943.5163 |
| 1024.00 | -5958.433 |
| | 3002.7692 |
| 1024.00 | -6106.7687 |
| | 2901.3776 |
| 1024.00 | -6014.9968 |
| | 3025.3239 |
| 1024.00 | -6023.4827 |
| | 3115.6453 |
| 1024.00 | -6204.8323 |
+-----------------+----------------+
46 rows in set (0.00 sec)

[adityas]

If it helps, I've only noticed this happening for 64 bit counters.

[noname]

Just a information..

In thold/includes/polling.php:

Code: Select all

switch ($t_item['data_source_type_id']) {
        case 2: // COUNTER
                if ($item[$t_item['name']] >= $t_item['oldvalue']) {
                        // Everything is normal
                        $currentval = $item[$t_item['name']] - $t_item['oldvalue'];
                } else {
                        // Possible overflow, see if its 32bit or 64bit
                        if ($t_item['oldvalue'] > 4294967295) {
                                $currentval = (18446744073709551615 - $t_item['oldvalue']) + $item[$t_item['name']];
                        } else {
                                $currentval = (4294967295 - $t_item['oldvalue']) + $item[$t_item['name']];
                        }
                }
                $currentval = $currentval / $polling_interval;
                break;
        case 3: // DERIVE
                ...

Thold plugin will try to adjust counter when actual value "jumped" its border between positive and negative. (because the counter is signed variable)
And, for COUNTER data type, it calculates current rate by subtracting old value (=previous rate) from current actual value.

I have no idea why your problem occurred..

Code: Select all

+-----------------+----------------+
| threshold_value |    current     |
+-----------------+----------------+
|   1073741824    |  917085699.167 | ...(A)
|    134217728.00 | -772338689.353 | ...(B)
|                 |  387605445.023 |
|       ...       |       ...      |


1073741824 = 1024 ^3 = 1G
134217728.00 = 1024 ^2 *128 = 128M

For example, did you perform some operations between (A) and (B)?
- change counter from 32bit to 64bit, or from 64it to 32bit
- change status of the threshold settings (enabled/disabled) when it was triggered
- or other changes

Sorry if I confused you.

I recommend, try to disable/uninstall thold plugin and re-install if possible.

[adityas]

Thanks for taking the time to reply.

To answer your questions:

a) Counter value was never changed. It was/is always a 64 bit counter
b) Did not change any setting / status
c) This is a fresh install of thold. This is the first ever thold I'm attempting to set up.

[adityas]

If anyone is actually reading the thread, I've found the problem. Running the actual SQL query that the polling.php script runs:

Code: Select all

mysql> select thold_data.percent_ds, thold_data.data_type, thold_data.cdef, thold_data.rra_id, thold_data.data_id, thold_data.lastread, thold_data.oldvalue, data_template_rrd.data_source_name as name, data_template_rrd.data_source_type_id, data_template_data.rrd_step as polling_interval FROM thold_data LEFT JOIN data_template_rrd on (data_template_rrd.id = thold_data.data_id) LEFT JOIN data_template_data ON ( data_template_data.local_data_id = thold_data.rra_id ) WHERE data_template_rrd.data_source_name != '' AND thold_data.rra_id IN (116386);
+------------+-----------+------+--------+---------+---------------+--------------+-------+---------------------+------------------+
| percent_ds | data_type | cdef | rra_id | data_id | lastread      | oldvalue     | name  | data_source_type_id | polling_interval |
+------------+-----------+------+--------+---------+---------------+--------------+-------+---------------------+------------------+
| bytes      |         0 |  122 | 116386 |  352688 | 376395652.977 | 252160755246 | bytes |                   3 |              300 | 
+------------+-----------+------+--------+---------+---------------+--------------+-------+---------------------+------------------+

If you notice, the data_source_type_id is "3", which equates to DERIVE. However, this is not a DERIVE, but a 64 bit counter.

Manually polling the device proves this as well:

Code: Select all

JUNIPER-SMI::jnxMibs.5.1.1.5.73.78.71.82.69.83.83.45.82.69.65.67.72.0.0.0.0.0.0.0.0.0.0.0.10.66.101.115.116.69.102.102.111.114.116.0.0.0.0.0.0.0.0.0.0.0.0.0 = Counter64: 52665051987

So why does cacti/thold think its a derive? And how do I fix it?

[noname]

>> So why does cacti/thold think its a derive?

Do you have Data Template whose Data Source Type is set to DERIVE?

Try to perform the below query to find that:

% mysql -uUSER -pPASS cacti -e 'SELECT r.id,r.local_data_id,r.data_template_id,t.name,data_source_type_id,r.data_source_name FROM data_template AS t LEFT JOIN data_template_rrd AS r ON r.data_template_id=t.id WHERE r.data_source_type_id=3'

Example of the output:

Code: Select all

+------+---------------+------------------+----------------------------+---------------------+------------------+
| id   | local_data_id | data_template_id | name                       | data_source_type_id | data_source_name |
+------+---------------+------------------+----------------------------+---------------------+------------------+
| 1897 |             0 |              214 | Interface - Traffic (test) |                   3 | traffic_in       |
+------+---------------+------------------+----------------------------+---------------------+------------------+

[TheWitness]

I suspect that the reason there are not updates to this is due to the user figuring out the errors in his/her ways. Let me know if this is not the case.

[zakki]

Now I use RPN Expression -> ABS(|ds:traffic_in|) until permanent fix applied.

[TheWitness]

Now that's an interesting approach to solving an RRDtool problem and a good use of the new Thold functionality.

[dlalaina]

Please, someone help me with this, the attached image has informations about the problem.