InfluxDB/Grafana/Bosun + Cacti = CereusTransport - Updated!

[phalek]

Hi all,

I've created a small plugin which stores all of the polled data into an InfluxDB or Bosun. The data can then be used to be displayed using e.g. Grafana or processed further.

More information and a download link can be found here: https://www.urban-software.com/products ... ansporter/

Thanks to shd for adding bosun support to it


Have a look at this screenshot of a working installation:

Grafana_InfluxDB_CereusTransporter.png (191.78 KiB) Viewed 26549 times

Grafana_InfluxDB_CereusTransporter_LoadAverages.png (121.04 KiB) Viewed 26541 times

[shd]

I've created a small plugin which stores all of the polled data into an InfluxDB. The data can then be used to be displayed using e.g. Grafana or processed further.

Link?

[phalek]

Here you go:

http://blog.network-outsourcing.de/prod ... ansporter/

[shd]

Here you go:
http://blog.network-outsourcing.de/prod ... ansporter/

This plugin heavily interferes with the actual poller process. 5 minute polling should be ok but 1 minute polling may not be able to finish in time. Keep a look at your polling times !

Have you tried batching multiple points at once, as per "Writing multiple points" part of documentation?

There is a possibility to hit max post size, so some size splitting may be needed.

[phalek]

Hence version 0.1. It's a proof-of-concept right now, working but not finished yet. Still fun to work with

The next version will use a PHP module for interfacing with influxdb, allowing more possibilites like adding multiple data points or udp connections (Using https://github.com/influxdb/influxdb-php ).

[phalek]

Update: I got the influxdb write down to a few seconds ( grab Version 0.2 from here: http://blog.network-outsourcing.de/prod ... ansporter/) , though I'm still unable to do a multi-data update.

Anyone knows a working way of doing mass-updated to a influxdb ? I've tried the following version:

Fails with a "bad timestamp" issue:

Code: Select all

curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary 'load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673 
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673'

Fails with an "invalid number" issue:

Code: Select all

curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary 'load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68' 

Succeeds, but doesn't update the database at all:

Code: Select all

curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '
load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68
' 

Succeeds with timestamp, but doesn't update the database at all:

Code: Select all

curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '
load_5min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
load_1min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673 
load_15min,host=myTesthost,title="myTesthost\ -\ PRD\ -\ Load\ Average\ -\ 1\ Minute" value=1.68 1450870689673
' 

The last one does complete within 2 seconds, so it's the fastest version, but as it doesn't update the influxdb it's of no use.
Anyone with some influxdb skills that may help out ?

[shd]

Update: I got the influxdb write down to a few seconds ( grab Version 0.2 from here: http://blog.network-outsourcing.de/prod ... ansporter/) , though I'm still unable to do a multi-data update.

btw, you have hardcoded host name for curl:

Code: Select all

                $cmd = "curl -i -XPOST 'http://xliap114:8086/write?db=mydb' --data-binary '"; 

[phalek]

fixed

The settings are now in the "misc" tab, as was with v0.1

[phalek]

Version 0.3 is now available. The multi update part has been fixed using a InfluxDB/PHP module available ( compatible with PHP >5.3 )

The update time is now taking about 1/4 of the original time ( Depending on the number of datasources per host ).

Please note: The configuration has been changed from URL based to host/port/db_name. So make sure to update your settings (misc tab) accordingly !

http://blog.network-outsourcing.de/prod ... ansporter/

[phalek]

Version 0.4 is now available. This fixes an issue with data points getting lost ( same local_data_id in cacti, eg. traffic_in, traffic_out of network interface eth0 ).

http://blog.network-outsourcing.de/prod ... ansporter/

[theslim1]

Great plugin! It's exactly what I've been looking for. However, I'm having a little issue. The plugin is not picking up all of my data sources and pushing them to InfluxDB, it's missing quite a few per host actually actually. See my debug log below:

Code: Select all

01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] Finished adding [2] of data points for [192.168.1.3]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] Adding [2] of data points for [192.168.1.3]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Appending the following data to request: [influxdb] [1453166282] [192.168.1.3] [cisco\ switch\ -\ Traffic\ -\ Nu0] [traffic_out] [0]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Appending the following data to request: [influxdb] [1453166282] [192.168.1.3] [cisco\ switch\ -\ Traffic\ -\ Nu0] [traffic_in] [0]
01/18/2016 05:18:05 PM - CereusTransporter: Poller[0] DEBUG: Adding the following data to influxdb: [1453166282] [192.168.1.3] 

The host (192.168.1.3) has many more data points then it is seeing. I'm experiencing this issue on about half of my hosts. I'm using the latest version of the plugin, with Cacti 0.8.8a. Here's some screenshots of the number of data sources and graphs that actually exist for the above host,

https://www.dropbox.com/s/bvmgu2xlfz5r6 ... 3.PNG?dl=0
https://www.dropbox.com/s/m6gp825968ugf ... h.PNG?dl=0

Any ideas?

[phalek]

Can you send me a complete log for one polling cycle containing only the debug messages of CereusTransport ?

It's processing the data ordered by "timestamp", so it may actually containg several entries for the same host but scattered around as the poller didn't collect them all at the very same second

[theslim1]

Here's the complete log,

https://www.dropbox.com/s/43lo2ek3iekia ... i.txt?dl=0

Let me know if you need anything else to help debug. I'm also using 1 minute polling, not sure if that causes issues or makes a difference.

[shd]

I also saw that many datapoints were not being pushed into db, wasn't satisfied with the speed of script and with a lack of Bosun support. So I've started to change the script. Well, it became a rewrite.

Here are my changes:

0.45 (changes by simonov@gmail.com)
* Added Bosun export support (http://bosun.org/). Also OpenTSDB. May have broken InfluxDB support, haven't tested it, but should be working.

* Changed settings back to DB URI instead of db, port and so on in separate settings. Made configuration simpler.

* Fixed population of data array by key - version 0.4 was loosing too much data (for example, if there was more than one traffic_out, only one was saved)

* Converted plugin_CereusTransporter_data to a Memory table. Nice improvement in speed. Make sure to make mysql memory table parameters bigger
(tmp_table_size and max_heap_table_size). Table will be recreated automatically by the upgrade script (hopefully).
Data is considered temporary and gets deleted on every sending pass.

* Added much more additional data to metrics: host and hostname, data source type (gauge, counter), index type and value if used, units and so on.
Unit name is taken from Vertical Label of Graph Template for datasource. If there are multiple Graph Templates, only one value will be taken.
Vertical Label text should be from this list, or bosun will skip it: https://godoc.org/bosun.org/metadata#Unit
Metrics info for Bosun is sent to different api outlet.

* Pre-populated all needed info from DB. No SQL queries are made for each metric. HUGE improvement in speed. Now it takes about 90 seconds on average
to send 168k data points into Bosun.

* Additional stats in poller logs. Example:
CereusTransporter: Poller[0] STATS: Time:98.82 LookupTableSize:32685 LookupUnitsTableSize:22363 MetricsCount:168862 AverageHostTime:0.12
SlowestHostnames:[host1: 3.544s, host2: 2.784s, host3: 2.216s, host4: 2.159s]

Changes are not much tested, but they are working in my environment. No guarantees.

[phalek]

I also saw that many datapoints were not being pushed into db, wasn't satisfied with the speed of script and with a lack of Bosun support. So I've started to change the script. Well, it became a rewrite.

thanks shd for this. I'll add it to my code and also upload it to my bitbucket.org account for better tracking.

It's still a kind of proof-of-concept plugin, but i'm glad some of you do find it useful and inspiring to enhance it.


For the data, I think there may be cases where the tables gets deleted (poller_top) while it's still working on the data ( overrunning poller processes ) , so maybe changing the deletion of old data should be done after the processing of it ( e.g delete from `plugin_CereusTransporter_data` where timestamp < oldest_processed_data_item_timestamp ).

It would be nice to know the difference in updating the rrd files and adding the data into inflixdb/bosun ....