InfluxDB/Grafana/Bosun + Cacti = CereusTransport - Updated!

[thunder367]

Hi!

Cacti version 0.8.8a
Plugin CereusTransporter version 0.64
There is a problem when the plugin works with the database.

In log file Cacti error code 1114:
08/08/2018 02:41:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1114', SQL:"INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1533728465',8225,'unicast_in','0') '
08/08/2018 02:41:09 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1114', SQL:"INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1533728465',8225,'unicast_out','0') '
08/08/2018 02:41:10 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.168.40.252', and OID:'.1.3.6.1.2.1.1.3.0'

mysql -u cacti -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1796
Server version: 5.5.60-0+deb7u1 (Debian)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| cacti |
| cacti_dev |
+--------------------+
3 rows in set (0.00 sec)

mysql> use cacti;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+-------------------------------+
| Tables_in_cacti |
+-------------------------------+
| cdef |
| cdef_items |
| colors |
| data_input |
| data_input_data |
| data_input_fields |
| data_local |
| data_template |
| data_template_data |
| data_template_data_rra |
| data_template_rrd |
| graph_local |
| graph_template_input |
| graph_template_input_defs |
| graph_templates |
| graph_templates_gprint |
| graph_templates_graph |
| graph_templates_item |
| graph_tree |
| graph_tree_items |
| host |
| host_graph |
| host_snmp_cache |
| host_snmp_query |
| host_template |
| host_template_graph |
| host_template_snmp_query |
| plugin_CereusTransporter_data |
| plugin_config |
| plugin_db_changes |
| plugin_hooks |
| plugin_realms |
| poller |
| poller_command |
| poller_item |
| poller_output |
| poller_reindex |
| poller_time |
| rra |
| rra_cf |
| settings |
| settings_graphs |
| settings_tree |
| snmp_query |
| snmp_query_graph |
| snmp_query_graph_rrd |
| snmp_query_graph_rrd_sv |
| snmp_query_graph_sv |
| user_auth |
| user_auth_perms |
| user_auth_realm |
| user_log |
| version |
+-------------------------------+
53 rows in set (0.00 sec)

mysql> select count(*) from plugin_CereusTransporter_data;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from plugin_CereusTransporter_data;
+----------+
| count(*) |
+----------+
| 1204 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from plugin_CereusTransporter_data;
+----------+
| count(*) |
+----------+
| 1642 |
+----------+
1 row in set (0.01 sec)

mysql> select count(*) from plugin_CereusTransporter_data;
+----------+
| count(*) |
+----------+
| 5334 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from plugin_CereusTransporter_data;
+----------+
| count(*) |
+----------+
| 5334 |
+----------+
1 row in set (0.00 sec)

mysql> select * from plugin_CereusTransporter_data limit 10;
+------------+---------------+----------------+----------+
| timestamp | local_data_id | key | value |
+------------+---------------+----------------+----------+
| 1533803703 | 9054 | cpu | 1 |
| 1533803703 | 9053 | cpu | 1 |
| 1533803703 | 6840 | nonunicast_out | 0 |
| 1533803703 | 6840 | nonunicast_in | 192 |
| 1533803703 | 9052 | unicast_in | 62713 |
| 1533803703 | 9052 | unicast_out | 21268 |
| 1533803703 | 6839 | unicast_in | 31351229 |
| 1533803703 | 6839 | unicast_out | 30143511 |
| 1533803703 | 6838 | errors_in | 0 |
| 1533803703 | 6838 | discards_out | 0 |
+------------+---------------+----------------+----------+
10 rows in set (0.00 sec)

mysql> select count(*) from plugin_CereusTransporter_data;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)

mysql> select * from plugin_CereusTransporter_data limit 10;
Empty set (0.00 sec)

mysql> describe plugin_CereusTransporter_data;
+---------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+---------------+------+-----+---------+-------+
| timestamp | varchar(1024) | NO | | 0 | |
| local_data_id | int(11) | NO | MUL | 0 | |
| key | varchar(1024) | NO | MUL | 0 | |
| value | varchar(1024) | NO | | 0 | |
+---------------+---------------+------+-----+---------+-------+
4 rows in set (0.00 sec)

mysql> SHOW TABLE STATUS like '%CereusTransporter%';
+-------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+-----------------------------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+-----------------------------+
| plugin_CereusTransporter_data | MEMORY | 10 | Fixed | 0 | 3082 | 0 | 16571914 | 0 | 0 | NULL | 2018-08-09 09:57:46 | NULL | NULL | latin1_swedish_ci | NULL | | NMID CereusTransporter Data |
+-------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+-----------------------------+
1 row in set (0.01 sec)
On misc settings:
influxdb://cacti:********@localhost:8086/cacti

influx
Connected to http://localhost:8086 version 1.6.1
InfluxDB shell version: 1.6.1
> auth
username: cacti
password:
> show databases;
name: databases
name
----
cacti
> use cacti
Using database cacti
> show MEASUREMENTS;
name: measurements
name
----
cisco.router.5.minute.cpu
cisco.router.sb.5.minute.cpu.utilization
host.mib.cpu.utilization
host.mib.hrsystemuptime
interface.errors.discards
interface.non.unicast.packets
interface.traffic
interface.unicast.packets
linux.hard.drive.space
linux.memory.free
linux.memory.free.swap
snmp.linux.uptime
ucd.net.cpu.usage.nice
ucd.net.cpu.usage.system
ucd.net.cpu.usage.user
ucd.net.load.average.1.minute
ucd.net.load.average.15.minute
ucd.net.load.average.5.minute
ucd.net.memory.buffers
ucd.net.memory.cache
ucd.net.memory.free
unix.hard.drive.space
unix.logged.in.users
unix.ping.host
unix.processes

SOLUTION:
-------------

mysql> drop table plugin_CereusTransporter_data;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TABLE `plugin_CereusTransporter_data` (`timestamp` VARCHAR(1024) NOT NULL DEFAULT '0',`local_data_id` INT(11) NOT NULL DEFAULT '0',`key` VARCHAR(1024) NOT NULL DEFAULT '0',`value` VARCHAR(1024) NOT NULL DEFAULT '0',INDEX `local_data_id` (`local_data_id`),INDEX `key` (`key`)) COMMENT='NMID CereusTransporter Data' COLLATE='latin1_swedish_ci' ENGINE=MyISAM;
Query OK, 0 rows affected, 1 warning (0.06 sec)

mysql> SHOW TABLE STATUS like '%CereusTransporter%';
+-------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+-----------------------------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+-----------------------------+
| plugin_CereusTransporter_data | MyISAM | 10 | Dynamic | 729 | 36 | 26336 | 281474976710655 | 17408 | 0 | NULL | 2018-08-09 12:20:20 | 2018-08-09 12:20:24 | NULL | latin1_swedish_ci | NULL | | NMID CereusTransporter Data |
+-------------------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+-----------------------------+
1 row in set (0.00 sec)

[phalek]

Thanks for reporting this. I'll have a look into the setup/update routine which looks like it fails to setup that table correctly.


Issues can also be reported here: https://bitbucket.org/thurban/cereustra ... tatus=open

[alro2111]

Hello,

I have installed the cacti 1.1.38 with CereustTransporter 0.66 and influxdb.
The configuration of the misc tab is influxdb://cacti:password@localhost:8086/cacti.
In influxdb I have created the cacti user and the cacti database.
I always get the same error. "CereusTransporter ERROR: is not a valid scheme"

Can you help me?

2018/10/19 08:13:08 - CereusTransporter DEBUG: Finished adding the data to influxdb: [10.x.x.x] [Time: 0.0001]
2018/10/19 08:13:08 - CereusTransporter ERROR: is not a valid scheme
2018/10/19 08:13:08 - CereusTransporter DEBUG: Adding the following data to influxdb: [1539929583] [10.13.1.1]
2018/10/19 08:13:08 - CereusTransporter DEBUG: Finished adding the data to influxdb: [10.x.x.x] [Time: 0.0001]
2018/10/19 08:13:08 - CereusTransporter ERROR: is not a valid scheme
2018/10/19 08:13:08 - CereusTransporter DEBUG: Adding the following data to influxdb: [1539929582] [10.56.7.11]
2018/10/19 08:13:08 - CereusTransporter DEBUG: Finished adding the data to influxdb: [10.x.x.x] [Time: 0.0001]
2018/10/19 08:13:08 - CereusTransporter ERROR: is not a valid scheme
2018/10/19 08:13:08 - CereusTransporter DEBUG: Adding the following data to influxdb: [1539929582] [10.11.80.1]
2018/10/19 08:13:08 - CereusTransporter DEBUG: Finished adding the data to influxdb: [localhost] [Time: 0.0015]
2018/10/19 08:13:08 - CereusTransporter ERROR: is not a valid scheme
2018/10/19 08:13:08 - CereusTransporter DEBUG: Adding the following data to influxdb: [1539929582] [localhost]
2018/10/19 08:13:08 - CereusTransporter DEBUG: Finished adding the data to influxdb: [] [Time: 0.0001]
2018/10/19 08:13:08 - CereusTransporter DEBUG: Adding the following data to influxdb: [1539929581] []
2018/10/19 08:13:08 - DBCALL DEVEL: SQL Exec: "DELETE FROM `plugin_CereusTransporter_data`"

2018/10/19 08:13:08 - DBCALL DEVEL: SQL Assoc: "SELECT `timestamp`, `local_data_id`, `key`, `value` FROM plugin_CereusTransporter_data ORDER BY `timestamp`,`local_data_id`,`key`"

2018/10/19 08:13:08 - DBCALL DEVEL: SQL Cell: "SELECT COUNT(local_data_id) FROM plugin_CereusTransporter_data"

2018/10/19 08:13:08 - CereusTransporter DEBUG: influxdb Units lookup table size: [152]

2018/10/19 08:13:08 - DBCALL DEVEL: SQL Assoc: " SELECT DISTINCT rrd.local_data_id ,rrd.data_source_name ,gtg.vertical_label FROM graph_templates_item gti INNER JOIN
data_template_rrd rrd ON rrd.id=gti.task_item_id INNER JOIN graph_templates_graph gtg ON gtg.local_graph_id=gti.local_graph_id WHERE gti.local_graph_template_item_id
<> 0 AND gtg.vertical_label <> ''"

2018/10/19 08:13:08 - CereusTransporter DEBUG: influxdb Lookup table size: [158]

2018/10/19 08:13:08 - DBCALL DEVEL: SQL Assoc: " SELECT ds.id ,host.hostname ,host.description ,data.name_cache ,(CASE WHEN rrd.data_source_type_id=1 THEN
'gauge' WHEN rrd.data_source_type_id=2 THEN 'counter' WHEN rrd.data_source_type_id=3 THEN 'counter' WHEN rrd.data_source_type_id=4 THEN 'counter' END) AS
rate ,data_template.name AS metric ,host_template.name AS host_type FROM data_template_data data INNER JOIN data_local ds ON ds.id=data.local_data_id INNER
JOIN host ON host.id=ds.host_id INNER JOIN host_template ON host_template.id=(CASE host.host_template_id WHEN 0 THEN (SELECT id FROM host_template ORDER BY id LIMIT 1)
ELSE host.host_template_id END) INNER JOIN data_template ON data_template.id=data.data_template_id INNER JOIN data_template_rrd rrd ON rrd.local_data_id=data.local_data_id
WHERE data.local_data_template_data_id <> 0 AND host.disabled <> 'on' GROUP BY ds.id"

2018/10/19 08:13:08 - DBCALL DEVEL: SQL Assoc: " SELECT ds.id ,MAX(CASE WHEN type_code='index_type' THEN input.value END) index_type ,MAX(CASE WHEN type_code='index_value'
THEN input.value END) index_value FROM data_template_data data INNER JOIN data_local ds ON ds.id=data.local_data_id INNER JOIN data_input_fields field ON
field.data_input_id=data.data_input_id AND field.input_output='in' AND field.type_code IN ('index_type', 'index_value') INNER JOIN data_input_data input ON
input.data_template_data_id=data.id AND input.data_input_field_id=field.id WHERE data.local_data_template_data_id <> 0"
2018/10/19 08:13:08 - CereusTransporter INFO: Adding data to influxdb

[phalek]

Can yuo check and confirm that you can access the influxdb using that username and password ?
Is influxdb really running on your cacti box (localhost) ?

[alro2111]

Can yuo check and confirm that you can access the influxdb using that username and password ?
Is influxdb really running on your cacti box (localhost) ?

Hi phalek,

I think that the connection to database is correct.

-- > show grants for cacti
database privilege
cacti ALL PRIVILEGES

> quit
XXXXXXXX@cacti:~$ curl -G http://localhost:8086/query --data-urlencode "u=cacti" --data-urlencode "p=password" --data-urlencode "q=SHOW DATABASES"
{"results":[{"series":[{"name":"databases","columns":["name"],"values":[["_internal"],["cacti"]]}]}]}

-- Yes, database and cacti are in the same host.
I attached two pictures

Best regards.

[phalek]

Maybe the influx client isn't compatible with your influxdb version. What version are of InfluxDB are you running ?

[alro2111]

Hi,

Request-Id: c9c0d89b-dd20-11e8-8005-000000000000
X-Influxdb-Version: 1.0.2
Date: Wed, 31 Oct 2018 15:22:32 GMT

[joystickj]

Hi there,

im using Cacti 1.1.38

I installed the plugin and it is showing up under Plugin Settings.

But it does nothing after that. These are the only logline in cacti.log. it is also not populating the MySQL tables. Any ideas what to do?

[luisbcn]

Hi!

I'm using Cacti 1.1.38 with CereusTransporter 0.66, and my issue is that this plugin turns to disabled status after few minutes:

CereusTransporter_0.66.JPG (28.54 KiB) Viewed 8676 times

In log file we only can show something like this, but no errors:

Code: Select all

2019/01/31 22:21:59 - CereusTransporter INFO: Adding data to influxdb
2019/01/31 22:22:00 - SYSTEM DSSTATS STATS: Type:HOURLY, Time:0.3529
2019/01/31 22:22:13 - CereusTransporter STATS: Time:13.57 LookupTableSize:21184 LookupUnitsTableSize:20480 MetricsCount:10748 AverageHostTime:0.00 SlowestHostnames:[192.168.1.98: 0.028s, 192.168.1.194: 0.018s, 192.168.1.7: 0.012s, 192.168.1.212: 0.009s]

We have a high number of SNMP Gets (more than 25.000), I don't know if this data could be a hint...
Do you have any ideas about this "auto-disabling" of this plugin?

Thank you for your time!

[phalek]

Hi

The Cacti log should state a reason why it is being disabled. Can you look for anything related to this ?

[luisbcn]

Thank you phalek for your quick answer!

I activated DEVEL log and added the plugin Cereus Transportr to "plugin debug" list, and this is the only lines that I can found in cacti.log:

Code: Select all

2019/02/01 08:00:07 - CereusTransporter INFO: Adding data to influxdb 
2019/02/01 08:00:08 - CereusTransporter DEBUG: influxdb Lookup table size: [21184] 
2019/02/01 08:00:10 - CereusTransporter DEBUG: influxdb Units lookup table size: [20480] 
2019/02/01 08:00:10 - DBCALL DEVEL: SQL Cell: "SELECT COUNT(local_data_id) FROM plugin_CereusTransporter_data" 
2019/02/01 08:00:10 - DBCALL DEVEL: SQL Assoc: "SELECT `timestamp`, `local_data_id`, `key`, `value` FROM plugin_CereusTransporter_data ORDER BY `timestamp`,`local_data_id`,`key`" 
2019/02/01 08:00:10 - DBCALL DEVEL: SQL Exec: "DELETE FROM `plugin_CereusTransporter_data`" 
2019/02/01 08:00:10 - CereusTransporter DEBUG: Adding the following data to influxdb: [] [] 
2019/02/01 08:00:10 - CereusTransporter STATS: Time:6.86 LookupTableSize:21184 LookupUnitsTableSize:20480 MetricsCount:3409 AverageHostTime:0.00 SlowestHostnames:[192.168.1.5: 0.005s, 192.168.1.6: 0.005s, 192.168.1.31: 0.005s, 192.168.1.102: 0.005s] 
2019/02/01 08:00:54 - CereusTransporter DEBUG: Preparing for the next pass, deleting all data from plugin_CereusTransporter_data (if there was any) 
2019/02/01 08:00:54 - DBCALL DEVEL: SQL Exec: "DELETE FROM `plugin_CereusTransporter_data`" 
2019/02/01 08:00:58 - DBCALL DEVEL: SQL Exec: "INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1549004403',179,'bigip_tmm_mod_cnt','8') " 
2019/02/01 08:00:58 - DBCALL DEVEL: SQL Exec: "INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1549004404',180,'bigip_tmm_mem_used','920009') " 
2019/02/01 08:00:58 - DBCALL DEVEL: SQL Exec: "INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1549004403',184,'bigip_tmm_cpu_tot','1026944903995558812') " 
2019/02/01 08:00:58 - DBCALL DEVEL: SQL Exec: "INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1549004404',185,'bigip_tmm_cpu_idle','61528393946585352') " 
2019/02/01 08:00:58 - DBCALL DEVEL: SQL Exec: "INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1549004403',189,'bigip_tmm_mod_cnt','8') " 
[...]

We noticed that in all the cases, the "INSERT INTO" to plugin_CereusTransporter seems to be interrupted by a init of Spine process:

Code: Select all

2019/02/01 08:16:04 - DBCALL DEVEL: SQL Exec: "INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1549005304',92759,'bigip_tmm_mem_used','99900') "
2019/02/01 08:16:04 - DBCALL DEVEL: SQL Exec: "INSERT INTO `plugin_CereusTransporter_data` (`timestamp`, `local_data_id`, `key`, `value`) VALUES ('1549005313',92761,'bigip_tmm_mod_cnt','U') "
2019/02/01 08:16:04 - SYSTEM STATS: Time:6.5297 Method:spine Processes:10 Threads:15 Hosts:0 HostsPerProcess:0 DataSources:0 RRDsProcessed:0
2019/02/01 08:16:04 - DBCALL DEVEL: SQL Exec: "REPLACE INTO settings (name, value) VALUES ("stats_poller",?)"
2019/02/01 08:16:04 - DBCALL DEVEL: SQL Exec: "INSERT INTO poller (id, total_time, last_update, last_status, status)  VALUES (?, ?, NOW(), NOW(), 2)  ON DUPLICATE KEY UPDATE total_time=VALUES(total_time), last_update=VALUES(last_update),  last_status=VALUES(last_status), status=VALUES(status)"
[...]

No errors are showed, but all the times that we detected the plugin switched to "disabled state", a Spine process appears at the cacti.log.

Could be this the reason? Spine is affecting to the correct operation of CereusTransporter plugin?

Thank you!!

[macsun]

Hi phalek,
My cacti is version 0.8.8h,CereusTransport is version 0.64,InfluxDB 1.7.4. I have one problem, my cacti host Poller Interval is 1min,but CereusTransport not writing data every minute, sometimes 2min,sometimes 3min.

cacti.png (19.84 KiB) Viewed 8605 times

Grafana.png (24.16 KiB) Viewed 8605 times

log.png (51.17 KiB) Viewed 8605 times

The data actually written in InfluxDB is not continuous.

influxdb.png (21.05 KiB) Viewed 8605 times

Please help me fix it.Thank you.

[Osiris]

The plugin turns it self off due to an error in the plugin. Turn logging to low, and you should be able to see the forest from the trees.

[luisbcn]

Hi Osiris,

As I explained:

I activated DEVEL log and added the plugin Cereus Transportr to "plugin debug" list, and this is the only lines that I can found in cacti.log:

No Trees, no forest...

Any suggestion?

[phalek]

There seems to be an issue with either something overrunning or a coding issue.

The log entries display two DELETE statements for the data:

Code: Select all

2019/02/01 08:00:10 - DBCALL DEVEL: SQL Exec: "DELETE FROM `plugin_CereusTransporter_data`"
...
2019/02/01 08:00:54 - DBCALL DEVEL: SQL Exec: "DELETE FROM `plugin_CereusTransporter_data`"

My guess is that this is causing the issue. Need to look into this further.

So this is Cacti 1.1.38 with CereusTransporter 0.66 and 1-Minute polling. What is the usual run-time for the 1-minute polling, is it always way below 60 seconds?