Troubleshoot AUTOM8 rules not creating graphs after upgrade

[madady]

Hello everyone!
I've upgraded my cacti from 0.8.8b to 1.2.19 and (after some fixes) it seems to be working fine. Except for the AUTOM8 section (used to have the automate plugin, but now the functionality is builtin). The rules have been imported and look ok. Here's an example rule: https://imgur.com/a/iuRXvU9

If, for a specific rule I do "Show Matching Devices." or "Show Matching Objects" I do get a long list of devices/interfaces, so I expect that the rule matches my data correctly.

But when I do "Apply automation rules" on a newly added device, it does not create new graphs for those interfaces.
I enabled debugging in poller_automation.php and redid my steps. I was able to get this debug output:

Code: Select all

2022-01-17 14:10:35 - AUTOM8 TRACE form_actions called, action: 6
2022-01-17 14:10:35 - AUTOM8 TRACE form_actions, items: a:1:{i:0;s:4:"5334";}
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_update_device() Device[5334]
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_update_device() Device[5334], sql: SELECT gt.* 		FROM graph_templates AS gt 		INNER JOIN host_graph AS hg 		ON gt.id=hg.graph_template_id 		WHERE hg.host_id=5334 		AND gt.id NOT IN ( 			SELECT gl.graph_template_id 		FROM graph_local AS gl 			WHERE host_id=5334 		)
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_update_device() Found DQ[10] for Device[5334]
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - start - data query: 10
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - sql: SELECT agr.id, agr.name, agr.snmp_query_id, agr.graph_type_id FROM automation_graph_rules AS agr INNER JOIN host_snmp_query AS hsq ON agr.snmp_query_id = hsq.snmp_query_id WHERE agr.snmp_query_id = ? AND hsq.host_id = ? AND enabled="on" - found: 4
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - rule=9 name: 01. SNMP - Interface Statistics - In/Out Bits (64-bit Counters) - SAS
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_matching_objects_filter() called rule id: 9
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() called: [{"id":"27","rule_id":"9","rule_type":"1","sequence":"1","operation":"0","field":"h.description","operator":"14","pattern":""},{"id":"28","rule_id":"9","rule_type":"1","sequence":"2","operation":"1","field":"h.snmp_version","operator":"12","pattern":"2"},{"id":"29","rule_id":"9","rule_type":"1","sequence":"3","operation":"1","field":"h.status","operator":"14","pattern":""},{"id":"30","rule_id":"9","rule_type":"1","sequence":"4","operation":"1","field":"h.disabled","operator":"10","pattern":"0"}], prefix: 
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() returns:   `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_matching_objects_filter() returns:   `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - create sql: SELECT h.id AS host_id, h.hostname, 				h.description, ht.name AS host_template_name 				FROM host AS h 				LEFT JOIN host_template AS ht 				ON h.host_template_id = ht.id WHERE (  `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0') AND h.id=5334 AND h.deleted = "" matches: 1
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] create_dq_graphs() Device[5334] - snmp query: 10 - rule: 01. SNMP - Interface Statistics - In/Out Bits (64-bit Counters) - SAS
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() called: [{"id":"33","rule_id":"9","sequence":"1","operation":"0","field":"ifOperStatus","operator":"7","pattern":"Up"},{"id":"34","rule_id":"9","sequence":"2","operation":"1","field":"ifHwAddr","operator":"16","pattern":""},{"id":"35","rule_id":"9","sequence":"3","operation":"1","field":"ifHwAddr","operator":"8","pattern":"00:00:00:00:00:00"},{"id":"54","rule_id":"9","sequence":"7","operation":"1","field":"","operator":"0","pattern":""},{"id":"53","rule_id":"9","sequence":"8","operation":"3","field":"","operator":"0","pattern":""},{"id":"37","rule_id":"9","sequence":"9","operation":"0","field":"ifType","operator":"1","pattern":"thernet"},{"id":"55","rule_id":"9","sequence":"10","operation":"2","field":"ifType","operator":"1","pattern":"Lag"},{"id":"56","rule_id":"9","sequence":"11","operation":"4","field":"","operator":"0","pattern":""}], prefix:  a.
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() returns:    a.`ifOperStatus` <=> 'Up' AND  a.`ifHwAddr` NOT LIKE ''  AND  a.`ifHwAddr` <> '00:00:00:00:00:00' AND (  a.`ifType` LIKE '%thernet%' OR  a.`ifType` LIKE '%Lag%' )
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - rule=10 name: 02. SNMP - Interface Statistics - MBU Pachets in (64-bit Counters) - SAS
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_matching_objects_filter() called rule id: 10
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() called: [{"id":"31","rule_id":"10","rule_type":"1","sequence":"1","operation":"0","field":"h.description","operator":"14","pattern":""},{"id":"32","rule_id":"10","rule_type":"1","sequence":"2","operation":"1","field":"h.snmp_version","operator":"12","pattern":"2"},{"id":"33","rule_id":"10","rule_type":"1","sequence":"3","operation":"1","field":"h.status","operator":"14","pattern":""},{"id":"34","rule_id":"10","rule_type":"1","sequence":"4","operation":"1","field":"h.disabled","operator":"10","pattern":"0"}], prefix: 
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() returns:   `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_matching_objects_filter() returns:   `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - create sql: SELECT h.id AS host_id, h.hostname, 				h.description, ht.name AS host_template_name 				FROM host AS h 				LEFT JOIN host_template AS ht 				ON h.host_template_id = ht.id WHERE (  `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0') AND h.id=5334 AND h.deleted = "" matches: 1
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] create_dq_graphs() Device[5334] - snmp query: 10 - rule: 02. SNMP - Interface Statistics - MBU Pachets in (64-bit Counters) - SAS
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() called: [{"id":"38","rule_id":"10","sequence":"1","operation":"0","field":"ifOperStatus","operator":"7","pattern":"Up"},{"id":"39","rule_id":"10","sequence":"2","operation":"1","field":"ifHwAddr","operator":"16","pattern":""},{"id":"40","rule_id":"10","sequence":"3","operation":"1","field":"ifHwAddr","operator":"8","pattern":"00:00:00:00:00:00"},{"id":"42","rule_id":"10","sequence":"7","operation":"1","field":"ifType","operator":"1","pattern":"thernet"}], prefix:  a.
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() returns:    a.`ifOperStatus` <=> 'Up' AND  a.`ifHwAddr` NOT LIKE ''  AND  a.`ifHwAddr` <> '00:00:00:00:00:00' AND  a.`ifType` LIKE '%thernet%'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - rule=11 name: 03. SNMP - Interface Statistics - MBU Pachets out (64-bit Counters) - SAS
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_matching_objects_filter() called rule id: 11
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() called: [{"id":"35","rule_id":"11","rule_type":"1","sequence":"1","operation":"0","field":"h.description","operator":"14","pattern":""},{"id":"36","rule_id":"11","rule_type":"1","sequence":"2","operation":"1","field":"h.snmp_version","operator":"12","pattern":"2"},{"id":"37","rule_id":"11","rule_type":"1","sequence":"3","operation":"1","field":"h.status","operator":"14","pattern":""},{"id":"38","rule_id":"11","rule_type":"1","sequence":"4","operation":"1","field":"h.disabled","operator":"10","pattern":"0"}], prefix: 
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() returns:   `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_matching_objects_filter() returns:   `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - create sql: SELECT h.id AS host_id, h.hostname, 				h.description, ht.name AS host_template_name 				FROM host AS h 				LEFT JOIN host_template AS ht 				ON h.host_template_id = ht.id WHERE (  `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0') AND h.id=5334 AND h.deleted = "" matches: 1
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] create_dq_graphs() Device[5334] - snmp query: 10 - rule: 03. SNMP - Interface Statistics - MBU Pachets out (64-bit Counters) - SAS
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() called: [{"id":"43","rule_id":"11","sequence":"1","operation":"0","field":"ifOperStatus","operator":"7","pattern":"Up"},{"id":"44","rule_id":"11","sequence":"2","operation":"1","field":"ifHwAddr","operator":"16","pattern":""},{"id":"45","rule_id":"11","sequence":"3","operation":"1","field":"ifHwAddr","operator":"8","pattern":"00:00:00:00:00:00"},{"id":"47","rule_id":"11","sequence":"7","operation":"1","field":"ifType","operator":"1","pattern":"thernet"}], prefix:  a.
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() returns:    a.`ifOperStatus` <=> 'Up' AND  a.`ifHwAddr` NOT LIKE ''  AND  a.`ifHwAddr` <> '00:00:00:00:00:00' AND  a.`ifType` LIKE '%thernet%'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - rule=12 name: 04. SNMP - Interface Statistics - Errors - SAS
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_matching_objects_filter() called rule id: 12
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() called: [{"id":"39","rule_id":"12","rule_type":"1","sequence":"1","operation":"0","field":"h.description","operator":"14","pattern":""},{"id":"40","rule_id":"12","rule_type":"1","sequence":"2","operation":"1","field":"h.snmp_version","operator":"12","pattern":"2"},{"id":"41","rule_id":"12","rule_type":"1","sequence":"3","operation":"1","field":"h.status","operator":"14","pattern":""},{"id":"42","rule_id":"12","rule_type":"1","sequence":"4","operation":"1","field":"h.disabled","operator":"10","pattern":"0"}], prefix: 
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() returns:   `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_matching_objects_filter() returns:   `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_data_query() Device[5334] - create sql: SELECT h.id AS host_id, h.hostname, 				h.description, ht.name AS host_template_name 				FROM host AS h 				LEFT JOIN host_template AS ht 				ON h.host_template_id = ht.id WHERE (  `h`.`description` IS NOT NULL  AND `h`.`snmp_version` >= '2' AND `h`.`status` IS NOT NULL  AND `h`.`disabled` <= '0') AND h.id=5334 AND h.deleted = "" matches: 1
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] create_dq_graphs() Device[5334] - snmp query: 10 - rule: 04. SNMP - Interface Statistics - Errors - SAS
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() called: [{"id":"48","rule_id":"12","sequence":"1","operation":"0","field":"ifOperStatus","operator":"7","pattern":"Up"},{"id":"49","rule_id":"12","sequence":"2","operation":"1","field":"ifHwAddr","operator":"16","pattern":""},{"id":"50","rule_id":"12","sequence":"3","operation":"1","field":"ifHwAddr","operator":"8","pattern":"00:00:00:00:00:00"},{"id":"52","rule_id":"12","sequence":"7","operation":"1","field":"ifType","operator":"1","pattern":"thernet"}], prefix:  a.
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] build_rule_item_filter() returns:    a.`ifOperStatus` <=> 'Up' AND  a.`ifHwAddr` NOT LIKE ''  AND  a.`ifHwAddr` <> '00:00:00:00:00:00' AND  a.`ifType` LIKE '%thernet%'
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_update_device() Create Tree for Device[5334]
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_device_create_tree() Device[5334] called
2022-01-17 14:10:35 - AUTOM8 TRACE [PID: 1530] automation_execute_device_create_tree() Device[5334], matching rule sql: SELECT atr.id, atr.name, atr.tree_id, atr.tree_item_id,		atr.leaf_type, atr.host_grouping_type		FROM automation_tree_rules AS atr		WHERE enabled='on'		AND leaf_type=3 matches: 0

I'm unsure how to check further and troubleshoot why it doesn't create new graphs for this new device (note - there are two hosts with the same IP address, one of them already existed, the other entry was for testing - as far as I know this shouldn't matter, since the graphs are associated to a host id, right?).
Could you tell me what to check further?


Here's the technical support summary for my cacti instance:

Code: Select all

Technical Support [Summary]
General Information
Date	Mon, 17 Jan 2022 15:22:57 +0200
Cacti Version	1.2.19
Cacti OS	unix
RSA Fingerprint	38:e7:45:d5:c2:46:09:6b:48:d8:88:b1:74:b0:d4:c9
NET-SNMP Version	NET-SNMP version: 5.8
RRDtool Version Configured	1.7.0+
RRDtool Version Found	1.7.0
Devices	3,015
Graphs	138,877
Data Sources	SNMP Get: 3,177
SNMP Query: 269,819
Total: 272,996
Poller Information
Interval	300
Type	SPINE 1.2.19 Copyright 2004-2021 by The Cacti Group
Items	Action[0]: 440,899
Total: 440,899
Concurrent Processes	Name: Main Poller, Procs: 10
Max Threads	Name: Main Poller, Threads: 10
PHP Servers	10
Minimum Connections:	Current: 4096, Min Required: 300
Assumes 100 spare connections for Web page users and other various connections.
The minimum required can vary greatly if there is heavy user Graph viewing activity.
Each browser tab can use upto 10 connections depending on the browser.
Script Timeout	25
Max OID	20
Last Run Statistics	Time:44.4079 Method:spine Processes:10 Threads:10 Hosts:2890 HostsPerProcess:289 DataSources:440899 RRDsProcessed:0
System Memory
MemTotal	32,725,716,000
MemFree	3,746,276,000
MemAvailable	20,766,772,000
Buffers	371,320,000
Cached	16,477,092,000
Active	20,331,356,000
Inactive	7,210,324,000
SwapTotal	1,047,548,000
SwapFree	44,000
PHP Information
PHP Version	7.2.24
PHP OS	Linux
PHP uname	Linux cacti_alu_rocky 4.18.0-348.2.1.el8_5.x86_64 #1 SMP Mon Nov 15 20:49:28 UTC 2021 x86_64
PHP SNMP	Installed. Note: If you are planning on using SNMPv3, you must remove php-snmp and use the Net-SNMP toolset.
max_execution_time	300
memory_limit	1024M

Thanks!

[madady]

Hello again!
I managed to troubleshoot my issue (partially).
It turns out the automation rules were fine (in principle), but the data reported by the device didn't match the rules anymore.
For instance - the rule had this section:

Code: Select all

Item#5	8	(					
Item#6	9	 	ifType	contains	thernet		
Item#7	10	OR	ifType	contains	Lag		
Item#8	11     )	

To match Ethernet and Lag interfaces. However, ifType for the device reports back "6", which according to the SNMP MIB represents "ethernet-csmacd".
So - my conclusion is my installation is missing SNMP MIBs and cacti can't translate the values it gets back from the device to their name counterparts, causing the automations to fail (automations work when adding OR ifType contains 6, for instance).

I did one more thing - since I had data imported from a working old version, I tested on one device that I deleted and added manually. The "match all devices" and "match all objects" worked before because the old devices still had their translated SNMP definitions in their table (I didn't reindex all hosts). I figured it out when I changed the device definition to match only my test device and noticed it didn't match any interfaces for it.

The originally snmp-related installed packages are:

Code: Select all

rpm -qa | grep  snmp
net-snmp-utils-5.8-22.el8.x86_64
net-snmp-agent-libs-5.8-22.el8.x86_64
net-snmp-libs-5.8-22.el8.x86_64
php-snmp-7.2.24-1.module+el8.4.0+413+c9202dda.x86_64
net-snmp-5.8-22.el8.x86_64

Now - the puzzling thing is - it seems my system has the correct MIB for IF-MIB and it works on a snmpwalk (on the same device I tested the automation rules):

Code: Select all

IF-MIB::ifType.1 = INTEGER: softwareLoopback(24)
IF-MIB::ifType.2 = INTEGER: softwareLoopback(24)
IF-MIB::ifType.35684352 = INTEGER: ethernetCsmacd(6)
IF-MIB::ifType.35717120 = INTEGER: ethernetCsmacd(6)
IF-MIB::ifType.35749888 = INTEGER: ethernetCsmacd(6)

Why won't php/cacti make use of the MIBs? Do I need to make special configuration to allow it? Or look elsewhere?
Thanks!

[madady]

I've tried two things to force load the mibs. Both didn't change the output of the ifType:


1. /etc/php.d/20-snmp.ini:

Code: Select all

; Enable snmp extension module
extension=snmp
mibdirs=/usr/share/snmp/mibs/
mibs=all

2. /etc/snmp/snmp.conf:

Code: Select all

mibdirs /usr/share/snmp/mibs/
mibs +all

Any suggestions?
Thanks!

[madady]

I'm still stuck trying to understand why cacti is not using the MIBs to resolve the interface type as a name (see my post above).
I did the following experiment. Created a /var/www/html/snmp_test.php script with the following content:

Code: Select all

<?php
  $session = new SNMP(SNMP::VERSION_2C, "192.168.100.150", "my_community");
  $ifType = $session->get("IF-MIB::ifType.35913728");
  echo "$ifType\n";

?>

If I do a SNMPGet, I get this:

Code: Select all

$  snmpget -v2c -c my_community 192.168.100.150 IF-MIB::ifType.35913728
IF-MIB::ifType.35913728 = INTEGER: ethernetCsmacd(6)

If I run the php file from CLI, I get this:

Code: Select all

$ php test_snmp.php
INTEGER: ethernetCsmacd(6)

If I navigate via the browser and have the web server start the php script (which I assume is what happens when I refresh the host via cacti), I get:

Code: Select all

INTEGER: ethernetCsmacd(6) 

But when I reindex the host with:

Code: Select all

$ php poller_reindex_hosts.php --id=5334 --force --debug
WARNING: Do not interrupt this script.  Reindexing can take quite some time
DEBUG: There are '1' data queries to run
DEBUG: Re-Index Complete: Number[1], TotalTime[1.41], QueryTime[1.41], Device[5334], Description[test_device], DQ[10], Items[134], Orphans[0]

... I still get the type as numeric only, without the textual description.
In the old system, interface type would look like "ethernetCsmacd(6)", while admin status column looked like "up(1)" (and now I no longer have the (1)).
I think the root cause isn't how php queries the SNMP data, but how it stores it in cacti database.

What should I do to troubleshoot further? Where should I look? I don't see any special processing in the template file used: https://rpa.st/MI3Q