End Device Tracking Add-On for Cacti

[TheWitness]

The application can stand alone on a separate box, but right now, I have it integrated with the Cacti poller. That part is not really required however.

TheWitness

[TheWitness]

Ok,

For everyone watching this post, here is some status. I have taken a half day off from work to get this thing out for beta. I am currently working on making "updated" files and not patches. Cigamit will work on a 0.8.6 compatible plugin. Here are some details:

First, you will be required to test using the SVN version of Cactid (aka Cactid 0.8.6h).

Second, here is a list of new and modified files from that base configuration:

<b>New Files:</b>
mactrack_device_types.php
mactrack_devices.php
mactrack_scanner.php
mactrack_sites.php
mactrack_view.php
poller_mactrack.php
images/tab_mactrack.gif
include/top_mactrack_header.php
include/html/inc_mactrack_device_type_filter_table.php
include/html/inc_mactrack_filter_table.php
include/html/inc_mactrack_view_filter_table.php
lib/mactrack/mactrack_cabletron.php
lib/mactrack/mactrack_functions.php
lib/mactrack/mactrack_cisco.php

<b>Modified Files:</b>
poller.php
include/config_arrays.php
include/config_constants.php
include/config_form.php
include/config_settings.php
include/top_header.php
include/top_graph_header.php
lib/functions.php

Lastly, if you are currently running plug-in's, you can not install this product with those plugins. You should wait for cigamit to provide his plugin version of the product.

I do need alpha/beta testers and folks to develop additional scanners for other vendors switches. Therefore, please help me with that. I will accept patches because there are bugs and additional features such as database import capabilities that I would be glad to receive.

Please do not overlay this on a production system right now.

Thanks,

TheWitness

[TheWitness]

Ok,

Phase 1 Complete: You now have what I have at the office.

In my environment, I currently scan over 1600 switches, hub and routers using 8 concurrent processes in less than 1 hour. Most of these devices are SNMPv2, so the scanning goes pretty quick. If I increase the concurrent processes to 12, it takes only 30 minutes.

<b>IMPORTANT NOTE:</b> In order to properly associate MAC's with IP's, it is essential that every "site" have at least 1 router defined. This applies pretty universally, maybe with the exception of a L2/L3 switch.

To those who choose to go there, I wish you luck. Please submit patches/bug reports to me and I will update this post for a while until I think I am ready to formally release.

Again, you MUST run from the SVN version (BRANCH_0_8_6) of Cacti for this add-on to function properly.

Regards,

TheWitness

Here are some key points/todo's:

• 1) Entering Device Types will currently not affect scanning. There is an incomplete function called mactrack_scanner.php that requires modifications prior to you being able to add/remove devices from the Device Types table and have that impact scanning.
  2) There are spurious UI issues. Some have been reported to me, but I have yet been able to make time to fix.
  3) CSV Import Functions have not been implemented yet. I use phpMyAdmin to add/remove rows from the main tables right now.
  4) There are a few issues with the proper detection of trunk ports on Cisco devices. This will only impact scanning if some "did I do that" local configuration corner cases.
  5) The "ignore ports" field is currently non-functional. I will get this fixed pretty quickly. This only has an impact if you are using a user port as an uplink (cascading switches).
  6) The reset of the database maintenance time under Settings is not 100% functional. Again, time is (not in this case) money, and I have not made a cent with this...

Installation:

To install, follow the instructions below:

• 1) Take the attached files and replace with your existing files. <b>Please note, existing plugins will be destroyed!! If you use plugins, wait for Cigamit</b>.
  2) Then, once you have completed this, you will need to run the following command from the <cacti_webroot> directory: "mysql -u root -p cacti < mactrack.sql"
  3) Login to Cacti and change permissions for the users who will be administrating the product to give them access to it.

That's it I think

Customization notes:

The method that I have choosen to implement verdor specific scan's is to create a file under "<cacti_webroot>/lib/mactrack" called "vendor_functions.php" where "vendor" is your vendor code/name. For example, I have "mactrack_cabletron.php" and "mactrack_cisco.php" today. I would expect an "mactrack_hp.php" and others moving forward.

The base scanning core functions are included in "mactrack_functions.php". These functions include:

<b>valid_snmp_device</b> - this function will take a host and first use the snmp_readstring specified for the host. If that is a valid readstring, it will return TRUE. If not, it will scan the list of snmp_readstrings in the host, and if it finds a good one, it will use it instead and update the host with the new restring as the default. If none of the readstrings work out, the host is marked "down".

<b>find_scanning_function</b> - this is an incomplete function that will utilize the host information and attempt to locate the correct scanning function and any special circumstances around that function.

<b>get_arp_ip_ranges</b> - this function scans the arp table of a device and inserts the ordered mac/ip pairs into a temporary table for later association with the device mac's.

<b>get_generic_switch_ports</b> - this will likely be a popular scanning function for an otherwise dumb switch. It will work with any switch where VLAN information is not required. You pass it a few standard parameters and off it goes.

<b>get_base_dot1dTpFdbEntry_ports</b> - this is a core function that scans the dot1d bridge table and returns valid ports to the parent function for subsequent processing. This function is critical to the overall scanning process.

<b>xform_mac_address</b> - this function will take an ugly mac address from snmp and format it as such "XX:XX:XX:XX:XX:XX". Pretty boring.

<b>xform_standard_indexed_data</b> - this function takes and OID, a device and an alternate readstring as input and returns the last octet of the OID as an index and the value as a value pair. This is a very common translation function throughout.

<b>xform_cisco_workgroup_port_data</b> - this function is similar to the xform_standard_indexed_data function with the exception that Cisco, for reasons unknown to me, decided to make the last 2 octets mean something like card/port. Therefore, it uses the last two to build the relationship rather than the last octet.

<b>xform_indexed_data</b> - this function combines the features of the previous 2 functions in that you can specify the number of octets to use in determining the association. The function could, although I havn't gotten around to it, replace the two prior functions.

<b>db_process_add</b> - this function essentially add's a process entry to a table that keeps track of the number of concurrently running data collection processes.

<b>db_process_remove</b> - this function removes a process from the process table.

<b>db_update_device_status</b> - this function updates a device, once it has been scanned using the mactrack_scanner.php function. It will update port counts, snmp readstrings, etc.

<b>db_store_device_port_results</b> - this function is pretty strait forward, once you have made the association of MAC to port for the entire switch, this function stores that data into a temporary table.

<b>mactrack_walk</b> - this function is essentially no longer required. It at one time optimized the snmpwalk process with snmpv2 devices by using the bulkwalk process. Since I have added this to the base 0.8.6h product, I will, at some time, remove the function.

Please familiarize yourself with those functions in addition to the functions contained in both the Cisco and Cabletron files to begin to understand how your products can be integrated.

[emerillat]

Thanks so much for taking extra time out of your schedule to get this out. I will take a look at it today and see how hard it will be to get it up and going on our Nortel switches.

[cigamit]

Cigamit will work on a 0.8.6 compatible plugin.

I now have the plugin version up and running, I'm just trying to nail down a few things to make sure its running perfectly. I should have a finalized copy posted by the end of the day.

[williem]

Larry,

I am looking forward to testing this!! I will have to set up a new machine and get the latest version of cact out of SVN. I notice that Nortel is not in the current version. Is it planned? We ahve a good mix of Nortel 8600 and Cisco 6500. We also have a lot of BLN routers.

Regards,

Willie

[TheWitness]

All,

Two updated files. Found a bug this morning:

lib/mactrack_functions.php
lib/mactrack_cisco.php

TheWitness

[emerillat]

Any news on the plugin? I am going through the cabletron file right now to see what would need to be changed for a Nortel switch.

[nuinco]

Any news from Cigamit on the plugin version?

[cigamit]

Any news from Cigamit on the plugin version?

I will see about getting it out today, I have fairly busy all weekend.

I am mainly trying to correct some issues I have come across. I have fixed all but the last issue, which is that its not associating IPs with the MAC Addresses that it finds on my network (It is probably me that is doing something wrong). Since ARP tables are new to me, this has been a learning experience.

I have corrected the issue that only the first X devices are actually polled, where X is the total number of processes you allow.

I will see about getting the plugin out in a few minutes. We can then start reporting bugs from there.

[nuinco]

Great Thanks.
We appreciate the effort.

[cigamit]

Here is the current version of the plugin.

There are currently several bugs with this still (selecting ALL from mactrack_view.php does display ALL, you have to select the Site), but I have fixed a few errors (mainly passed by reference errors), and added a few things, such as poller_mactrack.php now has an extra option "-f" that will force a scan now which is nice for debugging.

Let me know how it works for you. And BTW, this does require php-snmp to work!

EDIT: Attachment is below in another post, as I have fixed a few minor issues.

[TheWitness]

Cigamit,

To increase performance, I purposely kept the "Default" or "All" view on the MacTrack tab limited to showing no records, as the query can take a bit of time otherwise when you have 500,000 records to display.

TheWitness

[cigamit]

Cigamit,

To increase performance, I purposely kept the "Default" or "All" view on the MacTrack tab limited to showing no records, as the query can take a bit of time otherwise when you have 500,000 records to display.

TheWitness

Ok, great, I will not try to "fix" that then, heh. Another thing. The poller seems to go into its "cleanup" before all the processes are done since this is set to true instead of false in poller_mactrack.php

Code: Select all

	/* just in case we've run too long */
	$exit_mactrack = TRUE;

This is currently not fixed in the posted plugin since I just noticed it. I still am in the process of tracking down why its not updating the IPs for our MACs since I can see them in the mac_track_ips table.

I see that I also need to fix the images for the plugin version when you delete a device, that was my oops for the day.

Other than that, its starting to look really great!

EDIT: Added the latest plugin which fixes the poller exitting and the images on the delete device page are now fixed.

[TheWitness]

Cigamit,

In order to properly associate the IP's to MAC addresses for a site, you need to associate those routers with the site. If you miss a router, and it happens to carry the traffic for a specific subnet, then the association will not take place.

I will review your additional changes to add clarity and/or confirmation tonight if at all possible.

TheWitness