Auto TARGET generation?

Support questions about the Network Weather Map plugin

Moderators: Developers, Moderators

User avatar
wwwdrich
Cacti User
Posts: 91
Joined: Thu Feb 03, 2005 5:53 pm
Location: San Jose, CA
Contact:

Auto TARGET generation?

Post by wwwdrich »

Has anyone come up with a good way of generating the TARGET lines for weathermap config files? I have been writing some quick-n-dirty SQL code for my own uses, but it isn't portable as it makes assumptions on how I have named my devices and I have to manually cut-n-paste the filename from data_template_data into my config file.

I am leaning towards writing something like auto-overlib.pl that will pull an specially formatted comment out of the config file and generate the TARGET information, but I don't want to go down that road if someone else already has a tool for this. My thought was to put in a comment like

Code: Select all

LINK xxxx
    #INTERFACE hostname:intin--intout[,hostname:intin--intout]
so the script could generate not only the TARGET, but also INCOMMENT and OUTCOMMENT data. This could even be expanded to included the bandwidth, to auto generate BANDWIDTH and WIDTH (and in my case, SET comment_nudgeout).

My main goal -- not to be the only one here who knows how to extract that information from the Cacti configs. I am not even on the networking team (although I have forgotten more about networking than some of the people who are :-) ). At some point, I would really like to be able to turn maintenance of the map configs over to them, but I need a simpler way of configuring them before I can.

Thoughts?

Howie, I thought I remembered reading somewhere that you had thought about adding something like this directly into Weathermap. Did that ever go anywhere or am I imagining things?
- Dan
[i] "Step up to red alert!" "Are you sure, sir?[/i]
[i] It means changing the bulb in the sign..." - Red Dwarf[/i]
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

Have you tried the editor? You get to pick from a list of Cacti Data Sources by their Cacti title ("Localhost - Errors - 212.21.100.67/bge0")... that's fairly painless.

I'm interested in automatically generated maps generally, but tying the information together in a way that isn't site-specific is pretty tricky. My own maps mix layer 2 and layer 3 interfaces for example, where Extreme decided not to show interface traffic for SVIs but only actual ports. It would probably rely on some kind of naming standard either for interface descriptions or similar. Some devices (Nortel Baystack for one) don't actually expose the port descriptions to SNMP.

Every now and then this topic comes up - the actual generation part isn't so hard, once you have a way to figure out the topology. There are a number of sources for that (Discovery plugin, NeDi), but none of them are standard really, or 100%.

Re: nudge - I saw your screenshots and I like that look. I'll see about making it a bit less kludgy. I think I'll add 'contrast' as an option for comments, like it is for LABELFONTCOLOR too.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
wwwdrich
Cacti User
Posts: 91
Joined: Thu Feb 03, 2005 5:53 pm
Location: San Jose, CA
Contact:

Post by wwwdrich »

I'm actually in the middle of that, I like the *idea* of auto-generated maps, but in general I've found that the maps that come out are pretty useless. As an example, I use nedi to track device and host locations, but the maps are so crowded as to be completely useless.

Weathermap really solve the problem for me, as it generates the pretty pictures automatically, but lets me control the layout and exactly what appears on the maps. The only piece I find painful is getting the TARGET lines (and a few of the other details) into the config.

I haven't looked at the editor in ages, let me see what you've done with it and if it will work for me. From what I remember the last time I used it, I lost some of my stranger customizations when I tried it -- that may not be an issue any more.

And I may still go ahead and write my auto-target.pl script -- we'll see how many more of these massive maps like that server farm I have to generate. Our desktop switch map at our main site is going to be close, with two core routers and six multi-floor buildings with at least one switch per floor. Looking up all of those targets by hand is not going to be pleasant (nor is logging into all sorts of devices to get the topology, but that' just because I can't get a map out of the network team -- I'm not sure they know how any of these devices are connected :) )
- Dan
[i] "Step up to red alert!" "Are you sure, sir?[/i]
[i] It means changing the bulb in the sign..." - Red Dwarf[/i]
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

howie wrote: Something that goes from Cacti host-id (or hostname as it appears in cacti) and interface name (fa0/2) to a traffic graph and DS is probably pretty simple, since Cacti knows that stuff (somewhere - I think it'll be a pretty big query). So suppose you had either an IP, hostname or host_id for each NODE, and an pair of interface names for each LINK, it could do the rest pretty sensibly. As long as each link has at least one 'known' end. Could add in host status (according to Cacti/Thold) while it's there, too.
Working on this now. To buzzy to get to sleep at the moment :o
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

So after a couple of days of fighting with Drupal, I finished this off for some light relief.

Here's a first-cut script that takes a series of SET items from a map, and uses them to populate TARGET, OVERLIBGRAPH, INFOURL and BANDWIDTH.

You have to edit the top of the file to use it right now (it's just a test). It reads in one config, and writes out another new one. I wouldn't recommend writing back to the original, although it should work - this is using the same guts as the editor does to read and write things. You ALSO need to make sure that your editor-config.php exists, and is correct - we use those values to find Cacti.

Because the way INFOURL and OVERLIB work has changed - this script will ONLY work with Weathermap 0.95 and above. Drop it into your weathermap directory, and run it from the command-line php cacti-integrate.php

The things you need in your map are:
for NODEs, one of these 3:

Code: Select all

SET cacti_id 33
SET hostname Core Router 1
SET address 192.0.2.0
The 'address' must be exactly as Cacti has it. The hostname is the 'description' field in Cacti, exactly as Cacti has it.

for LINKs, you need either

Code: Select all

SET in_interface Gi0/1
SET out_interface Gi0/2
note that the in_interface is on the B router, and the out_interface is on the A router. The interface name can be ifName,ifDescr or ifAlias. So you can use your customer's name if you add that to the description in your routers. It has enough brains to swap over traffic_in and traffic_out if you only define an in_interface.

As long as all your links have at least one interface defined, and that interface is on a node where you added the IP or cacti-id, then it should produce a new config file, with BANDWIDTH (from ifSpeed), OVERLIBGRAPH, INFOURL (both back to Cacti), and TARGET.

You can use special tokens in the INCOMMENT and OUTCOMMENT to avoid repeating information:

Code: Select all

  INCOMMENT {link:this:in_interface}
  OUTCOMMENT {link:this:out_interface}
Also, for NODEs, it will fill in the missing bits of info (of the 3), and also add a link to the host.php page for that host (not so sure how useful that is), and a TARGET to show cacti's state for that host.

So, next step is to try and retrieve the interface names using SNMP and CDP/LLDP. That relies on the router's own hostname matching the one in Cacti though. And CDP/LLDP.
Attachments
cacti-integrate.php.txt
(7.4 KiB) Downloaded 512 times
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
wwwdrich
Cacti User
Posts: 91
Joined: Thu Feb 03, 2005 5:53 pm
Location: San Jose, CA
Contact:

Post by wwwdrich »

Wow, great work and fast too! I just gave it a try on a new config I am working on (for our other data center), and it seems to work mostly... One thing though, how can I setup links with multiple targets (i.e. aggregated links or trunks?).

As I was writing this, I figured out a few of my problems -- i.e. all nodes *must* have "set hostname" entries, or it can only get the info from the ones that do (so my links weren't coming up with the targets I expected).

There are still a couple of things that I expected in the resulting file that are missing or not quite what I expected:
  • How is it calculating the bandwidth? If I specify one device I get 10G links, the only only gives me 4.3G. They should all be 10G.
    I am not sure where it is looking for base_url -- but I ended up setting it in editor-config.php to make this work. Without it I end up with broken links in the NODE entries.
    My node labels now have a background color (purple of all things) instead of being transparent -- I have LABELBGCOLOR none on my default node config and I don't see anything in the node definition that should be overriding that
    What should your INFOURL link actually be doing? For my cacti it just takes me to the device list, it seems to be ignoring that host.php?id=xxx URL if there is no action specified.
    The TARGET line that is being generated is a URL, so isn't working.
Any thoughts?
- Dan
[i] "Step up to red alert!" "Are you sure, sir?[/i]
[i] It means changing the bulb in the sign..." - Red Dwarf[/i]
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

wwwdrich wrote:Wow, great work and fast too! I just gave it a try on a new config I am working on (for our other data center), and it seems to work mostly... One thing though, how can I setup links with multiple targets (i.e. aggregated links or trunks?).
You spend all the time you just saved on the other links, to do those ;-)

I can't think of a sensible way to add them. I guess it could list multiple interface names, but then you have to decide on a separator that will *never* appear in your interface names/descriptions.
wwwdrich wrote: As I was writing this, I figured out a few of my problems -- i.e. all nodes *must* have "set hostname" entries, or it can only get the info from the ones that do (so my links weren't coming up with the targets I expected).
You should need only one of the 3 possibilities. I was testing mainly with cacti_id.
wwwdrich wrote: There are still a couple of things that I expected in the resulting file that are missing or not quite what I expected:

How is it calculating the bandwidth? If I specify one device I get 10G links, the only only gives me 4.3G. They should all be 10G.
It fetches ifSpeed from the poller's SNMP cache... is there some ifXSpeed or something for 10G links? I am not lucky enough to have any...
wwwdrich wrote: I am not sure where it is looking for base_url -- but I ended up setting it in editor-config.php to make this work.
It should get it from Cacti's global.php if it's all setup OK. However, with PIA 2.0 I know that sometimes it doesn't get the base_url right for command-line tools.
wwwdrich wrote: Without it I end up with broken links in the NODE entries.
My node labels now have a background color (purple of all things) instead of being transparent -- I have LABELBGCOLOR none on my default node config and I don't see anything in the node definition that should be overriding that
Ooooh. I meant to add a USESCALE and a new SCALE too. What it's doing is half of this.
wwwdrich wrote: What should your INFOURL link actually be doing? For my cacti it just takes me to the device list, it seems to be ignoring that host.php?id=xxx URL if there is no action specified.
The TARGET line that is being generated is a URL, so isn't working.
Uuuh, is it? It should be a cactimonitor: TARGET... hmm.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
wwwdrich
Cacti User
Posts: 91
Joined: Thu Feb 03, 2005 5:53 pm
Location: San Jose, CA
Contact:

Post by wwwdrich »

Howie wrote:
wwwdrich wrote:Wow, great work and fast too! I just gave it a try on a new config I am working on (for our other data center), and it seems to work mostly... One thing though, how can I setup links with multiple targets (i.e. aggregated links or trunks?).
You spend all the time you just saved on the other links, to do those ;-)

I can't think of a sensible way to add them. I guess it could list multiple interface names, but then you have to decide on a separator that will *never* appear in your interface names/descriptions.
Bzzzt... thanks for playing! Next answer please. :)

That would be most of my interfaces, we do a lot of trunking in our core -- both for fail-over and for bandwith. How about something like ':::'? I know I've seen colons in interfaces, and might even have seen double-colon on some really strange serial interfaces, but I don't think I have ever seen a triple.
Howie wrote:
wwwdrich wrote: There are still a couple of things that I expected in the resulting file that are missing or not quite what I expected:

How is it calculating the bandwidth? If I specify one device I get 10G links, the only only gives me 4.3G. They should all be 10G.
It fetches ifSpeed from the poller's SNMP cache... is there some ifXSpeed or something for 10G links? I am not lucky enough to have any...
It's odd -- it works as long as I have our core router as the first device. If I have the edge switch listed first it comes up with 4.3G instead of 10G. I'll have to dig through the SNMP results and see if there is something funky on that interface (like a 32-bit query instead of 64-bit would be my guess).
Howie wrote:
wwwdrich wrote: I am not sure where it is looking for base_url -- but I ended up setting it in editor-config.php to make this work.
It should get it from Cacti's global.php if it's all setup OK. However, with PIA 2.0 I know that sometimes it doesn't get the base_url right for command-line tools.
That's one of the auto-magically calculated things in 0.8.7+. I'll bet it doesn't call a getcwd() (or whatever the PHP equiv is), so it can't get it when I run it with a relative path. Hardcoding it in the editor config is working for me for now.
Howie wrote:
wwwdrich wrote: Without it I end up with broken links in the NODE entries.
My node labels now have a background color (purple of all things) ...
Ooooh. I meant to add a USESCALE and a new SCALE too. What it's doing is half of this.
Hey, that's really neat! One thing that would be nice is the ability to use "none" in the scale entry for color.
Howie wrote:
wwwdrich wrote: What should your INFOURL link actually be doing? For my cacti it just takes me to the device list, it seems to be ignoring that host.php?id=xxx URL if there is no action specified.
The TARGET line that is being generated is a URL, so isn't working.
Uuuh, is it? It should be a cactimonitor: TARGET... hmm.
I end up with something like:

Code: Select all

        INFOURL http://blahblah.com/cacti/host.php?id=131
in my NODE entry after running the script.

The NODE target is ok, but the LINK TARGET has a URL instead of a path that looks like:

Code: Select all

        TARGET http://blahblah.com/cacti//rra/devicerrdname_4404.rrd
which is correct for everything after the //.[/code]
- Dan
[i] "Step up to red alert!" "Are you sure, sir?[/i]
[i] It means changing the bulb in the sign..." - Red Dwarf[/i]
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

OK... time for a version 0.1.1 then.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

wwwdrich wrote:That would be most of my interfaces, we do a lot of trunking in our core -- both for fail-over and for bandwith. How about something like ':::'? I know I've seen colons in interfaces, and might even have seen double-colon on some really strange serial interfaces, but I don't think I have ever seen a triple.
Done. ::: is a separator.
It's odd -- it works as long as I have our core router as the first device. If I have the edge switch listed first it comes up with 4.3G instead of 10G. I'll have to dig through the SNMP results and see if there is something funky on that interface (like a 32-bit query instead of 64-bit would be my guess).
I found the answer but not the solution... there's a ifHighSpeed in ifXTable which is the speed in MBit/sec instead of bit/sec. ifSpeed is a 32-bit value so it can't go above 4G. As far as I can see though, Cacti doesn't fetch ifHighSpeed, so I can't get at it. Assuming you don't have any fiberchannel on the same devices, you could probably add a bodge to say if speed=4G, make it 10G. In fact, thanks to 1000vs1024, you might be able to do that anyway. I haven't checked but it wouldn't surprise me if 4G-FC is 4000000000. The 32-bit max is 1024-base 4G (4294967295).
Hey, that's really neat! One thing that would be nice is the ability to use "none" in the scale entry for color.
You can do white in the scale, or none for the whole scale (so the node stays the same colour). I know what you mean though - I'd like to have a similar thing for layering links with an error outline.
The NODE target is ok, but the LINK TARGET has a URL instead of a path that looks like:

Code: Select all

        TARGET http://blahblah.com/cacti//rra/devicerrdname_4404.rrd
which is correct for everything after the //.
This is fixed now.
Attachments
cacti-integrate.php.txt
(7.4 KiB) Downloaded 425 times
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
wwwdrich
Cacti User
Posts: 91
Joined: Thu Feb 03, 2005 5:53 pm
Location: San Jose, CA
Contact:

Post by wwwdrich »

It doesn't seem to like the multi-interface LINK entries, I'm seeing errors like:

Code: Select all

LINK gldriv3-1
  We'll use the A end.
  Failed to find RRD file for 131/ethernet5/39:::ethernet5/40
The entry it is trying to parse is:

Code: Select all

LINK gldriv3-1
        NODES gld-core-1:375:0 gldriv3
        SET out_interface ethernet5/39:::ethernet5/40
        SET in_interface ethernet9/11:::ethernet9/12
        INCOMMENT {link:this:in_interface}
        OUTCOMMENT {link:this:out_interface}
I'm not sure where the 131/ (and 132/ in some of the other errors) is coming from.

To your other points, I figured that the interface speed was something like that. I go through the same thing trying to collect volume stats from our NetApps since they only support SNMP v1. It isn't really a problem for me with this, I can either hard-code the bandwidth or just use the other end of the link for my "A" device, as it returns the correct value.

For my scale request, white doesn't work -- I have a custom background that isn't white. Unfortunately, I can't show it to you since the company logo features rather promnently... :)

I need to look at the link layering, that could really come in handy for keeping an eye on our WAN links.

Thanks again!
- Dan
[i] "Step up to red alert!" "Are you sure, sir?[/i]
[i] It means changing the bulb in the sign..." - Red Dwarf[/i]
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

wwwdrich wrote:It doesn't seem to like the multi-interface LINK entries, I'm seeing errors like:

Code: Select all

LINK gldriv3-1
  We'll use the A end.
  Failed to find RRD file for 131/ethernet5/39:::ethernet5/40
That looks like the old version. The new one has an extra line with the interface name, even with one interface used:

Code: Select all

LINK rtr1-sw1a
  We'll use the A end.
  Interface: Gi0/1.22
    INFO http://cacti.local/cacti/graph.php?rra_id=all&local_graph_id=1453
    OVER http://cacti.local/cacti/graph_image.php?local_graph_id=1453&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
  Interface: Gi0/2.723
    INFO http://cacti.local/cacti/graph.php?rra_id=all&local_graph_id=1455
    OVER http://cacti.local/cacti/graph_image.php?local_graph_id=1455&rra_id=0&graph_nolegend=true&graph_height=100&graph_width=300
    SPEED 2000000000
Hmmm. Looks like I posted the old one twice :oops:

I'll fix that in a second.
I'm not sure where the 131/ (and 132/ in some of the other errors) is coming from.
The number is the cacti host_id.
To your other points, I figured that the interface speed was something like that. I go through the same thing trying to collect volume stats from our NetApps since they only support SNMP v1. It isn't really a problem for me with this, I can either hard-code the bandwidth or just use the other end of the link for my "A" device, as it returns the correct value.
That's odd. My reading suggested that *nothing* would show the right answer. Interesting.
For my scale request, white doesn't work -- I have a custom background that isn't white. Unfortunately, I can't show it to you since the company logo features rather promnently... :)
I assumed that was why you wanted it... I'll see about that. Colour handling is a little funny in weathermap. I wouldn't mind adding alpha too, but that's a whole other kettle of fish.

Stand by for a real Mk2 script.
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
Howie
Cacti Guru User
Posts: 5508
Joined: Thu Sep 16, 2004 5:53 am
Location: United Kingdom
Contact:

Post by Howie »

This time for sure...

I wrote the start of an automapper over the weekend, too. It uses the IP addresses and netmask information to figure out which interfaces are connected. Unfortunately that makes it useless for me, because on our Extreme switches the L3 interface counters are hopeless. It probably rules out Ciscos with SVIs too. Pity, because along with this script, it gets you from nothing to an ugly-but-correct map with no work :-)
Attachments
cacti-integrate.php.txt
mk2 cacti-integrate script.
(8 KiB) Downloaded 491 times
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
User avatar
wwwdrich
Cacti User
Posts: 91
Joined: Thu Feb 03, 2005 5:53 pm
Location: San Jose, CA
Contact:

Post by wwwdrich »

Ok, you dont' expect the value in the MIBs to actually be correct, now do you?! :) I'll have to query our Foundry devices to see what they are really reporting. Our MG8s report the correct numbers, all of the other ones are the 32-bit values. I supposed if you're querying with SNMP v2 there is no reason they couldn't return the full 64-bit value.

So one bug on my trunked interfaces -- I'm seeing errors like:

Code: Select all

PHP Notice:  Undefined offset:  2000000000 in /var/www/html/cacti-0.8.7b/plugins/weathermap/random-bits/cacti-integrate.php on line 221
Two trunked interfaces will yield 2GB, and it looks like this version doesn't know how to deal with that. I'm sure it will really flip out when it sees some of my other devices that have weird trunks like a 10 GB and two 100MB trunked links for a backup...

Can you replace the delimiter in the comments with a space? Seeing ethernet1/1:::ethernet1/2 looks a little odd.

And one other "nice to have" -- any chance of abbreviating the comments? Foundry likes long names on things, and "ethernet3/1:::ethernet3/2:::ethernet5/6" doesn't fit well on the links. I'm thinking along the lines of abbreviating the common names, "ethernet" becomes either "e" or "eth", "FastEthernet" could become "Fa" (although I think Cisco already does that), etc. Thoughts?
- Dan
[i] "Step up to red alert!" "Are you sure, sir?[/i]
[i] It means changing the bulb in the sign..." - Red Dwarf[/i]
User avatar
wwwdrich
Cacti User
Posts: 91
Joined: Thu Feb 03, 2005 5:53 pm
Location: San Jose, CA
Contact:

Post by wwwdrich »

Howie wrote:This time for sure...

I wrote the start of an automapper over the weekend, too. It uses the IP addresses and netmask information to figure out which interfaces are connected. Unfortunately that makes it useless for me, because on our Extreme switches the L3 interface counters are hopeless. It probably rules out Ciscos with SVIs too. Pity, because along with this script, it gets you from nothing to an ugly-but-correct map with no work :-)
You're braver than I am. I tried to do something like this by walking the mac addr table once upon a time and ended up with a nightmare. I'm currently running around saving the output of "show fdp nei" and "show cdp nei" and manually parsing the output. It's ugly, but it gets me most of the information I need in one (or two) places.

Netmasks don't do me much good on our network either as a lot of what I'm doing crosses subnet boundrys.
- Dan
[i] "Step up to red alert!" "Are you sure, sir?[/i]
[i] It means changing the bulb in the sign..." - Red Dwarf[/i]
Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests