VERY Slow Tree Display - 900+ Devices

[dononeil]

My tree display on a new cacti install is EXTREMELY slow. When I click on the 'graphs' link, it takes upwards of 30 seconds to show the tree.

I've got multiple trees, with the same devices in multiple locations, and unfortunatly we want to have an 'all' tree with a master list.

I've tried enabling export graphs, and it works fine, but it takes upwards of 30 minutes to export all the graphs, so it's pretty useless for pseudo real time data, and of course you can't zoom, or use any of the nifty real-time plugins for viewing, and it takes so much CPU that the poller process times out and I get gaps in my charts.

I've tried getting BOOST 1.8 to work with .8.7a, but any time I install that the graphs stop updating.

So... here's my question:

Is there some way to have a STATIC tree created and stored in an html file rather than one generated dynamically?

If not, is there any way to speed up the creation of the tree? I've already tweaked my MySQL settings to be as optimum as I can get them.

We're running a dual core Xeon 3Ghz, with SLES 10.2, and 4 GB RAM...64 Bit. We're going to be adding more RAM, but I doubt that will help...

Perhaps splitting the disk I/O onto another drive will help? The DB and Cacti are on the same drive.

ANY Optimization suggestions would be appreciated... I've got 10,000+ more devices I want to add, so I'm just scratching the surface here now with this performance issue. I really don't want to have to split cacti up into dozens of servers if I don't have to.

[Linegod]

Why go with a 'master' tree at all. Why not just use the 'Preview' or 'List' view?

[dononeil]

Where is the 'Preview' or 'List' view configured? I'm not seeing any options anywhere for such things.

[dononeil]

Ok, I'm dumb... I see your referring to the 2 options to choose a list or preview view instead of a tree view from the graph display interface...

The problem is that there will be literally thousands of devices we'll be adding, and we want to be able to find them relatively easy, through a tree structure... list and preview mode just give me a search function or pull-down... which is REALLY long.

[Linegod]

list and preview mode just give me a search function or pull-down... which is REALLY long.

Would it be any longer than the tree?

I know what you are getting at - there comes a point when the tree just can't handle the amount of devices that is being thrown at it.

We are looking at revamping the tree, and adding groups as well. This would, I believe, resolve your issue - IIRC, the initial group feature is slated for 0.8.8 (Alpha in Dec 08), so tree groups would come after that....

Hmmm... come to think of it, it could be possible to create a preview page that was driven by the tree as an interim fix....

[dononeil]

By LONG, I mean there are a lot of devices listed in the pull-down...

displaying is much faster in list since it doesn't have to re-create the tree.

My question is why isn't the tree view just generated statically, ie: any time a device is added, or tree entry updated, rathter than dynamically all the time? Seems like a waste of CPU to re-create it every time if nothing has changed/been added since it was last created.

I don't know how "easy" that would be, but it seems to me like it would be a more permanent fix to the current structure, rather than re-vamping the whole thing.

[dononeil]

Does anyone know the largest installation of Cacti???

We're planning on adding somewhere around 10,000 devices, which will generate upwards of 500,000 graphs. We realize we're not going to be able to do this all on one server, but we're trying to judge scalability right now. This graph tree issue is really slowing us down.

We're less than 10% of what we're expecting, and I'd hate to have 20 or more cacti servers!

[gandalf]

Does anyone know the largest installation of Cacti???

We're planning on adding somewhere around 10,000 devices, which will generate upwards of 500,000 graphs. We realize we're not going to be able to do this all on one server, but we're trying to judge scalability right now. This graph tree issue is really slowing us down.

We're less than 10% of what we're expecting, and I'd hate to have 20 or more cacti servers!

Well, 10k hosts is really huge. This requires careful planning, for sure. You may want to contact developers of cacti to prepare for such an installation.
Reinhard

[tuux1598g]

I don't know how "easy" that would be, but it seems to me like it would be a more permanent fix to the current structure, rather than re-vamping the whole thing.

We were experiencing the same problem - generating the graph tree from the database was a killer on a first-visit when the user didn't have the graph tree structure stored in a session. To fix this, we added a few lines of code to the create_dhtml_tree() function in lib/html_tree.php - the diff file is below if you want to give it a go. This has resulted in a great speedup for first-time visitors to the graph tree.

Code: Select all

*** lib/html_tree.php.orig      2008-09-03 12:22:11.000000000 +0100
--- lib/html_tree.php   2008-09-05 10:18:31.000000000 +0100
***************
*** 457,462 ****
--- 457,470 ----

        $dhtml_tree = array();

+       // First we will check that our cached file is still current (< 5 days old would be good).
+       $cache_file='./cache/dhtml_tree.php';
+       if ( file_exists($cache_file) && ( @filemtime($cache_file) > time()-43200 ) ) {
+               //import the DHTML tree from file
+               if ( @include($cache_file) )
+                       return $dhtml_tree;
+       }
+
        $dhtml_tree[0] = $start;
        $dhtml_tree[1] = read_graph_config_option("expand_hosts");
        $dhtml_tree[2] = "foldersTree = gFld("", "")\n";
***************
*** 575,580 ****
--- 583,599 ----
                }
        }

+        // Create the array into the cache file
+        if ( $cache_file_handle = fopen($cache_file,'w') ) {
+                fwrite($cache_file_handle,'<?php'."\n\n".'$dhtml_tree = array();'."\n\n");
+                foreach ( $dhtml_tree as $index => $value ) {
+                       $value=trim($value);
+                       fwrite($cache_file_handle,'$dhtml_tree[\''.$index.'\'] = \''.$value.'\'."\n";'."\n");
+                }
+                fwrite($cache_file_handle,"\n\n".'?>');
+                fclose($cache_file_handle);
+        }
+
        return $dhtml_tree;
  }

Let me know if you foresee any problems with this.


Regards.

[dononeil]

Thanks... that was basically what I was going to do if nobody posted a solution.. makes a huge difference!!!

Maybe something like this should be added to the next rev, with some sort of check to see (such as addding a time stamp to the DB when any new device/graph is added) if anything has changed with the tree so it is re-created only when changes are made.

Maybe a plugin?

[gandalf]

I don't know how "easy" that would be, but it seems to me like it would be a more permanent fix to the current structure, rather than re-vamping the whole thing.

We were experiencing the same problem - generating the graph tree from the database was a killer on a first-visit when the user didn't have the graph tree structure stored in a session. To fix this, we added a few lines of code to the create_dhtml_tree() function in lib/html_tree.php - the diff file is below if you want to give it a go. This has resulted in a great speedup for first-time visitors to the graph tree.

Code: Select all

*** lib/html_tree.php.orig      2008-09-03 12:22:11.000000000 +0100
--- lib/html_tree.php   2008-09-05 10:18:31.000000000 +0100
***************
*** 457,462 ****
--- 457,470 ----

        $dhtml_tree = array();

+       // First we will check that our cached file is still current (< 5 days old would be good).
+       $cache_file='./cache/dhtml_tree.php';
+       if ( file_exists($cache_file) && ( @filemtime($cache_file) > time()-43200 ) ) {
+               //import the DHTML tree from file
+               if ( @include($cache_file) )
+                       return $dhtml_tree;
+       }
+
        $dhtml_tree[0] = $start;
        $dhtml_tree[1] = read_graph_config_option("expand_hosts");
        $dhtml_tree[2] = "foldersTree = gFld("", "")\n";
***************
*** 575,580 ****
--- 583,599 ----
                }
        }

+        // Create the array into the cache file
+        if ( $cache_file_handle = fopen($cache_file,'w') ) {
+                fwrite($cache_file_handle,'<?php'."\n\n".'$dhtml_tree = array();'."\n\n");
+                foreach ( $dhtml_tree as $index => $value ) {
+                       $value=trim($value);
+                       fwrite($cache_file_handle,'$dhtml_tree[\''.$index.'\'] = \''.$value.'\'."\n";'."\n");
+                }
+                fwrite($cache_file_handle,"\n\n".'?>');
+                fclose($cache_file_handle);
+        }
+
        return $dhtml_tree;
  }

Let me know if you foresee any problems with this.


Regards.

Please post this fix as given by http://www.cacti.net/bugs.php for inclusion into next bugfix release
Reinhard