gexport issues

[thrca]

I recently upgraded from cacti 0.8.8g to 1.2.7. Most everything went well, except for the graph exports, which I was heavily using. They were previously built in, but now they were moved into a seperate plugin.
My base url for cacti is myserver.mydomain.com/cacti/ (This is important for later)

I pulled the gexport plugin from github, installed and enabled it. I currently have configured local exports (I was using FTP, but want to make sure the exports are correct first).

I configured as follows:
Method: local
Args: rsync -zav
Theme: classic
Presentation: Tree
Effective User: n/a
Tree: 3 trees selected (I tried "All Trees" which didnt appear to work at all.. 3 trees for testing)
Expand Devices/Sites: On (I was expecting this to create one folder per Tree branch, which it is not.
Default view thumnails: checked
Max Graphs to export: 9999 (setting to 0 as the tip indicates does NOT work)
Export thumbs: checked
Clear dir: not checked
Dir: /var/www/export

The issues I have found so far:
Old export system made a structure where each branch of the tree had a seperate folder, contained within was the graph_x.html for all the devices in that branch, and a graphs folder with the png files for that branch. The new plugin creates all the graph_x.html for all the devices in the root export folder, regardless of the setting of "Expand Devices/Sites" which indicates it should follow the old behavior.

The main.css that is created is referencing URLs for images that do not exist. It has relative URLs to things like "./../../../images/shadow_gray.gif". Clearly, this makes some assumtions about my layout that are incorrect. I can't figure out where they would expect my export to go to match this. /var/www/export/ is the export path, so based on that, the CSS is expecting /images/ to be in /. This won't work. Why is export not creating shadow_gray.gif in the upload process?

Maximum Graphs to Export: Setting this to 0 does not in fact mean unlimited, because when I set to 0, no graphs export and I see "Reached maximum graphs setting of 0" when running poller_export.php manually.


Is there a way to get this working again where I get graphs per branch in my tree? If I need to download a dev version (or older version, etc.) I can.. I also can help with code if there is specific things I should test in code and I can commit them back to the git.


Jim

[thrca]

This appears to fix the issue with graph_max=0

Code: Select all

diff --git a/functions.php b/functions.php
index 03789ac..9a0d3c8 100644
--- a/functions.php
+++ b/functions.php
@@ -776,7 +776,7 @@ function export_graphs(&$export, $export_path) {

                        $exported++;

-                       if ($exported >= $export['graph_max']) {
+                       if ($exported >= $export['graph_max'] && $export['graph_max'] > 0) {
                                db_execute_prepared('UPDATE graph_exports
                                        SET last_error="WARNING: Max number of Graphs ' . $export['graph_max'] . ' reached",
                                        last_errored=NOW()

[thrca]

Addition to be more concise in debug messages about what is going on

Code: Select all

diff --git a/functions.php b/functions.php
index 9a0d3c8..6266bb5 100644
--- a/functions.php
+++ b/functions.php
@@ -690,6 +690,8 @@ function export_graphs(&$export, $export_path) {
                $user = -1;
        }

+    export_debug('Export presentation is ' . $export['export_presentation']);
+
        if ($export['export_presentation'] == 'tree') {
                if ($trees != '0') {
                        $sql_where = 'gt.id IN(' . $trees . ')';
@@ -697,7 +699,7 @@ function export_graphs(&$export, $export_path) {

                $trees = get_allowed_trees(false, false, $sql_where, 'name', '', $total_rows, $user);

-               export_debug('There are ' . sizeof($trees) . ' to export');
+               export_debug('There are ' . sizeof($trees) . ' trees to export');

                if (sizeof($trees)) {
                        foreach($trees as $tree) {
@@ -714,7 +716,7 @@ function export_graphs(&$export, $export_path) {
                                'local_graph_id', 'local_graph_id'
                        );

-                       export_debug('There are ' . sizeof($graphs) . ' to export for all trees.');
+                       export_debug('There are ' . sizeof($graphs) . ' graphs to export for all trees');

                        if (sizeof($graphs)) {
                                foreach($graphs as $local_graph_id) {

[thrca]

Curiously, in function "export_graphs", it was returning 0 for sizeof($graphs)... Further analysis reveals that the tree ids (in my test case) that it was looking for was 24, 63. The table graph_tree_items contains only rows for 22,34. Why the discrepancy?

[thrca]

So the mis-count was because debug was only reporting graphs not associated with hosts.. I made the following adjustments to debugging to account for this.

Code: Select all

diff --git a/functions.php b/functions.php
index 6266bb5..297aa90 100644
--- a/functions.php
+++ b/functions.php
@@ -704,19 +704,18 @@ function export_graphs(&$export, $export_path) {
                if (sizeof($trees)) {
                        foreach($trees as $tree) {
                                $ntree[] = $tree['id'];
+                export_debug('Tree \'' . $tree['name'] . '\' with id \'' . $tree['id'] . '\' is allowed');
                        }
                }

                if (sizeof($ntree)) {
                        $graphs = array_rekey(
-                               db_fetch_assoc('SELECT DISTINCT local_graph_id
+                db_fetch_assoc('SELECT DISTINCT local_graph_id
                                        FROM graph_tree_items
                                        WHERE local_graph_id > 0
                                        AND graph_tree_id IN(' . implode(', ', $ntree) . ')'),
-                               'local_graph_id', 'local_graph_id'
-                       );
-
-                       export_debug('There are ' . sizeof($graphs) . ' graphs to export for all trees');
+                'local_graph_id','local_graph_id'
+            );

                        if (sizeof($graphs)) {
                                foreach($graphs as $local_graph_id) {
@@ -726,11 +725,15 @@ function export_graphs(&$export, $export_path) {
                                }
                        }

+            export_debug('There are ' . sizeof($graphs) . ' graphs not in hosts to export for all trees');
+
                        $hosts = db_fetch_cell_prepared('SELECT GROUP_CONCAT(DISTINCT host_id)
                                FROM graph_tree_items
                                WHERE graph_tree_id IN(?)',
                                array(implode(', ', $ntree)));

+            export_debug('There are ' . sizeof(explode(',',$hosts)) . ' hosts to export for all trees');
+
                        if ($hosts != '') {
                                $sql_where = 'gl.host_id IN(' . $hosts . ')';
                                $graphs = get_allowed_graphs($sql_where, 'gtg.title_cache', '', $total_rows, $user);
@@ -743,6 +746,8 @@ function export_graphs(&$export, $export_path) {
                                        }
                                }
                        }
+
+                       export_debug('There are ' . sizeof($ngraph) . ' total graphs to export for all trees');
                }
        }else{
                if ($sites != '') {

[netniV]

If you can submit your change as a PR, we can review and commit that. I'll try and have a look at your other comments tomorrow if I have time.

[thrca]

PR created for the max_graphs issue and some additional logging.. Regarding the old behavior of creating a folder per tree branch, this was a LOT more involved and probably not needed for new users of exports, so I just forked and started re-working this into my version. I depend on this old structure because I have external systems that are depending on it, so I had to reproduce it as close as possible. I may re-write my external systems to use the new structure, if I can figure it out, but requires a bunch of regex parsing of the generated html files.

[netniV]

To be fair, I think there may be other users who like the subfolder idea. I don't use it but I know it sounds good to me.

It may be worth having a branch that contains the folder work so it can be compared/pulled if it looks good.

[thrca]

For the most part, I scrapped the branch-per-folder thing and overhauled how my 3rd party system is reading data. The issue I am having currently is related to how the export works. When I have all trees selected (103 trees), it does not successfully export. I have attempted selecting all the trees individually, and somewhere around 63-67 trees, it stops working properly, but before that, it works successfully.

It appears to go through the export correctly, but fails when writing the graph tree data. I have a hunch that it is selecting all the data into an array, but overrunning the amount of memory allocated to each process, so I have to play with the memory settings for PHP to see if I can get it working.

The way I currently have it built for my fork is that I export all the graphs and thumbnails images, then rather than writing all the graph_*.html and stuff, I only am creating a .json file with data about the graphs, which I then ingest using my own system to display the data as needed.

[netniV]

Sounds interesting. Will check it out when your done to see if I want to pull any of it back.

[thrca]

I made corrections on issue#42, my apologies if I mucked up the git process, I am a bit rusty on git.

I mentioned in that conversation, there is still an issue where gexport fails if there is too many graphs to export. I believe this is a memory limit issue in the way it loads all the data to parse for export. I can select about 50% of my graphs before gexport fails.

While I didn't have a ton of time to dedicate to the problem, I tried several ways of resolving it, to no avail, including attempting to write out data during the loop. My gexport is quite heavily modified, so it would be challenging for me to merge specific bits into develop, but I am willing to help in any way I can, even if that just means providing a spot where we can debug the issue with enough graphs to cause the problem.