graph_xport 5 minute values for timespans

[pestilence]

The new cacti has the export plugin in its core now, its lovely and way better than what I produced as a plugin.
Recently though I came to a problem, a request from one of the network departments, I keep 5 minute values in my rrd files, and with the plugin the departments requesting the 5 minute information could retrieve this data, the new cacti had not this support, since it uses the rrdtool xport function.
I had to find a way to produce this behaviour inside the new cacti, so here is a quick and dirty fix (maybe we can clean it sometime ) I modified the rrd.php file located under the /lib folder, line 1306:

Code: Select all

 $xport_opts =
                "-m 1000000" . RRD_NL .
                "--start=$xport_start" . RRD_NL .
                "--end=$xport_end" . RRD_NL .
                "--step 300" . RRD_NL;

The -m flag allows the xport file to include all rows requested for a big timespan (say a week, I will have to find a function to calculate the needed rows for the timespans) and the --step flag orders the xport tool to use the 5 minute steping of cacti for the results.

[TheWitness]

So, why is it pulling it from the wrong RRA? Maybe you should dump one of your RRDfiles and PM it to me.

Larry

[pestilence]

So, why is it pulling it from the wrong RRA? Maybe you should dump one of your RRDfiles and PM it to me.

Larry

Hey Larry!
It's not pulling it from the wrong RRA, if you check the RRDTOOL documentation the default -m (max rows) is 400 which means that data gathered (even if you request 5 minute Average) gets transformed so that it can fill the max rows.
Changing it and making it a big number allows the xport function to poll the rrd and get the ammount of information it needs.
I am sorry if I can't explain it simpler, just make a manual query on a RRD file for a long timespan (say 2 weeks) and see what it returns with the default options (it increases the stepping to allow all the data to fit into the 400 rows -default- ), if you increase the rows (through the -m flag) it allows you to get the data as you requested.

[ralfk]

I have set --maxrows to 6000. Higher values are bad on my server, since the execution time exceeds 30 seconds (which is my maximum in php.ini).

[TheWitness]

Ok, can you log a feature request for this at http://bugs.cacti.net. I have an idea.

Larry

[pestilence]

Ok, can you log a feature request for this at http://bugs.cacti.net. I have an idea.

Larry

Yeah! Doing it right now Larry

[TheWitness]

I have set --maxrows to 6000. Higher values are bad on my server, since the execution time exceeds 30 seconds (which is my maximum in php.ini).

That's a long time. Maybe I should increase the timeout value in graph_xport.php. What do you think is reasonable?

You can do this in graph_export.php

Code: Select all

ini_set("max_execution_time", 300);

TheWitness

[ralfk]

That's a long time. Maybe I should increase the timeout value in graph_xport.php. What do you think is reasonable?
You can do this in graph_export.php

Thanks for the hint. I think waiting for more than 30 seconds for a server response is not reasonable.

However, I am wondering why it takes so much time to export 6000 rows from a RRD database. On command line the "rrdtool xport" takes a second for 64000 rows with 3 DS and a total export size of about 4 MBytes. Maybe the XML parsing is so slow ?

[TheWitness]

That's a long time. Maybe I should increase the timeout value in graph_xport.php. What do you think is reasonable?
You can do this in graph_export.php

Thanks for the hint. I think waiting for more than 30 seconds for a server response is not reasonable.

However, I am wondering why it takes so much time to export 6000 rows from a RRD database. On command line the "rrdtool xport" takes a second for 64000 rows with 3 DS and a total export size of about 4 MBytes. Maybe the XML parsing is so slow ?

Or buggy. Does it actually complete? What version of RRDtool are you using? Some versions are segfaulting during export.

TheWitness

[ralfk]

I had version 1.2.23, which was in deed segfaulting on my 64 bit system.
With 1.2.25 the "rrdtool xport" works fine on command line and completes in Cacti.

Exporting 6000 rows with Cacti takes about 30 seconds. On command line "rrdtool xport" takes less than 1 second (well, even without converting XML to CSV). Looks strange for me.

[TheWitness]

Interesting. I guess it could be a memory issue in PHP. What do you have the memory_limit set to. I will have to look at this later as I have to be on the road now.

TheWitness

[ralfk]

First I had memory_limit set to 64M. That was not sufficient for 6000 rows. After setting to 128M Cacti export worked. So, a memory limit was not an issue any more. No other errors in apache log.

Hm, strange. I have just tested again exporting 6000 rows with Cacti. Now it takes 1-2 seconds. I swear that it was about 30 seconds last time. Ah, okay, now after repeating the export, the execution time increases significantly up to 20 seconds, but next time again 1-2 seconds. It seems not to be traceable (my server has enough CPU power and free memory, so that should not be an issue). When I set --maxrows to 60000, Cacti delivers a blank page after 1 second, no error message in apache log. "rrdtool xport" works fine with --maxrows = 60000

PHP version: 5.2.0
Apache: 2.2.3

[ralfk]

Okay, I think it was still a memory setup problem with PHP.

After setting memory_limit to 512M and --maxrows = 40000, I was able to export 35000 rows with 4 DS in 2 seconds. 384M was too less. This is sufficient for me fetching 1 year CSV-data at 900 STEP interval.

Hm, but why does Cacti allocate about 400-500 MB memory to create export data with 2,5 MB size ?

[TheWitness]

What was the size of the RRDtool xport? Please advise. I will check on it. Could be a memory leak in PHP you know.

TheWitness

[ralfk]

Size of "rrdtool xport": 2.6 MByte
Size of Cacti export: 3.5 MByte
Size of the rrdtool database file: 4.3 MByte
No. of exported rows: 35000
No of DS per row: 4
Time preset: 1 year
STEP = 900
RRA: 38400 at 1 PDP per row
PHP memory_limit = 512M (384M too less)
rrdtool --maxrows = 40000
Cacti Execution time: ~ 1-2 seconds