Graph export slow, parallel graph_export possible?

[phibs]

Graph export slow, parallel graph_export possible? We have many graphs and the export process seems extremely slow especially on an 8 core machine.

ExportDuration: 236.0610
TotalGraphsExported: 5790

It seems that only 1 rrdtool process is ever spawned leading me to believe the process is sequential. Is it possible to make this a parallel effort or is there an external cron-based script that already does this?

TIA,

Bryan[/code]

[TheWitness]

Sure it is, but it could crash your box.

TheWitness

[phibs]

lol not if it did X at a time where X is the # of cores or a configurable #? I'd really like it Or have it create the graphs in a scratch directory then remove the current dir and rename the scratch dir? There is a 5 minute period where graphs just don't load

[TheWitness]

Here is how you accomplish this:

1) create a export_processes table with the following rows:
pid (int) unsigned not null default '0', starttime timestamp default CURRENT_TIMESTAMP. The primary key is the pid.
2) set a variable "Maximum Concurrent RRD Upates" in the global_settings.php file. Make it a dropdown with from 1-8 as options.
3) in the graph export, query the table, if there are less than X entries in the table use the "exec_background" function to call a tiny script called export_rrdgraph.php.
4) in that script get your current pid, add it to the table, and update the graph, and then remove the pid from the table before exiting.
5) Add some timeout logic to remove stale entries from the export_processes when the timestamp is > X seconds. Maybe add an entry in global_settings.php for what is reasonable, say 10 seconds.

QED.

When done, make a patch and send it to http://bugs.cacti.net

TheWitness