Graph export slow, parallel graph_export possible?

Anything that you think should be in Cacti.

Moderators: Developers, Moderators

Post Reply
phibs
Posts: 2
Joined: Wed Sep 23, 2009 10:54 am

Graph export slow, parallel graph_export possible?

Post by phibs »

Graph export slow, parallel graph_export possible? We have many graphs and the export process seems extremely slow especially on an 8 core machine.

ExportDuration: 236.0610
TotalGraphsExported: 5790

It seems that only 1 rrdtool process is ever spawned leading me to believe the process is sequential. Is it possible to make this a parallel effort or is there an external cron-based script that already does this?

TIA,

Bryan[/code]
User avatar
TheWitness
Developer
Posts: 17047
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Post by TheWitness »

Sure it is, but it could crash your box.

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages


For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
phibs
Posts: 2
Joined: Wed Sep 23, 2009 10:54 am

Post by phibs »

lol not if it did X at a time where X is the # of cores or a configurable #? I'd really like it :) Or have it create the graphs in a scratch directory then remove the current dir and rename the scratch dir? There is a 5 minute period where graphs just don't load :(
User avatar
TheWitness
Developer
Posts: 17047
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:

Post by TheWitness »

Here is how you accomplish this:

1) create a export_processes table with the following rows:
pid (int) unsigned not null default '0', starttime timestamp default CURRENT_TIMESTAMP. The primary key is the pid.
2) set a variable "Maximum Concurrent RRD Upates" in the global_settings.php file. Make it a dropdown with from 1-8 as options.
3) in the graph export, query the table, if there are less than X entries in the table use the "exec_background" function to call a tiny script called export_rrdgraph.php.
4) in that script get your current pid, add it to the table, and update the graph, and then remove the pid from the table before exiting.
5) Add some timeout logic to remove stale entries from the export_processes when the timestamp is > X seconds. Maybe add an entry in global_settings.php for what is reasonable, say 10 seconds.

QED.

When done, make a patch and send it to http://bugs.cacti.net

TheWitness
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages


For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests