Slow over a WAN

[slow-on-wan]

The Problem:
When I add in hosts at remote sites, my total time takes up to 55 seconds. I have Cacti run every minute & I need that level of granularity. I've had to disabled many remote hosts/switches moment to avoid rrd corruption from an over-run. Only a fourth of the desired remote hosts have been added. The average host time is 11.7ms. The top end is around 60ms for 9 hosts/switches at a site many states away. The max for any one hosts is 77ms. I can add in more local hosts/switches and things are ok. Once I add in remote hosts/switches, total time quickly go downhill.

The System:
It is a brand new 8 core system (16 hyperthreaded), 12GB RAM, that has a 4 SSD HD(s) RAID5 slice 100% dedicated to the /rrd folder. I've tested a dry run of all of my RRDs at once. This slice/folder has a new RRD file write speed of ~440MB/sec (10x faster than a typical SATA II drive). From the system log, here is a typical run with many remote hosts/switches disabled:
SYSTEM STATS: Time:29.4402 Method:cmd.php Processes:64 Threads:N/A Hosts:71 HostsPerProcess:2 DataSources:13756 RRDsProcessed:2819
I have a negative speed advantage with spine. In fact, I found spine to be incredibly buggy once I went past 50 hosts.

So here are my questions:
Is there any way to WAN optimize Cacti?
I suspect that any new hosts added are polled last (FIFO). Without destroying my current graphs, how do I get newly added hosts to be polled first? Or in any order I desire?

Thanks you for your time!

[gandalf]

At least use spine instead of cmd.php to use at least some of your cpu's
R.

[slow-on-wan]

Since I migrated everything to this new box, I re-downloaded spine, ./configure & make, etc. Re-pointed my spine path to the correct path. Try to run it & I get:
http://bugs.cacti.net/view.php?id=1815

So I go down the path of PIA installation due to its suggestion. So I get:
[Fri Feb 25 18:54:34 2011] [error] [client xxx.xxx.xxx.xxx] PHP Fatal error: Call to undefined function api_plugin_hook() in /cacti/auth_login.php on line 284
Can't even log back into Cacti :-(
So I look around & I see this post for TODAY:
http://forums.cacti.net/viewtopic.php?f=2&t=41310

Even try to run the poller manually & get this one line in the log:
PHP Fatal error: Call to undefined function api_plugin_hook() in /cacti/poller.php on line 111

...and I think: YES! I have a backup of everything, restore to get out of the rabbit's hole & I can go out drinking tonight! Oh... and this effort was all an effort of appeasement as the box's CPU is hovering around 1.25% avg & the "load" is around 1.53.

PLEASE at least answer how I can try to get certain hosts to be polled first vs the normal FIFO method.

[gandalf]

I'd like to tackle those errors first. IMHO they are a result of incomplete PIA installation. I don't see the relation to the freebsd topic you've linked to.

BTB, moving specific hosts to a specific place in the polling sequence is not supported by cacti.
R.

[slow-on-wan]

Fixed it! Changed the cmd.php "Maximum Concurrent Poller Processes" to 256 threads. Problem solved.