Slow Page Load - User Experience

[kernelpat]

Greetings. I have a great Cacti implementation and I am so pleased. The page load (user experience) has slowed down and I'm running out of things to tune. I know very little about MySQL or RRDTool and I was hoping for some pointers.

Platform:
- Sunfire X2100/AMD 2Ghz 64bit Opteron/1.5GB RAM/Seagate 250GB 10K-RPM SATA.
- Gentoo (32 bit) v2005.1 stage 3/kernel 2.6.15-gentoo-r1
- Lighttpd 1.3.16 with FCGI wrapping PHP 4.4.0-r4/php-cgi 4.4.0-r5
- Cacti (Gentoo patched) 0.8.6g_p20051023 / Cactid 0.8.6f_p1
- MySQL 4.1.14
- RRDTool 1.2.6-r1

Stats:
- 02/23/2006 09:15:46 AM - SYSTEM STATS: Time:44.5326 Method:cactid Processes:60 Threads:60 Hosts:49 HostsPerProcess:1 DataSources:20915 RRDsProcessed:7003
- 02/23/2006 09:11:03 AM - SYSTEM STATS: Time:62.0613 Method:cactid Processes:60 Threads:60 Hosts:49 HostsPerProcess:1 DataSources:20915 RRDsProcessed:7003
- 02/23/2006 09:05:46 AM - SYSTEM STATS: Time:45.5473 Method:cactid Processes:60 Threads:60 Hosts:49 HostsPerProcess:1 DataSources:20915 RRDsProcessed:7003

* Yup...I have quite a few switches with over 300 interfaces each. I'm only pulling bits in/out and errors/discards (2 graphs per interface.)

So, when I reached 2,500 datasources the page load started slowing down. But this is only noticeable on the graph screen. The console section is still very fast. I created a heading titled "aaa" so it would be at the top of the tree. I then made it the default graph screen and left it empty. Still, right after I login it takes 10-12 seconds to pull up "an empty page?" Moving around the graph screen is an 8-10 second wait--minimum--even when selecting an interface that will pull up 2 graphs only.

So, MySQL and RRDTool are out of the box. I don't know how to measure or tune them. Any thoughts or suggestions will be followed and appreciated. And if an additional machine is the cure...then something to take to my boss would be appreciated also

Any comments on the number of threads/processes would be nice. I maxed out the "SNMP OID's Per SNMP Get Request" to 60.

Thank you,

Kernelpat

[lard]

Hi,

Just thought I would add a "me too" to this - I believe it is RRDTool that is at fault with our systems because as you mentioned above the console interface is fine, just creating the graphs that has slowed down,

I did have a situation last week where a server required reloading because it had stopped creating any graphs - though I panicked slightly and didn't investigate the problem before reloading to fix it,

If it happens again I will get as much info beforehand then post here,

Best,

Lard

[kernelpat]

I found a posting by Rony and The Whitness regarding the number of processes and threads. Looks like I was off by a lot!

http://forums.cacti.net/about11203.html

I changed mine to: 2 processes and 15 threads. This had a small effect on the polling period but has not changed the aforementioned page load issue.

02/23/2006 01:50:43 PM - SYSTEM STATS: Time:41.5373 Method:cactid Processes:2 Threads:15 Hosts:49 HostsPerProcess:25 DataSources:20915 RRDsProcessed:7003
02/23/2006 01:45:52 PM - SYSTEM STATS: Time:50.7988 Method:cactid Processes:2 Threads:15 Hosts:49 HostsPerProcess:25 DataSources:20915 RRDsProcessed:7003
02/23/2006 01:40:43 PM - SYSTEM STATS: Time:41.7680 Method:cactid Processes:2 Threads:15 Hosts:49 HostsPerProcess:25 DataSources:20915 RRDsProcessed:7003
02/23/2006 02:10:44 PM - SYSTEM STATS: Time:42.4861 Method:cactid Processes:2 Threads:15 Hosts:49 HostsPerProcess:25 DataSources:20915 RRDsProcessed:7003

Any additional comments on the number of threads/processes would be appreciated.

Thanks for responding Lard. Anyone have tips for measuring rrdtool performance?

[gandalf]

First, for rrdtool performance disk speed is critical. You may improve this using more cache memory. If you resized your rrd's, this will have negative effects on performance.
Some time ago I wrote an rrdtool stress driver to update some 1000 or 10000 rrds in non-sequential order. My laptop was handling this without any problems, but it will nearly block parallel rrdtool read access.
rrdtool 1.2.6 is outdated and has some errors. rrdtool 1.2.x does support some nice features and is know to work faster than rrdtool 1.0.x, but only for complex graphs like smokeping.
Reinhard