Disk IO performance with rrdtool

[chryzo]

I just switched to spine for (one of) my Cacti installation, hoping to get better runtime.
With cmd.php:

Code: Select all

07/30/2009 04:16:53 AM - SYSTEM STATS: Time:111.8731 Method:cmd.php Processes:10 Threads:N/A Hosts:161 HostsPerProcess:17 DataSources:8753 RRDsProcessed:4134

After som tuning and debugging (buggy php script), I got spine running:

Code: Select all

07/30/2009 01:52:23 PM - SYSTEM STATS: Time:141.8422 Method:spine Processes:4 Threads:30 Hosts:161 HostsPerProcess:41 DataSources:8753 RRDsProcessed:4134

NOTE: The devices in this installation is polled over WAN, so the runtime tends to go up and down
Not the improvement I was looking for, so I've spent the best of today trying to figure out why Cacti is so slow.
Looking at top while the poller runs, it seems like the actual polling is quite quick (~60sec), but rrdtool keeps the poller running for another minute or so. Running iostat while the poller runs confirms something is wrong - I only get 1MB/sec write tops (and 100% util). Trying to write 16GB random data with dd, i get an average of 74MB/sec write speed. I've moved the rra catalogue to a new disk, and got slightly better runtime, but still >1MB/sec write speed.
Is this Cacti hitting the limit of my disks (two 250 GB SATA 7.2k rpm, no RAID - one mounted as / and one as rra), or is there something else I can do?

[gandalf]

Yep, disk activity is the limiting issue.
using rrdtool 1.2.23 or up and fadvise-capable Linux kernels will avoid read-ahead overhead that makes no sense with rrdtool.
You will want to use boost to improve beyond that
Reinhard

[chryzo]

I fail at google - this is an old issue with RRDTool:
http://net.doit.wisc.edu/~dwcarder/rrdcache/
Ok - so boost, new disks, new OS (x64) and more RAM is the way to go. Or perhaps try to get RRD 1.2.23 installed first.

[gandalf]

RRDCached has been implemented with 1.3.x
But there are some drawbacks using 1.3.x in large environments: But you should judge on your own
Reinhard

[chryzo]

Moving the RRA's to RAM-disk made some difference
Runtime now:

Code: Select all

09/03/2009 03:25:15 PM - SYSTEM STATS: Time:14.5538 Method:spine Processes:8 Threads:20 Hosts:160 HostsPerProcess:20 DataSources:8741 RRDsProcessed:4142 

Tossed the old server, stole a new one with more CPU's (2x DualCore Xeon) and all the RAM I could find (12GB). Switched to 64-bit OS, and created a tmpfs file-system to store the RRA's (and some backup / sync scripts to prevent data-loss).
I know I could have used Boost, but this was sort of a test of how much performance-gain we could get without pulling out all the aces.

[psyber]

Is anybody using rrdcached with cacti?

I'm running cacti on a VM and the I/O is killing performance even to local disk. At this rate it won't be long before I have to migrate off the VM for physical hardware (Ewwww!).

I see 1.4 is out anyone have success or failure with that?

I'm currently running a snapshot release of 1.3.something and until I passed the 2000 data source mark I didn't see any serious issues with disk I/O (that is no one was complaining yet) I was going to try updating from the snap to something stable but now I see rrdcached and I'm thinking that's the way to go and if there's a possibility of 1.4 working and all the cool stuff to come from that branch that would be the cherry on top.

[gandalf]

We do run rrdcached on our nagios installation successfully, but not on cacti. We do use boost instead which works fine since quite a long time. Both are aiming at quite the same: reduce I/O
R.