Has anyone thought about using the rrdtool method of storing data in a real database, such as mysql? Let me elaborate.
I have pretty much come to the conclusion that I can't use cacti to graph an environment of my size (1100+ hosts, 300k data sources) with a 5 minute polling interval. Polling isn't a problem; cactid can poll my machines very fast; its the processing of the datasources. Even with boost it just takes too long and takes up too much resources to process the rrd files on the hard drive to be feasible. So, I thought that there has got to be a way to eliminate this bottleneck.
The rrd method is nice; we can store data in a file that doesn't increase in size even after you update it because it averages older data as new data gets added. The bad part about this, is that since data is stored in single files on the hard drive it has to have some hard disk activity whenever the file is processed. Combine thousands of hard disk updates at once and things really start to slow down.
Would it be possible to combine the strengths of the rrd method and mysql (or other dbs)? Could we use the rrd method of updating data but use mysql to store it? This way, I would think the database size would remain fixed and it there wouldn't be massive amounts of writes the hard disk.
Please let me know if this is not clear.
using rrdtool method for a real database
Moderators: Developers, Moderators
Re: using rrdtool method for a real database
Yohoo!
Not really a helpful approach to your issue, but have you tried some performance tuning options?
-use RAID1 for the rra/
-FS tuning
+ReiserFS for small files like the rras
+ext2 instead of ext3
+if ext3 external journal (or some journal tuning)
+mount with "noatime"
-A lot of RAM, so the rra could be stored in memory (approx 16GB for your site perhaps?)
Just an idea. And, use Cacti to monitor your Cacti server
Not really a helpful approach to your issue, but have you tried some performance tuning options?
-use RAID1 for the rra/
-FS tuning
+ReiserFS for small files like the rras
+ext2 instead of ext3
+if ext3 external journal (or some journal tuning)
+mount with "noatime"
-A lot of RAM, so the rra could be stored in memory (approx 16GB for your site perhaps?)
Just an idea. And, use Cacti to monitor your Cacti server

- gandalf
- Developer
- Posts: 22383
- Joined: Thu Dec 02, 2004 2:46 am
- Location: Muenster, Germany
- Contact:
Re: using rrdtool method for a real database
Wow, that's quite huge. The 1100+ hosts are not the problem, but the 300k datasources will. lately, there has been some progress on rrdtool using optimized hard disk access (fadvise) to prevent read ahead cache usage that is nonsense in case of rrdtool. On the rrdtool list, users have posted to run about 300k ds with success.matcraig wrote:I have pretty much come to the conclusion that I can't use cacti to graph an environment of my size (1100+ hosts, 300k data sources) with a 5 minute polling interval. Polling isn't a problem; cactid can poll my machines very fast; its the processing of the datasources. Even with boost it just takes too long and takes up too much resources to process the rrd files on the hard drive to be feasible. So, I thought that there has got to be a way to eliminate this bottleneck.
Reinhard
Who is online
Users browsing this forum: No registered users and 5 guests