using rrdtool method for a real database

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Developers, Moderators

Post Reply
matcraig
Posts: 6
Joined: Tue Jun 05, 2007 5:32 pm

using rrdtool method for a real database

Post by matcraig »

Has anyone thought about using the rrdtool method of storing data in a real database, such as mysql? Let me elaborate.

I have pretty much come to the conclusion that I can't use cacti to graph an environment of my size (1100+ hosts, 300k data sources) with a 5 minute polling interval. Polling isn't a problem; cactid can poll my machines very fast; its the processing of the datasources. Even with boost it just takes too long and takes up too much resources to process the rrd files on the hard drive to be feasible. So, I thought that there has got to be a way to eliminate this bottleneck.

The rrd method is nice; we can store data in a file that doesn't increase in size even after you update it because it averages older data as new data gets added. The bad part about this, is that since data is stored in single files on the hard drive it has to have some hard disk activity whenever the file is processed. Combine thousands of hard disk updates at once and things really start to slow down.

Would it be possible to combine the strengths of the rrd method and mysql (or other dbs)? Could we use the rrd method of updating data but use mysql to store it? This way, I would think the database size would remain fixed and it there wouldn't be massive amounts of writes the hard disk.

Please let me know if this is not clear.
User avatar
knebb
Cacti User
Posts: 138
Joined: Tue Sep 19, 2006 11:29 am

Re: using rrdtool method for a real database

Post by knebb »

Yohoo!

Not really a helpful approach to your issue, but have you tried some performance tuning options?

-use RAID1 for the rra/
-FS tuning
+ReiserFS for small files like the rras
+ext2 instead of ext3
+if ext3 external journal (or some journal tuning)
+mount with "noatime"
-A lot of RAM, so the rra could be stored in memory (approx 16GB for your site perhaps?)

Just an idea. And, use Cacti to monitor your Cacti server :wink:
User avatar
gandalf
Developer
Posts: 22383
Joined: Thu Dec 02, 2004 2:46 am
Location: Muenster, Germany
Contact:

Re: using rrdtool method for a real database

Post by gandalf »

matcraig wrote:I have pretty much come to the conclusion that I can't use cacti to graph an environment of my size (1100+ hosts, 300k data sources) with a 5 minute polling interval. Polling isn't a problem; cactid can poll my machines very fast; its the processing of the datasources. Even with boost it just takes too long and takes up too much resources to process the rrd files on the hard drive to be feasible. So, I thought that there has got to be a way to eliminate this bottleneck.
Wow, that's quite huge. The 1100+ hosts are not the problem, but the 300k datasources will. lately, there has been some progress on rrdtool using optimized hard disk access (fadvise) to prevent read ahead cache usage that is nonsense in case of rrdtool. On the rrdtool list, users have posted to run about 300k ds with success.
Reinhard
Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests