Target:
- 600+ hosts, 70 000+ data sources, 300 000+ data items
- one week per-5-minutes statistics in RRA
- 5 minutes poller interval
My hardware configuration is:
- 2 x Quad-core Intel Xeon 5440
- 16Gb RAM
- 4x147G SAS 15k (RAID10 or RAID6)
Software:
MySQL
0) You will have to use dedicated MySQL instance for single database. I use MySQL daemon running on the same host that cacti runs.
1) Migrate to InnoDB to be able to use row locks.
2) Create indexes: Default cacti's scheme is lacking indexes. Aplly those mentioned in this tweak
3) Place full DB into RAM. That is, on memory disk. Since DB is used as configuration storage that is roughly constant and volatile storage of polled values it can be done. It will give a considerable boost both for webinterface and polling. 2...3Gb memory disk will be convenient for 300k data source items.
For FreeBSD recipe is to add following line into fstab
Code: Select all
md /base mfs rw,-s3g,-m0,noatime 0 0- backup
Code: Select all
#!/bin/sh PATH="/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin:/usr/bin:/bin" cdate="$(date -j '+%H-%M')"; folder_date="$(date -j '+%Y-%m-%d')"; backupdir="/opt/backup/cacti/$folder_date" [ -d $backupdir ] || mkdir -p $backupdir; latest_dump_filename="$backupdir/../db-latest.tbz" backup_filename="$backupdir/db-$cdate.tar" backup_memory_root="/backup-base" backup_memory_dir="$backup_memory_root/mysql" set -e date # stage 1. backup DB [ -d $backup_memory_dir ] && rm -rf $backup_memory_dir /usr/local/bin/innobackupex-1.5.1 --no-timestamp $backup_memory_dir /usr/local/bin/innobackupex-1.5.1 --apply-log $backup_memory_dir tar -C $backup_memory_root -cf - . | /usr/local/bin/7z a $backup_filename.bz2 -si -tbzip2 -mmt=5 rm -rf $backup_memory_dir # stage 1a. make a latest-symlink for a fresh-generated tarball [ -h $latest_dump_filename ] && rm $latest_dump_filename ln -s $backup_filename.bz2 $latest_dump_filename - restore
Code: Select all
#!/bin/sh PATH="/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin:/usr/bin:/bin" backupdir=/var/backup/cacti latest_dump_filename="$backupdir/db-latest.tbz" echo "Restoring last MySQL dump" /usr/local/bin/7z x -so $latest_dump_filename | tar -C /base -xpf -
- restore latest SQL dump just after bringing up MySQL server during boot
- backup DB just before stopping MySQL daemon
Code: Select all
>cat /usr/local/etc/rc.d/cacti_mysql
#!/bin/sh
#
# PROVIDE: cacti_mysql
# BEFORE: mysql
. /etc/rc.subr
name="cacti_mysql"
rcvar=`set_rcvar`
load_rc_config $name
: ${cacti_mysql_enable="NO"}
command="/path/to/cacti-mysql-unpack.sh"
command_args=""
run_rc_command "$1"
Code: Select all
cacti_mysql_enable="YES"
[ "X$_name" = "Xmysql" ] && {
stop_precmd="sh /path/to/backup-cacti.sh"
}
#wait till cacti's mysql is dumped from memory to disk
rcshutdown_timeout="120"
Code: Select all
[mysqld]
skip-locking
key_buffer = 512M
query_cache_size = 128M
max_allowed_packet = 16M
table_cache = 512
sort_buffer_size = 128M
net_buffer_length = 8K
read_buffer_size = 1M
read_rnd_buffer_size = 32M
myisam_sort_buffer_size = 8M
max_heap_table_size = 4G
tmp_table_size=1G;
log_slow_queries
long_query_time = 2
log_long_format
innodb_buffer_pool_size = 256M
5) Convert poller_output to ENGINE=MEMORY (text fields can be converted to varchar(32...255)), poller_output_boost ENGINE=MyISAM ROW_FORMAT=FIXED (ROW_FORMAT has to be set in SQL dump in CREATE TABLE sequence, so you will have to perform dump|restore procedure with editing SQL dump). Make sure you have converted poller_output_boost.output into varchar(32...255) or ROW_FORMAT will be silently ignored by MySQL.
6)Put backup script into cron with 2-3 hours interval.
Cacti & Spine
- turn off all max_execution_time, increase memory_limit to 1G or so in all cacti's scripts (`grep -R' will help)
- install spine in honor of cmd.php, let it use 1.2...1.4x$no_cpus threads. In my case it is 10 threads.
- install plugin architecture
- install boost plugin. '1 hour' boost update interval is a good starting point.
- apply patches patch1, patch2
Code: Select all
11/03/2008 06:36:17 PM - SYSTEM STATS: Time:76.5106 Method:spine Processes:1 Threads:10 Hosts:535 HostsPerProcess:535 DataSources:222959 RRDsProcessed:0
11/03/2008 06:31:14 PM - SYSTEM STATS: Time:73.7684 Method:spine Processes:1 Threads:10 Hosts:535 HostsPerProcess:535 DataSources:222959 RRDsProcessed:0
11/03/2008 06:26:05 PM - SYSTEM STATS: Time:64.7133 Method:spine Processes:1 Threads:10 Hosts:535 HostsPerProcess:535 DataSources:222959 RRDsProcessed:0
11/03/2008 06:21:52 PM - SYSTEM BOOST STATS: Time:1229.8073 RRDUpdates:2673120
11/03/2008 06:21:13 PM - SYSTEM STATS: Time:69.5011 Method:spine Processes:1 Threads:10 Hosts:535 HostsPerProcess:535 DataSources:222959 RRDsProcessed:0
11/03/2008 06:16:15 PM - SYSTEM STATS: Time:72.1974 Method:spine Processes:1 Threads:10 Hosts:535 HostsPerProcess:535 DataSources:222959 RRDsProcessed:0
