I have noticed that with my setup where I store RRD file on NFS mount point polling performance was very bad.
After some investigation I have found out that SNMP/Script polling was performing very well (within 10s) but rrdtool would take very long time (thousands of seconds) to finish writing data.
The problem here was that rrdtool will write one RRD file at a time and over NFS opening files is slow. With about 7k RRD files this would not scale.
I have solved the problem by parallelizing RRD file write with rrdtool wrapper script:
Code: Select all
#!/usr/bin/python
import zlib
import subprocess
import sys
import time
import os
rrd = "/usr/bin/rrdtool"
process_no = 16
processes = []
def spawn():
processes.append(subprocess.Popen([rrd, "-"], stdin=subprocess.PIPE))
def pipe(process, data):
processes[process].stdin.write(data)
if len(sys.argv) > 1 and sys.argv[1] == "-":
pno = process_no
while pno:
spawn()
pno -= 1
while True:
line = sys.stdin.readline()
if not line:
break
args = line.lstrip().split(" ")
if len(args) < 2 or args[0] != "update":
pipe(0, line)
else:
cmd = args[0]
file = args[1]
hash = zlib.crc32(file)
process = hash % process_no
#print "cmd: ", cmd, "file: ", file, "hash: ", hash, "process no: ", process
pipe(process, line)
else:
os.execlp(rrd, *([rrd] + sys.argv[1:]))With this script used instead of rrdtool (path in settings page) I got my poller run times down to about 90s.
I beleve that it should be easy to rewrite this in PHP and include in Cacti release.
