Cacti performance & graph problem

[geertn444]

Hi,

Our cacti system is already running +/- 3 years with no real problems, but lately, the graph are frequently showing gaps. The user interface is also getting slower and when the gaps happen, Weathermap graphs stay completely empty (grey lines).
I have done some troubleshooting and can say the following:
* The poller takes 115-150 seconds to complete. 167 hosts. so well within 300 seconds poll cycle.
* Logging is set to MEDIUM. I don't see anything strange in the log.

* Using TOP, i have noticed that our %wa utilisation is almost always high: > 50% and sometimes running at 100% (even outside the poll cycle)

top - 10:46:59 up 28 days, 17:33, 1 user, load average: 13.14, 10.25, 8.42
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 18.8%us, 3.5%sy, 0.0%ni, 0.5%id, 76.7%wa, 0.2%hi, 0.3%si, 0.0%st
Mem: 1028236k total, 1016312k used, 11924k free, 19984k buffers
Swap: 2650684k total, 61812k used, 2588872k free, 245920k cached

This indicates an "IO Bound" system, but i am no Linux specialist. Does this indicate a disk problem, maybe fragmentation ?

* This is the disk space free:
df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 264445 143906 106886 58% /
tmpfs 514116 0 514116 0% /lib/init/rw
udev 10240 76 10164 1% /dev
tmpfs 514116 0 514116 0% /dev/shm
/dev/sdb1 48062440 19012788 26608176 42% /var
/dev/sda6 2885780 69904 2669288 3% /tmp
/dev/sda9 65962184 2061980 60549452 4% /home
/dev/sda5 4806904 548088 4014632 13% /usr

Any ideas on how to get the system back running smoothly ?

[Linegod]

Just to be complete, attach the output from Console/System Utilities/Technical Support

Attach the outputs of the following commands as well:

Code: Select all

# ps axfvu >cacti_support.txt
# echo "##END##" >>cacti_support.txt
# cat /etc/fstab >>cacti_support.txt
# echo "##END##" >>cacti_support.txt
# ls -la /path/to/your/rra/ >>cacti_support.txt

[geertn444]

That will be too large to attach here. (will it preview?)

[geertn444]

Weathermap is also giving the following error messages:

e/rra/lvlm1ab_traffic_in_11047.rrd on config line 132 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:08 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/sl1d_traffic_in_9886.rrd on config line 125 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:08 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/wav0aa_traffic_in_5844.rrd on config line 117 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:08 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/rg_traffic_in_858.rrd on config line 110 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:08 PM - WEATHERMAP: Poller[0] ReadData: LIN, target: /usr/share/cacti/site/rra/r322e_traffic_in_11722.rrd on config line 103 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:08 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/waaa_traffic_in_5629.rrd on config line 95 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:07 PM - WEATHERMAP: Poller[0] RRD ReadData: Data is not most recent entry (3) for link: /usr/share/cacti/site/rra/spt2_traffic_in_12370.rrd
07/17/2009 05:09:06 PM - WEATHERMAP: Poller[0] RRD ReadData: Data is not most recent entry (3) for link: /usr/share/cacti/site/rra/wimpls_traffic_in_11985.rrd
07/17/2009 05:09:06 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/la5mpls_traffic_in_11731.rrd on config line 344 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:06 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/mompls_traffic_in_11521.rrd on config line 337 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:06 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/vilmpls_traffic_in_11514.rrd on config line 330 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:05 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/milmpls_traffic_in_11506.rrd on config line 323 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:04 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/gasmpls_traffic_in_11465.rrd on config line 316 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:04 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/sptbx_traffic_in_12322.rrd on config line 309 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:04 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/spthtn_traffic_in_5504.rrd on config line 302 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:04 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/sptde2_traffic_in_3446.rrd on config line 295 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:03 PM - WEATHERMAP: Poller[0] RRD ReadData: Data is not most recent entry (3) for link: /usr/share/cacti/site/rra/s_traffic_in_5524.rrd
07/17/2009 05:09:03 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/spe_traffic_in_2259.rrd on config line 281 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:03 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/spe2_traffic_in_5074.rrd on config line 274 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:03 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/spt_traffic_in_5088.rrd on config line 267 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:03 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/sp_traffic_in_390.rrd on config line 260 had no valid data, according to WeatherMapDataSource_rrd
07/17/2009 05:09:03 PM - WEATHERMAP: Poller[0] ReadData: LINK , target: /usr/share/cacti/site/rra/st_traffic_in_5117.rrd on config line 253 had no valid data, according to WeatherMapDataSource_rrd

[taliz]

That looks very much like the problem is with the IO being too slow.
I would suggest looking into optimizing the filesystem and IO system.
See http://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD

I have done this myself and lowered our %wa from close to 100% to almost 0 on our Centos 5 i386 Cacti-server.
I experimented some and what I ended up doing is this:
echo "16" > /proc/sys/vm/dirty_background_ratio
echo "99999" > /proc/sys/vm/dirty_expire_centisecs
echo "99999" > /proc/sys/vm/dirty_expire_centisecs
echo "9999" > /proc/sys/vm/dirty_writeback_centisecs
echo "512" > /sys/block/cciss\!c0d0/queue/nr_requests

(You can place those in /etc/rc.local to have it set when it boots)
The last line obviously needs to match whatever your raiddriver is.
You should play some with the values and see what works best for you.
Also, you should look into tuning your filesystem. I use ext3 and ended up mounting the rrd partition with the following in /etc/fstab:
/dev/cciss/c0d0p7 /var ext3 defaults,noatime,nodiratime,data=writeback,commit=120 1 2

"man mount" will tell you the mount options.