Maximum count of cacti graphs.

[nuran]

Cacti Version 0.8.7g
OS: FreeBSD 8.1-RELEASE amd64
SNMP Version NET-SNMP version: 5.5
RRDTool Version RRDTool 1.4.x
Hosts 192
Graphs 19999

Memory 4096mb
HDD - RAID5 - 3x- HP SAS 74gb 10K
CPU - X5140DI

Why cacti don't write graphs. Everything is ok. but when show gstat

Code: Select all

  L(q)  ops/s    r/s   kBps       ms/r    w/s   kBps       ms/w        %busy      Name
    0      0      0      0        0.0       0      0        0.0        0.0|   acd0
    6    482    388  14852      16.2     94   1519    125.4  100.3|    da0
    6    482    388  14852      16.2     94   1519    125.4  100.3|    da0s1
    0      0      0      0        0.0       0      0        0.0       0.0|    da0s1a
    0      0      0      0        0.0       0      0        0.0       0.0|    da0s1b
    0      0      0      0        0.0       0      0        0.0       0.0|    da0s1d
    0      0      0      0        0.0       0      0        0.0       0.0|    da0s1e
    6    482    388  14852      16.2     94   1519    125.5  100.3|    da0s1f

and in the cacti log shows that

01/26/2013 04:30:00 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 18727, Data Sources: traffic_in(DS[85]), traffic_out(DS[85]), traffic_in(DS[86]), traffic_out(DS[86]), traffic_in(DS[87]), traffic_out(DS[87]), traffic_in(DS[88]), traffic_out(DS[88]), traffic_in(DS[89]), traffic_out(DS[89]), traffic_in(DS[90]), traffic_out(DS[90]), traffic_in(DS[91]), traffic_out(DS[91]), traffic_in(DS[92]), traffic_out(DS[92]), traffic_in(DS[93]), traffic_out(DS[93]), traffic_in(DS[94]), traffic_out(DS[94]), traffic_in(DS[95]), Additional Issues Remain. Only showing first 20





What you recomended me? what i must do? which raid i must creat? Have a limit for graph in cacti?

our rra folder size 80gb

[gandalf]

The number of data sources is not a problem at all.
Are you using 1 minute polling?
It is know, that rrdtool causes disk to be busy. Either faster disk/SAN would help, or, better, disk update caching. This feature is provided by the BOOST plugin.
R.

[nuran]

Poller Interval 5 minutes
Cron Interval 5 minutes

After installing BOOTS plugin there is no changes. Everithing is the same.

[gandalf]

Which parameters did you select for BOOST?
Are you sure, that BOOST is running?
Please post output of "cacti/utilities.php?action=view_boost_status", to be found at "System Utilities"
R.

[nuran]

Current Boost Status
Boost On Demand Updating: Disabled
Total Data Sources: 39947
Boost Storage Statistics
Database Engine: MyISAM
Current Boost Tables Size: 0 Bytes
Avg Bytes/Record: 0 Bytes
Max Allowed Boost Table Size Unlimited
Estimated Maximum Records: Unlimited Records
Runtime Statistics
Last Start Time: 2013-01-28 12:38:10
Last Run Duration: 37 minutes 33 seconds
RRD Updates: 528585
Peak Poller Memory: 80.13 MBytes
Detailed Runtime Timers: RRDUpdates:528585 TotalTime:2253 get_records:489.24 results_cycle:1498.22 rrd_filename_and_template:21.74 rrd_lastupdate:205.47 rrdupdate:1265.03 delete:246.79 timer_overhead:~1
Max Poller Memory Allowed: 1024 MBytes
Run Time Configuration
Update Frequency: N/A
Next Start Time: 2013-01-28 13:38:10
Maximum Records: 1000000 Records
Maximum Allowed Runtime: 40 Minutes
Boost Server Details
Server Config Status: Enabled
Multiprocess Server: Multiple Process
Update Timeout: 2 Seconds
Server/Port: localhost@9050
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/local/bin/rrdtool
Image Caching
Image Caching
Image Caching Status: Disabled
Cache Directory:
Cached Files: WARNING: N/A
Cached Files Size: WARNING: Directory Does NOT Exist!!


Now we are disabled BOOTS. But when it was enabled


But when it was in enabled

sockstat -l4 | grep 9050

nothing showing

[gandalf]

Current Boost Status
Boost On Demand Updating: Disabled
Update Frequency: N/A

You usually will want to enable on demand updating, else your graphs won't show current data.
And then, you will want to have some meaningful "update frequency" set.
These are our statistics:

Code: Select all

Current Boost Status
Boost On Demand Updating:	Idle
Total Data Sources:	94970
Pending Boost Records:	935464
Archived Boost Records:	0
Total Boost Records:	935464
Boost Storage Statistics
Database Engine:	MyISAM
Current Boost Tables Size:	32 MBytes
Avg Bytes/Record:	36 Bytes
Max Allowed Boost Table Size:	Unlimited
Estimated Maximum Records:	Unlimited Records
Runtime Statistics
Last Start Time:	2013-01-28 15:56:50
Last Run Duration:	12 minutes 29 seconds (10% of update frequency)
RRD Updates:	2014573
Peak Poller Memory:	111.23 MBytes
Detailed Runtime Timers:	RRDUpdates:2014573 TotalTime:749 get_records:81.2 results_cycle:586.26 rrd_filename_and_template:88.42 rrd_lastupdate:65.25 rrdupdate:382.62 delete:77.53 timer_overhead:~8
Max Poller Memory Allowed:	1024 MBytes
Run Time Configuration
Update Frequency:	2 Hours
Next Start Time:	2013-01-28 17:56:50
Maximum Records:	2000000 Records
Maximum Allowed Runtime:	1.5 Hours

having an update time of 37 minutes while the interval only has 40 minutes is bad. First, 37 minutes for only 500,000 updates on less than 20,000 data sources is quite slow. Filesystem speed seems to be rather low. Perhaps you should increase to 1,000,000 records for the update, allowing an interval of 2h max. It is not expected, that 1,000,000 updates take 2* 37 minutes.
R.

[nuran]

Code: Select all

Boost On Demand Updating:	Running
Total Data Sources:	39386
Pending Boost Records:	77833
Archived Boost Records:	357234
Total Boost Records:	435067
Boost Storage Statistics
Database Engine:	MEMORY
Current Boost Tables Size:	231 MBytes
Avg Bytes/Record:	557 Bytes
Max Record Length:	137 Bytes
Max Allowed Boost Table Size:	4 GBytes
Estimated Maximum Records:	6822137 Records
Runtime Statistics
Last Start Time:	2013-02-07 9:10:11
Last Run Duration:	44 minutes 48 seconds (75% of update frequency)
RRD Updates:	470028
Peak Poller Memory:	55.31 MBytes
Detailed Runtime Timers:	RRDUpdates:470028 TotalTime:2689 get_records:8.52 results_cycle:2659.55 rrd_filename_and_template:3.23 rrd_lastupdate:282.06 rrdupdate:2368.72 delete:0.91 timer_overhead:~1
Max Poller Memory Allowed:	1024 MBytes
Run Time Configuration
Update Frequency:	1 Hour
Next Start Time:	2013-02-07 10:10:11
Maximum Records:	1000000 Records
Maximum Allowed Runtime:	40 Minutes
Boost Server Details
Server Config Status:	Enabled
Multiprocess Server:	Multiple Process
Update Timeout:	2 Seconds
Server/Port:	localhost@9050
Authorized Update Web Servers:	127.0.0.1
RRDtool Binary Used:	/usr/local/bin/rrdupdate
Image Caching
Image Caching Status:	Disabled
Cache Directory:	
Cached Files:	WARNING: N/A
Cached Files Size:	WARNING: Directory Does NOT Exist!!

After after setting cacti everything it ok.

but can't optimization why update frequency don't be 100%?

Last Run Duration: 44 minutes 48 seconds (75% of update frequency)


and why some graph don't update on time when we looking this graph?

[gandalf]

but can't optimization why update frequency don't be 100%?

Last Run Duration: 44 minutes 48 seconds (75% of update frequency)

percentage should be low. You only have 25% left until BOOST overrun will occur.
R.

[nuran]

but can't optimization why update frequency don't be 100%?

Last Run Duration: 44 minutes 48 seconds (75% of update frequency)

percentage should be low. You only have 25% left until BOOST overrun will occur.
R.

Thanks for help gandalf!

Now I installed new cacti server and our hardware configuration is:
Intel(R) Xeon(R) CPU E5420 @ 2.50GHz
10Gb RAM
3 of 10Gb memory allocated for MySQL partition
7x147G SAS 10k (RAID 5)

we set the cacti from this topic http://forums.cacti.net/viewtopic.php?f=6&t=29707


Every things work well. but we have a little bit. when we enter graphs tree page cacti can't on demand update every graphs. and lost this traffic which can't update.

this is our boots setting

boots_setting.JPG (204.54 KiB) Viewed 2489 times

and boots log

boots_log.JPG (167.65 KiB) Viewed 2489 times

which changes needs do for fix this problem?

[gandalf]

Stats are way better now. Please find a set of templates that report BOOST stats as Cacti graphs at 4th link of my sig. So you will see the change of stats and not only last value.
Regarding tuning, you should increase "MAX data source items per pass" and "MAX argument length" if you OS allows for that. This will increase effciciency of BOOST.

My personal experience when using BOOST is, that when a huge tree is loaded, it may indeed happen, that "some" graphs do not hold latest data. But on a refresh, latest data _always_ has been available. I've not seen permanent dropouts when using BOOST
R.

[nuran]

Stats are way better now. Please find a set of templates that report BOOST stats as Cacti graphs at 4th link of my sig. So you will see the change of stats and not only last value.
Regarding tuning, you should increase "MAX data source items per pass" and "MAX argument length" if you OS allows for that. This will increase effciciency of BOOST.

My personal experience when using BOOST is, that when a huge tree is loaded, it may indeed happen, that "some" graphs do not hold latest data. But on a refresh, latest data _always_ has been available. I've not seen permanent dropouts when using BOOST
R.

We set templates that report BOOST stats but they useless for getting information about bug , maybe we dont know how to do troubleshooting.

"MAX data source items per pass" we set as 50000 because we have now 40433 data sourses

SYSTEM STATS: Time:33.1862 Method:spine Processes:8 Threads:10 Hosts:199 HostsPerProcess:25 DataSources:40433 RRDsProcessed:0




"MAX argument length" we set as 65K, because our cacti installed on FreeBSD 9.1 amd64 platform and cacti say that:

When boost sends update commands to RRDtool, it must not exceed the operating systems Maximum Argument Length. This varies by operating system and kernel level. For example: Windows 2000 <= 2048, FreeBSD <= 65535, Linux 2.6.22-- <= 131072, Linux 2.6.23++ unlimited.



In our case not same as your we have ~50% dropouts on same page tree graphs.


Which changes i need do for fix this problem????????

[nuran]

where are you gandalf? answer to my question Please)

[gandalf]

"MAX data source items per pass" we set as 50000 because we have now 40433 data sourses

This in fact disables BOOST as it makes it run on virtually each polling cycle. You should increase to a value that's 10-20 fold the number of data sources.

"MAX argument length" we set as 65K, because our cacti installed on FreeBSD 9.1 amd64 platform

I'm not used to FReeBSD, but when that's the max argument length, then it's fine.
R.