BIG setup

[sodium]

Hi,

I've allways used cacti in an environments where 1 system is enough as poller/webserver/database, but now I'm asked to design and build a cacti environment for about 1000 devices with an average of 15 graphs per device (some devices have 200 graphs) on average 50 to a 100 people will be using the frontend and all rrd files will be checked (for trend analysis and error reporting) and as a bonus it must be build scalable so if more devices needed to be polled systems can be added....
The OS of choice is Linux.

so in short a big setup...

what I thought we should do:

6 pollers
2 web frontends (behind a L4 swith)
2 mysql servers in multimaster mode
2 analysis systems
1 generally accessable storage device (nfs/gfs, san/iscsi)???

I've seen in the source of cactid there are ways to work with multiple pollers....

Are there other users using a big setup?
Any pointers for what to use as a storage device?
What FileSystem was used and why?

[rony]

Disk IO is the biggest bottle neck, optimize that. SAN that multiple servers can read the same volume would be your best bet.

Mysql optimization will be key, you need to make sure that you have enough committed memory to help it query/update/insert as effecient as possible.

Concerning multiple pollers, well, it's not supporting in 0.8.6. But I suggest you email TheWitness for some help and guidance.

[TheWitness]

All depends on the types of devices. In my current, and soon to be former, environment, we currently have over 900 devices 25000 datasources and 11000 RRD files on a 4 way Hyperthreading 3.16GHz Caranahan with a RAID 5 array of 3 146GByte disks and a hot spare.

On that server, we have the database, the web site, the poller and in addition, we poll about 1700 network devices once every 4 hours with MacTrack.

The server does not get a hugh workout during this period. As Tony spoke, disk I/O has been the major bottleneck to date. We are looking at ways to make it not too much the bottleneck.

This server however, is used for Analysis and not by more than 4-5 concurrent users at a time. So, defining your user experience will be crucial. My final note came principally from my father. He called it the "carpenters creed". He always said, "better long than short".

TheWitness

P.S. The server is Windows2003 Enterprise, 8GBytes of RAM. I'll get a few utilization graphs and post them.

[colm]

hi,

im currently using cricket to monitor about 1000 devices and update over 30000+ rrds. cricket is ok but im getting tired of rejigging crons to get all polling completed in time.

i would consider changing over to cacti if i though it would be scalable enough and simpler to manage.

so some questions ...
0) what is the largest cacti deployment you guys know of?
1) can multiple polling servers be setup in cacti? does this work effectively?
2) can multiple poller process be setup in cacti? does this work effectively?
3) do pollers work entirely in parallel? in cricket device groups can be setup to poll in parallel but individual device and tagets polls are done synchronously.

colm

[rony]

I have seen larger installations then the one you propose.

In your situation, you would use Cactid as the poller. Larry (TheWitness) can give you more information about the settings, but off the top of my head, I would say 2 * cpu count for processes, and 20 threads and 40 OID requests at a time.

Mulitple pollers isn't supported, but for such a large installation, I would suggest a machine dedicated to polling. Idea setup, 3 server, one for database, one for web and one for poller. The poller server would contain all the rrd files and have a fast I/O sub system. The web server would simply nfs mount the rrd files, because that is a readonly process.

The poller, when configured for more than one process will work in parallel. You don't have control of which host each process will do, but it just split up evenly between hosts.

[netlogic]

I would appreciate more detail as to the setup of the scaled deployment a.k.a big setp.
Sodium if you got detailed answers or Thewitness or Ron.