How cacti agree with large amount of data sorces and graphs?
1. Is cmd.php can really gather information for all configured data sorces within 5 minutes, if there are several hundreds of DS's?
2. Displaing list of graphs, ds's took so long time if number of entries is above 100.
Probably it must be paged.
3. MySQL server load is 99% high.
_________________
Igor N Indick
JSC Doris
<font size=-1>[ This Message was edited by: rootik on 2002-02-25 01:49 ]</font>
cacti scalability (was cacti performance considerations)
Moderators: Developers, Moderators
>On 2002-02-06 16:17, raX wrote:
>Performance should improve with 0.6.5 as >SNMP is now build in.
I think Get SNMP network data script should
not use information about SNMP interfaces from database. Router reload causes complete
data gathering malfunction. MRTG is a good example.
It seems to me, data gathering process should be divided for several parallel processes. Each process cannot run more than 5 min.
BTW, high MySQL server load in my case caused
several multiple cmd.php processes not finished in 5 minutes.
>Putting the data source list in pages would >also make sense. Perhaps that will sneak >its way into version 0.6.6.
What about selecting DS's for graph items? Dropdown combobox with 100 or more list entries is a hard to use.
>Performance should improve with 0.6.5 as >SNMP is now build in.
I think Get SNMP network data script should
not use information about SNMP interfaces from database. Router reload causes complete
data gathering malfunction. MRTG is a good example.
It seems to me, data gathering process should be divided for several parallel processes. Each process cannot run more than 5 min.
BTW, high MySQL server load in my case caused
several multiple cmd.php processes not finished in 5 minutes.
>Putting the data source list in pages would >also make sense. Perhaps that will sneak >its way into version 0.6.6.
What about selecting DS's for graph items? Dropdown combobox with 100 or more list entries is a hard to use.
Igor N Indick
JSC Doris
JSC Doris
I have already moved to the php's internal snmpget() function but performance did not improve much at all, if any 
I'm up to 3 minutes 15 seconds on data gathering and I'm looking to add a whole lot more interfaces.
do you think it would be possible to have two or more installs of cacti then have those crons running at the same time but "ln -s cacti1/rra cacti2/rra" then configure all of the graphs into one installation?
I'm up to 3 minutes 15 seconds on data gathering and I'm looking to add a whole lot more interfaces.do you think it would be possible to have two or more installs of cacti then have those crons running at the same time but "ln -s cacti1/rra cacti2/rra" then configure all of the graphs into one installation?
Not sure what running two cacti's at the same time would do. You can go ahead and try it, but I don't see it working for some reason.
There are a few things that make SNMP support faster in 0.6.5, not just the PHP module. The file 'cmd_snmp_interfaces.php' is no longered shelled and executed each time data for an interface needs to be gathered; that is what helped me the most. Did I mention that 0.6.5 is out now? Go ahead and see if you notice a speed difference.
-Ian
There are a few things that make SNMP support faster in 0.6.5, not just the PHP module. The file 'cmd_snmp_interfaces.php' is no longered shelled and executed each time data for an interface needs to be gathered; that is what helped me the most. Did I mention that 0.6.5 is out now? Go ahead and see if you notice a speed difference.
-Ian
cacti 0.6.5:
select * from rrd_ds;
363 rows in set (0.01 sec)
ps ax|grep cmd
55311 ?? S 0:10.44 /usr/local/bin/php -q /usr/local/apache/ncc/cacti/cmd
57046 ?? S 0:06.98 /usr/local/bin/php -q /usr/local/apache/ncc/cacti/cmd
58834 ?? S 0:04.22 /usr/local/bin/php -q /usr/local/apache/ncc/cacti/cmd
60572 ?? S 0:01.62 /usr/local/bin/php -q /usr/local/apache/ncc/cacti/cmd
...all the same.
Each command line for data gathering run not more than 1 second. Measured by inserting runtime output in log file.
Seems only multithreaded C code will help.
select * from rrd_ds;
363 rows in set (0.01 sec)
ps ax|grep cmd
55311 ?? S 0:10.44 /usr/local/bin/php -q /usr/local/apache/ncc/cacti/cmd
57046 ?? S 0:06.98 /usr/local/bin/php -q /usr/local/apache/ncc/cacti/cmd
58834 ?? S 0:04.22 /usr/local/bin/php -q /usr/local/apache/ncc/cacti/cmd
60572 ?? S 0:01.62 /usr/local/bin/php -q /usr/local/apache/ncc/cacti/cmd
...all the same.
Each command line for data gathering run not more than 1 second. Measured by inserting runtime output in log file.
Seems only multithreaded C code will help.
Igor N Indick
JSC Doris
JSC Doris
My guess about the performance problems of cmd.php are related two things:
<OL>
<LI>everything is done in php, an interpreted language that is to my knowledge does not support threads or concurrency.
<LI>Use of external programs to fetch data. cmd.php calling something like snmpget incurs the additional overhead of building up and tearing down another process
</OL>
Despite this I'm still of the opinion that raX still did a fabulous job. His initial goal was to provide monitoring of his home network and he is obviously very proficient in php. So it is understandable that the current design does not scale well to those of us that have enough equipment to require monitoring hundreds or thousands of interfaces.</P>I believe the solution is to let PHP handle the things that PHP is good at. namely the web configuration and reporting system. We should use a threaded C application to replace cmd.php. This way we could process simulataneous data source requests to multiple machines, use snmpbulkget type requests and process thousands of interfaces on scores of switches/routers in seconds, not minutes.</P>
I'm a guru at C, average at Posix threads but I could use help in understanding what the current cmd.php functions do and how they do it. Help programming would also be appreciated since my job is pretty demanding on time. I would also welcome input as to the design of such a new system.</P>
For instance... I thought it might be a nice feature if we could add a field to the src table that indicates a number between 0 and 4 inclusive. This number would indicate how minutes past a five minute mark a datasource should be polled. This way you could call cmd.c every minute by cron instead of every five minutes. Pass cmd.c the remainder of the current minute divided by 5. So if the
current minute is 23 call "cmd.c 3" and cmd.c would only gather data for the source with a "3" in this field. I like this idea because it evens out all the traffic and CPU resources generated by my SNMP monitoring machine. Basically instead of attempting to collect everything in one shot every five minutes we can spread the load out evenly throughout the day.</P>
Anyhow, that's just one idea I'ld like to hear any others so that our design of cmd.c can incorporate them or make the proper accomodations for them to be added later.
<OL>
<LI>everything is done in php, an interpreted language that is to my knowledge does not support threads or concurrency.
<LI>Use of external programs to fetch data. cmd.php calling something like snmpget incurs the additional overhead of building up and tearing down another process
</OL>
Despite this I'm still of the opinion that raX still did a fabulous job. His initial goal was to provide monitoring of his home network and he is obviously very proficient in php. So it is understandable that the current design does not scale well to those of us that have enough equipment to require monitoring hundreds or thousands of interfaces.</P>I believe the solution is to let PHP handle the things that PHP is good at. namely the web configuration and reporting system. We should use a threaded C application to replace cmd.php. This way we could process simulataneous data source requests to multiple machines, use snmpbulkget type requests and process thousands of interfaces on scores of switches/routers in seconds, not minutes.</P>
I'm a guru at C, average at Posix threads but I could use help in understanding what the current cmd.php functions do and how they do it. Help programming would also be appreciated since my job is pretty demanding on time. I would also welcome input as to the design of such a new system.</P>
For instance... I thought it might be a nice feature if we could add a field to the src table that indicates a number between 0 and 4 inclusive. This number would indicate how minutes past a five minute mark a datasource should be polled. This way you could call cmd.c every minute by cron instead of every five minutes. Pass cmd.c the remainder of the current minute divided by 5. So if the
current minute is 23 call "cmd.c 3" and cmd.c would only gather data for the source with a "3" in this field. I like this idea because it evens out all the traffic and CPU resources generated by my SNMP monitoring machine. Basically instead of attempting to collect everything in one shot every five minutes we can spread the load out evenly throughout the day.</P>
Anyhow, that's just one idea I'ld like to hear any others so that our design of cmd.c can incorporate them or make the proper accomodations for them to be added later.
I have three versions of cacti 0.6.4 running with php internal snmpget() I have each instance gathering about 100 interfaces and I have lowered the gathering time from about 3 minutes 15 seconds, down to about 2 minutes 15 seconds, for a total of about 300 interfaces. basically I created 3 totaly seperate installs, created cacti1/ cacti2/ cacti3/ ect... then I added about 100 different interfaces to each install, I then created the Graph Hierarchy for each install, I then hard coded an <option> (in include/top_graph_header.php) link to each graph hierarchy so they can all be accessed from one page, and moving between cacti1 2 or 3 is transparent to any user.
After cleaning out some of the duplicate datasources I had in a couple of my running instances of cacti I now have it collecting just over 300 SNMP interfaces in under 1 minute and 30 seconds, which used to be over 3 minutes, and I have all the graph pages linked together nicely for seamless usability between all instances.
<font size=-1>[ This Message was edited by: drub on 2002-02-08 18:19 ]</font>
<font size=-1>[ This Message was edited by: drub on 2002-02-08 18:19 ]</font>
That would be awsome, I have been watching the threads on cmd.c the part about gathering all of the commands to be executed first then creating a multi threaded queue is interesting, I have been contemplating trying to write/contribute something myself but have had limited time right now. I did add some additional features to the settings page on the instance I have running, which include,
changing the graph size globally,
changing the IP address for datasources globally, and
setting/unsetting Hard Max limits globally
changing the graph size globally,
changing the IP address for datasources globally, and
setting/unsetting Hard Max limits globally
Haven't had much time this week so not much more progress since Friday on cmd.c.</P>
But I have been trying to setup cacti 0.6.5 on a new machine and I am having one hell of a time getting it to update the data sources. I think it boiled down to the erroneous inclusion of a username in apache's crontab file.</P>
Anyhow, it got me to thinking... The cmd.c program should be designed to generate much, much more operational and/or error messages.</P>
If anybody has some thoughts on a good way to generate logs from multiple threads I'ld love to hear them. I don't think we can have all threads writing to the same file even if we put the writes in a mutex. There's just going to be times when the time completion of the threads is so different as to make the resulting log file difficult to parse. Writing to multiple files, one for each thread, also seems ungainly. Any ideas?</P>
maybe each thread will have to buffer up the stderr output (or other logged io stream) for each of its tasks and only write to the log file when the task is complete.</P>
I'll try to work toward cmd.c logging this way for now unless somebody has a better idea and I will also be designing to allow cmd.c to be run as a daemon.</P>
[Drub: I hope to get you 300 interfaces done in less than 5 seconds ;-]
But I have been trying to setup cacti 0.6.5 on a new machine and I am having one hell of a time getting it to update the data sources. I think it boiled down to the erroneous inclusion of a username in apache's crontab file.</P>
Anyhow, it got me to thinking... The cmd.c program should be designed to generate much, much more operational and/or error messages.</P>
If anybody has some thoughts on a good way to generate logs from multiple threads I'ld love to hear them. I don't think we can have all threads writing to the same file even if we put the writes in a mutex. There's just going to be times when the time completion of the threads is so different as to make the resulting log file difficult to parse. Writing to multiple files, one for each thread, also seems ungainly. Any ideas?</P>
maybe each thread will have to buffer up the stderr output (or other logged io stream) for each of its tasks and only write to the log file when the task is complete.</P>
I'll try to work toward cmd.c logging this way for now unless somebody has a better idea and I will also be designing to allow cmd.c to be run as a daemon.</P>
[Drub: I hope to get you 300 interfaces done in less than 5 seconds ;-]
I would settle for 500-800 interfaces under 1 minute I am actually up to 405 Interfaces at just under 2 minutes running 4 simultaneous instances of cacti, Man I would love to get these all back in to one instance.
RAX: I was wondering if it would be possible to make the Edit Graph Heirarchy page into a collapsable tree like the Graph View, after adding 200-300 graphs it gets difficult to keep organized
RAX: I was wondering if it would be possible to make the Edit Graph Heirarchy page into a collapsable tree like the Graph View, after adding 200-300 graphs it gets difficult to keep organized
Who is online
Users browsing this forum: No registered users and 7 guests