raX,</P>
I'm giving an initial shot at converting cmd.php to a threaded C app. Hopefully I'll be able to extend that app to use SNMP API for even more speed improvement since all of my stuff is SNMP heavy.</P>
I'm going through cmd.php and I understand the mysql queries building the commands for active data sources. One thing is confusing me since I'm not familiar with php.</P>
Where are these commands actually being executed? cmd.php itself doesn't seem to contain the code that is carrying out the actual execution.</P>
You also seem to be doing a lot of redundant queries in subfunctions. Do you seen any reason why I can't build all the command lines in one function and pass this arry of commands to the threaded command handlers?</P>
Would you also mind it if I propose some syntactical changes that allow users the freedom to do things like: snmpget yada yada | sed -e 's/[^ ]+ //'? I don't know what these proposals are yet and I don't have a decent estimate of how useful they will be. But I think it would be good to allow users to more easily utilize system commands that don't produce pre-formatted output. like us snmpget people?.
Converting cmd.php to threaded C...
Moderators: Developers, Moderators
First of all; let me thank you for even attempting this! This is something that I would have little time for at the moment but is still very important.
I will try and explain a bit. I am going to use the 0.6.5 version of 'cmd.php' as an example.
The command is actaully executed on line 72 using backticks.
I will be the first to admit that some of the SQL in the file can be hard to understand. This is mainly do to the processing required to handle the user's variables in the command; like <arg>.
Cmd.php's job is as follows (in order):
- Select all of the active data sources. (29)
- Loop through each data source and gather data input source data for that data source (39).
- Get the command to execute/expect back and fill in the path variables (48-69).
- Depending on what type of data input source this is either execute the command and get something back or call a function to get something back (72-87).
- Ask the database what we should be expecting back from this command (102-106).
- Insert the appropriate data into the 'src_data' table (116).
- Call the 'rrdtool_function_create' function to create a new .rrd file (if one does not exist) (125).
- Call the 'rrdtool_function_update' to update the .rrd file with the gathered information.
The only place where repetition occurs in this code is within the function 'rrdtool_function_update'. This amounts to only 1 or 2 SQL strings though which is not much; but still could be integrated better I guess.
Actually assuming everything is ecsaped correctly you should be able to create a data input source that pipes something to sed. There maybe something preventing this from working, but cacti is designed to let this sort of thing work.
Hope this helps you some more,
-Ian
I will try and explain a bit. I am going to use the 0.6.5 version of 'cmd.php' as an example.
The command is actaully executed on line 72 using backticks.
I will be the first to admit that some of the SQL in the file can be hard to understand. This is mainly do to the processing required to handle the user's variables in the command; like <arg>.
Cmd.php's job is as follows (in order):
- Select all of the active data sources. (29)
- Loop through each data source and gather data input source data for that data source (39).
- Get the command to execute/expect back and fill in the path variables (48-69).
- Depending on what type of data input source this is either execute the command and get something back or call a function to get something back (72-87).
- Ask the database what we should be expecting back from this command (102-106).
- Insert the appropriate data into the 'src_data' table (116).
- Call the 'rrdtool_function_create' function to create a new .rrd file (if one does not exist) (125).
- Call the 'rrdtool_function_update' to update the .rrd file with the gathered information.
The only place where repetition occurs in this code is within the function 'rrdtool_function_update'. This amounts to only 1 or 2 SQL strings though which is not much; but still could be integrated better I guess.
Actually assuming everything is ecsaped correctly you should be able to create a data input source that pipes something to sed. There maybe something preventing this from working, but cacti is designed to let this sort of thing work.
Hope this helps you some more,
-Ian
hehe! How obvious... I missed the `$str` syntax in the ParseDelineatedLine(...)! Thanks!</P>
Ok. I can probably make some further progress with this knowledge. Hopefully BY the end of this weekend I'll have a nutshell of program that builds the commands prepares them for execution by threads. I won't have time to actually write the threads or update the databases until next week.</P>
Next week I'll try to implement the actual threads and then follow that with actually updating the rra files.</P>
For anybody with large networks counting on the threaded C app: No guarantees on any of these time frames. They are just an estimate of how long I think it will take for me to implement it correctly coupled with how much time I think I have available to implement and how much time I really want to burn doing it (And how much attention my wife can afford to lose
. That's my official disclaimer.
Ok. I can probably make some further progress with this knowledge. Hopefully BY the end of this weekend I'll have a nutshell of program that builds the commands prepares them for execution by threads. I won't have time to actually write the threads or update the databases until next week.</P>
Next week I'll try to implement the actual threads and then follow that with actually updating the rra files.</P>
For anybody with large networks counting on the threaded C app: No guarantees on any of these time frames. They are just an estimate of how long I think it will take for me to implement it correctly coupled with how much time I think I have available to implement and how much time I really want to burn doing it (And how much attention my wife can afford to lose
. That's my official disclaimer.I'm lean on time. So right now the first version will follow an algorithm similar to cmd.php and will not have the database routines abstracted or compartmentalized in the interests of time.</P>
However, your ideas are correct and should be implemented. I will keep them in mind and will attempt to make judgements based on later abstracting the database specifics.</P>
The current implementation is focused on performing all the database queries in one section. This is done to build a complete list of commands that need to be executed in order to gather the sources. This list is completely built before any commands are executed.</P>
This is done for two reasons
<OL>
<LI>First its done so that we can build a complete command queue that threads will be able to work on without having to do addition database connections or queries. Basically, threads do the actual command processing and nothing else.
<LI> The second reason more closely matches your desire. We can then compartmentalize all of the database routines into something like BuildCommandQueue(). Its not as abstract as the database code should be but it should allow somebody more familiar with various databases to write that part in the future.
<LI> The last reason I can think of is that possibly cmd.c will become a daemon and take over the scheduling as it is currently done by cron. This would allow cacti to gather multiple datasources that have different frequency intervals or that desire intervals with a finer resolution than one minute. To do this I believe we should maintain a queue of commands that are independent from the databases so that we don't waste time with redundant queries.
</OL>
However, your ideas are correct and should be implemented. I will keep them in mind and will attempt to make judgements based on later abstracting the database specifics.</P>
The current implementation is focused on performing all the database queries in one section. This is done to build a complete list of commands that need to be executed in order to gather the sources. This list is completely built before any commands are executed.</P>
This is done for two reasons
<OL>
<LI>First its done so that we can build a complete command queue that threads will be able to work on without having to do addition database connections or queries. Basically, threads do the actual command processing and nothing else.
<LI> The second reason more closely matches your desire. We can then compartmentalize all of the database routines into something like BuildCommandQueue(). Its not as abstract as the database code should be but it should allow somebody more familiar with various databases to write that part in the future.
<LI> The last reason I can think of is that possibly cmd.c will become a daemon and take over the scheduling as it is currently done by cron. This would allow cacti to gather multiple datasources that have different frequency intervals or that desire intervals with a finer resolution than one minute. To do this I believe we should maintain a queue of commands that are independent from the databases so that we don't waste time with redundant queries.
</OL>
Great !!!I'm giving an initial shot at converting cmd.php to a threaded C app
Although I can't help you technical
, I'll be first in the row to help you testing...My "problem" with the great cacti tool is the commandline PHP cronjob, which is a little problem with only PHP as an Apache module. I know, I could build PHP not as a module only
, but there're several reasons why I don't.So drop a line if you need some testers...
All of the support is greatly appriciated. jwiegley: excellent point about the possibility of cacti becoming a daemon. If this happened, it would open up a whole new world of possibilities including gathering data using different intervals. Cacti could then be much more aware of things like overflows and take the appropriate actions.
I will see about separating the database code in the next couple of versions. For now I'll go ahead and write my own database functions, making it easier to use other SQL servers. However, more work will obviously have to be done to make cacti work correctly with PostgreSQL, MS-SQL, and Oracle.
If we can get this whole c thing rolling, I can see cacti becoming very scalable in the near future. I look forward to such changes.
-Ian
I will see about separating the database code in the next couple of versions. For now I'll go ahead and write my own database functions, making it easier to use other SQL servers. However, more work will obviously have to be done to make cacti work correctly with PostgreSQL, MS-SQL, and Oracle.
If we can get this whole c thing rolling, I can see cacti becoming very scalable in the near future. I look forward to such changes.
-Ian
I don't know if you ever heard of ADODB (http://php.weblogs.com/ADODB), perhaps it makes the porting much easier (hmm, writing it down isHowever, more work will obviously have to be done to make cacti work correctly with PostgreSQL, MS-SQL, and Oracle.
). I saw the guys of PostNuke CMS use it, but for now their implementation works only with MySQL. So maybe it isn't _that_ easy...Who is online
Users browsing this forum: No registered users and 0 guests