Hello mousie,

It's a good thing that I have not slighted anyone. But there are signs that can sometimes be seen, even without tone of voice, that indicate misunderstandings are about to blossom.

What we have been discussing is ... hmmm ... a matter of gearing. grid.org has been successfully running a LigandFit project for Cancer for years. Target proteins have literally hundreds of thousands of work units run against them trying to find a drug that will interfere with the target. Apparently there is some sort of storage hierarchy involved siphoning off the results as they are returned. But the scoring program keeps the target protein at some specific level of the storage hierarchy as long as points are to be awarded for returned results. Remove the target and the results are not awarded points (and probably are not siphoned off correctly).

But the HPF project is geared very differently. Each protein needs about 10,000 structural predictions. A single work unit might require, say, 200 structural predictions. [Now I start waving my hands in front of the white board - Grin] So we need 50 work units for that protein. So we send out 300 work units (redundancy factor of 6). How many work units get sent out per protein for a typical Cancer job? 300,000? 3 million?

The point is, the target protein area of storage is going to be used much more extensively in the HPF project. Looking at the size of grid.org ( http://vspx27.stanford.edu/DCcomparison.html ) I would say that it needs at least 3 times the disk space of WCG to handle the flood. IF IT USES MINIMUM REDUNDANCY. But thinking of the hard disk version of Moore's Law and considering the year in which each server system was set up, I would expect that grid.org would have started off with less hard disk space. If that was enough for the projects it was running, then great! If it's not broken, don't fool around with success.

But if it turns out to be too small for HPF, then once you fill up the target protein space, all you can do is increase redundancy and keep siphoning off the results that are returned until the wall clock says that you can remove an old target protein and put in a new one. So you are not using member's computers at maximum efficiency. Or you can do other things, such as cut down the wall clock time that members are allowed to return results in. One good possibility would be to radically trim the number of computers allowed to process work units by setting very high minimum requirements.

The problem is that there is too much computer power for the hard disk space to use with maximum efficiency. It is like taking a car with a 425 horse power motor and giving it wheels off a tricycle. Or think of one of those Victorian bikes with an enormous front wheel. Do a Looney Tunes shrink on the front wheel and expand the rear wheels in proportion. The gearing ratios will be all wrong.

But there is no problem with validity. It is just a matter of speed. The resized bike will still get you to your destination, as will the car, but the engineering efficiency will not be optimal. Give the car normal wheels and it will move faster. Similarly, upgrade the hard disk drive to, for example, 3 times whatever the size that WCG uses, and grid.org has the computer power to run through HPF 3 times as fast. Apparently there is a critical limiting factor somewhere in the storage hierarchy at grid.org that has been limiting the speed with which it runs the HPF project.

This is not guaranteed to be a good representation of the situation. It is simply how I interpret Robby Brewer's explanations, including a few ideas from what Viktors has posted. I might be totally misinterpreting what they have said.

But I usually have an opinion on everything. Right or wrong.
[Was it Groucho Marx who said: "That's my opinion of the matter, and if you don't like it.. Well, I have others."]

mycrofth

Added: All the smilies in vaio's post indicate that he is being humorous, but in case someone who is not familiar with this situation reads this post, I had better hastily add that Robby Brewer has the new server equipment standing by at grid.org and is waiting for some software before going through the massive upgrade. So this post is just discussing some of the history leading up to the upgrade. [History is a Series of Unfortunate Events That ... ]

Hmm... There are a couple spots I'm not sure I agree on with your analyzation. Speculation on the stats aside...agreement on them is not necessary for my most recent curiosity, which has yet to be touched upon.


Again I ask...at this point, I'm simply inquiring about some sort of standard or something set forth so that the final data from both is at least known to have been anaylzed similarly...this repeated in my most recent two posts in the original thread, though seemingly avoided there and yet again here. I'm not so focused on stats now, but more the big picture. I can list differences I've noticed, as probably others that have spent time at both places can. Statistically different. And client different(not appearance, but that one will run happily and the other will crash unexplainably on some peoples machines). Differences aside, what do they hold in common...or perhaps, what *should* they hold in common? And what has been done to ensure this is true? Does the different ways of splitting units to be completed affect the final output...should there be some minimum set redundancy...and what's up with one client working and the other crashing, causes one to think there's some sort of difference software-wise(no, the problem with crashing software is not here, and not some problem I have that needs fixing, but more a point that there is some difference). Does how the issue both have with disk space is handled have an affect on things? This is what I mean when I enquire about the validity of final data...sure, both will turn in data to the best of their ability. But what is set to ensure that what's turned in from both have gone through a similar process...especially when they seem so different, and especially when one is being watched so close by ISB while the other seems to be ignored, at least from a user standpoint, and to the point where the differences can't be explained. I would think for scientific purposes, there should be some form of control, and perhaps there is, though has yet to be mentioned.


I never meant for this to be rolled into some huge conflict as it seems to have...lol.

Hi to Mousie Mycrofth et al

I think we moaned so much at the start of the project about "Monster Work Units" that Rick Alther sliced and diced the source data so that a more granular shorter work unit became the norm. Rick also optimised the agent software so that it didn't become such a memory hog, maybe or maybe not, this agent update code was shared with the UD/Grid.org crew I just don't know.

We had very long threads trying to persuade the powers that be that shorter work units would encourage more participation in the project as you would see the results of your endevours much sooner and therefore continue with your crunching. Likewise the personal stats update 4 times a day instead of the once per day over at Grid.org (at least while I was there) this was to improve the feedback loop back to new crunchers

IBM listened and the rest is history.

The validity of our output is equal to that over at grid.org we just campaigned harder on behalf of the newbie.

It's the same comparison as having 1 work unit that takes 20 hours or 2 that take 10 hours a piece

The end result is the same here at WCG or over at Grid.org

as an ex ud/grid.org cruncher myself (5 years of run time and 500,000 points before jumping ship to help out over here - processors hadn't broken the Ghz barrier then ) I would say we are more alike than any differences between us and I am reassured by Rich Bonneau's responses in the HPF forum that we are all producing valid data both UD/Grid and WCG.

The HPF forum also came about due to our experience over at UD where Cancer Research progress info was to say the least thin on the ground so once again we campaigned for feedback from the Scientists who we are assisting and Rich Bonneau's has been great and very gracious in providing that feedback.

Long may this continue as other new projects come online.

Keep asking those questions that's what science is all about.

Just don't go switching off
Dave

Thanx David,

You caught my omissions. Now, about: Rick Alther needed an additional month to cut Rosetta from about 75MB of RAM and 300 MB of Virtual Memory to about 25 MB of RAM and 200 MB of Virtual Memory. We started running it on WCG in mid-December and Moose got a copy over at grid.org. But that was during the Christmas season, so Moose waited till he got back to beta test it. It went active on grid.org around mid-January. According to Rick, there was no change in the actual code being used, but all sorts of subroutines used for other things [protein-protein docking, high-res folding] and their associated arrays were stripped out. So we have been running the same Rosetta program, except for 1 month around the turn of the year.

mycrofth