thanks for the update, I am sure we all will be waiting to start crunching again

Thanks for the update, and good luck in nailing down and solving the problems.
I think that in parallel to you guys working on the science, someone should be working to prevent the slowdown in computer speed that happens on multi-core computers when more than 1 instance of CEP (CHARMM) is running simultaneously.
During the recent CEP run, many people micromanaged their computers to ensure that CEP WUs ran 1 at a time. For example, CA Sekerob said on Feb 6, "I only permit them to run 1 by 1, which is not the idea, but they run that way about 20% faster in any combination of jobs on the other cores." This was acceptable at the start of your so-far-troublesome project, but will not be so in the future, especially when DDDT Phase 2 comes on line (What's the latest news from Galveston?- watowich, 31/1/09), as DDDT Phase 2 is also projected to use CHARMM (http://www.utmb.edu/discoveringdenguedrugs%2Dtogether/Technical%20Details.htm).
I suspect that the slowdown is associated with the huge numbers of memory page-faults generated throughout the program runs. In the Beta Test forum, I conjectured a reason for these page-faults Re: CEP is experiencing a huge PF Delta rate. I did not know about the multi-core performance degradation at the time, but this degradation may be because the operating systems handle memory allocation in a single thread, which bottlenecks when it gets too many requests from the multiple cores. And as I suggested then, these myriad calls to the OS probably degrade the performance of the program when it's running on only 1 core, too.

I think that this matter deserves some priority. To quote Stan Watowich, head researcher of DDDT, "I suspect that the grid will need to grow even more to return our Phase 2 calculations as fast as we would like ... we will see shortly ...". (Link above)

Perhaps the maintainers of CHARM at Harvard can improve their software. In the current economic climate, there must be plenty of voluntary or paid spare programmer-hours available somewhere. The solution may be as simple as diverting a couple of functions that call directly on the operating system to allocate/deallocate memory, to other functions that use a buffered scheme like malloc() and free() in C. Public-domain source code should be available for such schemes, eg from UseNet, alloc.c . - HTH
PS: And it took me quite a while to dig out the links and the source code.

Thanks for the update!

Personally, think Rickjb did a great job in pulling facts together and providing pathways to other information and preemptively reducing many more posts.
It's the concept of "Meeting and Exceeding the customers expectations".

Personally I dislike half liners that do not display the querier actually took effort to find an answer. Those are bound to receive curt answers or much delayed or none.

Thanks, Rickjb.

Thanks for the great update! It is much appreciated. The amount of info you provide in every update always helps us understand what is going on "behind the scenes"!

I am very patient with this project. I am sticking around to see the end of this project's goals!