Must say I'm dissapointed at the Linux implementation of MIP.
Pulling my Linux cores out of MIP to work on other projects. Windows cores are ok here and could stay.
Hopefully we will get some respons from techs or scientists on the progress of this issue.

Well - the saga continues. I am having to micro manage my WUs because 100% MIP is a disaster (50% is pretty bad too) I have 20 threads in 3 Macs. My oldest Mac which has only 4 threads is running SCC exclusively as it seizes up with MIP. The other two quad core (8 thread) Macs are set to run a maximum of 4 threads each of MIP when I make WUs available through device manager. I am still getting less than 20 points per hour CPU for MIP compared with nearer 25 points / h for SCC. I think that when I get my MIP Sapphire I will abandon the project entirely. Big shame!

It is comforting to notice that I am not alone with my observations.
However, why did nobody take these issues into account on scientist side? ... and provide some feedbacks?
Cheers,
Yves

Sorry it's taken us a while to respond. Please don't leave, we appreciate the time ALL of you have invested in the project. It has taken us a while to reproduce and then really pin down what causes the problem.

The short version is that Rosetta, the program being used by the MIP to fold the proteins on all of your computers*, is pretty hungry when it comes to cache. A single instance of the program fits well in to a small cache. However, when you begin to run multiple instances there is more contention for that cache. This results in L3 cache misses and the CPU sits idle while we have to make a long trip to main memory to get the data we need. This behavior is common for programs that have larger memory requirements. It's also not something that we as developers often notice; we typically run on large clusters and use hundreds to thousands of cores in parallel on machines. Nothing seemed slower for us because we are always running in that regime.

I don't know all of the details about how the points are assigned, and I don't know if/how the credit assignment will be modified. But I believe that issue stems from the fact that a single instance Rosetta is well behaved (very few cache misses) on most consumer chips, but on machines with smaller caches and few memory channels a second (or third or forth) instance cannot fit in to the caches and you see the run time scaling issues which result in fewer points/hour (i.e. if a single instance of Rosetta had these cache issues the scaling from one to multiple instances would not be as dramatic nor would the change in points/hour).**

We are looking to see if if we can improve the cache behavior. Rosetta is ~2 million lines of C++ and improving the cache performance might involve changing some pretty fundamental parts. We have some ideas of where to start digging, but I can't make any promises.

Long term, identifying these issues may end up improving Rosetta for everyone that uses it so pat yourselves on the back for that!

Doug

* a newer version of the program used in HPF1 & 2
** this might be the case for machine with very small (less than 4MB) caches, it's just always slow

Thanks for the update. Over the next 3 weeks I will be finishing up some current work and then will be moving about 280 threads to MIP. The plan is stay there for at least 90 days. I'm not contributing for the points so the credit drop doesn't bother me in the least. Good luck with the optimization activities