Hi davidjharder and TPCBF, improving the servers are our #1 priority and we will share any news about improvements or changes to them when they happen.

Hello Cyclops

Well, if that's really your top priority...
Then I would recommend that you first of all contact the administration / programmer of the MCM project.

I monitored client downloads on my computers and found that almost all the bandwidth when working with WCG projects is occupied by downloading just the same large file (about 80-90% of the traffic falls on this one single file).
It is a "mcm1.dataset-sarc1.txt " file
102MB in size is one of the main databases of this project, which is the SAME for a huge numbers of WUs. I'm not sure, but I assume that this file is general/common for all current tasks of the project. But due to an incorrect configuration, this file is not saved among the permanent project files (such as executable files), but is downloaded again and again with each task package(each pack of WUs send by sever to client). Creating a huge (and absolutely unnecessary!) load on the WCG file servers and network capacity.

This feature (sharing, saving and reusing a single file across multiple WUs without downloading it again) has been around for a long time at BOINC and has been used successfully by many projects, including (but not limited to, just for example which in run too) Rosetta@Home and Einshein@Home.

And for starters, this file can be at least elementarily compressed by an archiver for transmission over the network. I opened this file - it is essentially a regular (only structured) text file, and therefore it compresses very well - even the most standard/vanilla ZIP gives about 3 times compression (from 102 MB to 33 MB). But right now it downloads as uncompressed text.
Such a waste of a network bandwidth!

This may not solve the problems with an insufficient number of sockets / connections on the servers (because it's just one file download for several WUs), but it will definitely solve the problems with the lack of their bandwidth. due to it size (102 MB versus <1 MB for almost all other downloads, average is about ~100 KB).

Well, Cyclops,
why not have "the tech team" work on the underlying issues rather than trying to mask the problem instead?
We had download problems when there were no MCM1 WUs being send out, so that one single large file just can't be the source of the problems...
Ralf

I am myself for more than a couple of decades in the IT business. And I am therefor also aware that there always can be issues creeping up when deploying a new system, or migrating an existing one.

But in our case here, what is going on is beyond comprehension.

For one, there have likely been talks between IBM and Krembil BEFORE they made the announcement of the move back in September 2021, that's now 14 months ago. That is not something that anyone, on either side, just decides over a lunch break. And during those talks, Krembil should have already had an idea of the scope and generally necessary resources required to take over the project. If they did not have that info, this would been downright stupid at that point.

Second, there were another 5 months, between the announcement and the shutdown of the system in February 2022. That would have been the right time to start putting all the resources in place required to run the project by Krembil.
At latest after the next three months, from the shutdown in February until the reactivating of the forum in May 2022, when doing the actual, physical migration, they should have been aware of most of the tripwires and other obstacles in the way to get the project operational again.
So when restarting "for testing" in June, I did expect that there will be a few weeks in which things might not work smoothly and a lot of fine tuning would still have to happen. But the scope and need for resources should by that time, 9 months into the transition, already been obvious.
But instead, yet another 5 months later, it just seems to be utter pandemonium, with things being amiss at a very basic level in this whole setup. It is incomprehensible that by now, a total of 14 months in, not having the resources is still an issue.
And on top of all of this, not having proper communication from Krembil just makes things only worse for us, the volunteers. Platitudes like "this issue is on the radar" or "fixing this is our #1 priority" are of no help here. On the contrary, it just enhances the feeling that Krembil is totally in over their collective heads at this point...


Ralf