| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Official Messages
Forum: News
Thread: 2022-09-15 Update (Networking & Workunits) |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 214
|
|
| Author |
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7846 Status: Offline Project Badges: 
|
Slow downloads may be caused by the following. I only run Mapping Cancer Markers, but this project alone might be slowing downloads. My computer repeatedly downloads the following huge (107 MB) file... mcm1.dataset-sarc1.txt which could be slowing everything down. My computer tries to download more tasks but "Tasks are committed to other platforms" or some other error, so it runs out of tasks. When it runs out of tasks, BOINC deletes mcm1.dataset-sarc1.txt With IBM, perhaps certain files wouldn't be deleted when all tasks that needed it were finished? The solution could be for WCG to either... (1) not have mcm1.dataset-sarc1.txt be deleted until "sarc2" or whatever comes out (2) fixing the "Tasks are committed to other platforms" issue Regardless, perhaps the simple solution is for everyone to store at least a couple days of work and to only run a few projects (like only Mapping Cancer Markers)? No, that is not the cause of the slow downloads. That large MCM1 file,at least for me, downloads quicker than the small files do. Once it has a connection, it keeps that connection and continues its download. The problem seems to be to get a good connection in the first place. The issue of the large file needing to be downloaded each time you run out of MCM1 work units has always been there as far as I know. It is just more apparent now that the supply of work units is haphazard at best. Anyway, at this time both OPN and MCM seem to both have run out of work for the moment. 10/8/2022 7:22:04 PM | World Community Grid | No tasks are available for OpenPandemics - COVID 19 10/8/2022 7:22:04 PM | World Community Grid | No tasks are available for OpenPandemics - COVID-19 - GPU 10/8/2022 7:22:04 PM | World Community Grid | No tasks are available for Mapping Cancer Markers (Time is US Central Daylight Time -- UTC-5) Cheers
Sgt. Joe
*Minnesota Crunchers* |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 2173 Status: Offline Project Badges:
|
Anyway, at this time both OPN and MCM seem to both have run out of work for the moment. Was out of the house/office all day and just checked what's going on.The MCM1 well has been running dry at least since early this morning, if not since late last night. But I stll get OPN1 (no resends) and both my GPU host have plenty of OPNG WUs as well. Let's see how it looks on Sunday morning... Ralf |
||
|
|
Link64
Senior Cruncher Joined: Feb 19, 2021 Post Count: 206 Status: Offline Project Badges: 
|
I only run Mapping Cancer Markers, but this project alone might be slowing downloads. My computer repeatedly downloads the following huge (107 MB) file... mcm1.dataset-sarc1.txt which could be slowing everything down. Yes, that's insane, this file schould be "sticky" like similar reusable files from for example Milkyway or Einstein: <file>  |
||
|
|
poppinfresh99
Cruncher Joined: Feb 29, 2020 Post Count: 49 Status: Offline Project Badges: 
|
No, that is not the cause of the slow downloads. That large MCM1 file,at least for me, downloads quicker than the small files do. Once it has a connection, it keeps that connection and continues its download. The problem seems to be to get a good connection in the first place. The issue of the large file needing to be downloaded each time you run out of MCM1 work units has always been there as far as I know. It is just more apparent now that the supply of work units is haphazard at best. My thought was that the downloads server can only make so many connections. The small files keep a connection for a moment. The 107 MB file keeps it for a minute or more. IBM clearly had a better throughput and steady supply of work, but this file not being “sticky” just makes issues worse when issues occur, like coyotes waiting to attack the sick and weak. I am not sure how you can be so confident that this isn’t a cause of the problems. I do have a couple 4 MB sticky file from MCM still on my computer from 2021 (.tga files), so I’m not convinced that this was always a problem, though these aren’t .txt files like the 107 MB file. Also 107 > 4 could be an issue slowing everything down… [Edit 1 times, last edit by poppinfresh99 at Oct 9, 2022 1:45:05 PM] |
||
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7846 Status: Offline Project Badges:
|
I am not sure how you can be so confident that this isn’t a cause of the problems. Since you only need to download the large MCM1 once (as long as you have a steady supply of MCM1 work units), the download issues continue even when not downloading this file. Besides, the downloading of the OPN files should be entirely independent of the large MCM1 file. I would welcome any alternative explanation. Cheers
Sgt. Joe
*Minnesota Crunchers* |
||
|
|
Link64
Senior Cruncher Joined: Feb 19, 2021 Post Count: 206 Status: Offline Project Badges:
|
I do have a couple 4 MB sticky file from MCM still on my computer from 2021 (.tga files) Technically those are not "sticky" files, i.e. files, which are downloaded as part of a WU and kept for future WUs, they are "project files". But yes, I agree, even if this large file downloads faster once the connection is there (making the connection is what seems to take most time on the small ones), everyone downloading it again and again, probably even more often than if the system was stable and people were not running out of WUs, is for sure adding to the issue, even though it's hard to tell how much. I noticed this "issue" shortly after joining WCG when testing the different projects, so I copied it to the projects folder and hardlinked it back to the WCG folder each time BOINC deleted it using this batch file: fsutil hardlink create "www.worldcommunitygrid.org\mcm1.dataset-sarc1.txt" "mcm1.dataset-sarc1.txt" However this isn't something we should "need" to do. |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 2173 Status: Offline Project Badges:
|
I do have a couple 4 MB sticky file from MCM still on my computer from 2021 (.tga files) Technically those are not "sticky" files, i.e. files, which are downloaded as part of a WU and kept for future WUs, they are "project files". But yes, I agree, even if this large file downloads faster once the connection is there (making the connection is what seems to take most time on the small ones), everyone downloading it again and again, probably even more often than if the system was stable and people were not running out of WUs, is for sure adding to the issue, even though it's hard to tell how much. I noticed this "issue" shortly after joining WCG when testing the different projects, so I copied it to the projects folder and hardlinked it back to the WCG folder each time BOINC deleted it using this batch file: fsutil hardlink create "www.worldcommunitygrid.org\mcm1.dataset-sarc1.txt" "mcm1.dataset-sarc1.txt" However this isn't something we should "need" to do. Just far too many people (possibly including some a Krembil?) are all just looking at the symptoms rather than at the source of the problem., Ralf |
||
|
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2346 Status: Offline Project Badges: 
|
Since 2022-10-09, 21:20 UTC on one machine I haven't seen one transient HTTP error, during uploading nor during downloading. It's now 2 hours later (when I started writing this post).
----------------------------------------After 2 hours, from 21:20 - 23:20 UTC: 36 tasks were uploaded (13x OPNG, 16x MCM1, 6x OPN1, 1x ARP1) and 44 tasks were downloaded (33x OPNG, 5x MCM1, 5x OPN1, 1x ARP1). At 23:30 UTC, another 23 OPNG-tasks were downloaded, all also without any transient HTTP error. It looks like a sudden change: during the last five minutes until the last error was seen, that's between 21:15 and 21:20 UTC, I got 39 transient HTTP errors (5 during uploading and 34 while processing downloads). After that, no more errors. Is this because the ARP1-well has run dry for a big part?  (I'm only receiving resends of it.)EDIT: Uploading the ARP1-result took 96 seconds, downloading the ARP1-task took 6 minutes. [Edit 2 times, last edit by adriverhoef at Oct 10, 2022 12:04:41 AM] |
||
|
|
geophi
Advanced Cruncher U.S. Joined: Sep 3, 2007 Post Count: 113 Status: Offline Project Badges: 
|
The website suddenly got more responsive as well. Earlier today, it took a long time just to click on "Results" and actually list them. Now, it's virtually immediate.
|
||
|
|
sptrog1
Master Cruncher Joined: Dec 12, 2017 Post Count: 1592 Status: Offline Project Badges:
|
I do not have the measurements that adriverhoef does but things seem to be better. Updates seem without problems. I am not sitting with empty threads. I do not feel I have to push the transfer button to keep th research flowing. Keep uo the good work.
|
||
|
|
|