| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Official Messages
Forum: News
Thread: 2022-09-15 Update (Networking & Workunits) |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 214
|
|
| Author |
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1928 Status: Offline Project Badges: 
|
Hello! I noticed that this seems to have improved since Monday (2022-09-26) morning... Since this morning I have got about 100 new wu's mostly OPN1. Also some ARP1 and MCM was downloaded without any extra clicking on retry.  While writing this I just got 13 OPNG wu also with no retries, maybe we can see No OPNG WUs here... much more light in the end of the long tunnel!?😍  Let's just hope that those lights at the end of this deep dark tunnel aren't an oncoming freight train...  Ralf   |
||
|
|
bfmorse
Senior Cruncher US Joined: Jul 26, 2009 Post Count: 292 Status: Offline Project Badges: 
|
I agree Ralf!
I usually maintain a one to two day queue of WU’s and have not had a need to RETRY any downloads in the last couple of days! When I return from work or awake to the start of a day the WU queue is as it was before the move: completed, running or ready to start - any in the downloading state automatically completes and transitions to ready to start. As such, the Krembil/WCG support crew deserves a sincere THANK YOU for resolving completely, I hope, the “hung downloading” issues.  Bruce |
|||
|
||||
|
sam6861
Advanced Cruncher Joined: Mar 31, 2020 Post Count: 107 Status: Offline Project Badges: 
|
Download speed kind of got faster, but looks like little to no new ARP1 tasks for a while.
HAProxy is a proxy specially for load balancers and has no 64k TCP port limitation. Incoming, listening connection have no limitation, but outgoing connection is limited to a maximum of 65535 TCP ports per address due to a 16 bit number. Proxy mostly have to make outgoing connection, and I can guess outgoing connection fails with errors.More about TCP: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_ports Can search on google, port exhaustion for proxy / HAProxy. https://www.google.com/search?q=haproxy+ports+exhaustion Servers equipped with 6 to 8 cores generally achieve between 200000 and 500000 requests per second, and have no trouble saturating a 25 Gbit/s connection under Linux This quote appears to only talks about a web server, not proxy. |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1928 Status: Offline Project Badges:
|
Download speed kind of got faster, but looks like little to no new ARP1 tasks for a while. Once again, it seems you don't know (or only very little) what you are talking about...HAProxy is a proxy specially for load balancers and has no 64k TCP port limitation. Incoming, listening connection have no limitation, but outgoing connection is limited to a maximum of 65535 TCP ports per address due to a 16 bit number. Proxy mostly have to make outgoing connection, and I can guess outgoing connection fails with errors.More about TCP: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_ports Can search on google, port exhaustion for proxy / HAProxy. https://www.google.com/search?q=haproxy+ports+exhaustion Servers equipped with 6 to 8 cores generally achieve between 200000 and 500000 requests per second, and have no trouble saturating a 25 Gbit/s connection under Linux This quote appears to only talks about a web server, not proxy.Ralf |
||
|
|
Richard Haselgrove
Senior Cruncher United Kingdom Joined: Feb 19, 2021 Post Count: 360 Status: Offline Project Badges: 
|
Can search on google, port exhaustion for proxy / HAProxy. https://www.google.com/search?q=haproxy+ports+exhaustion The top results from that search date back almost 5 years, some more. Since then, there has been a concerted push, from many players, towards a more extensive use of https, in place of http.Servers equipped with 6 to 8 cores generally achieve between 200000 and 500000 requests per second, and have no trouble saturating a 25 Gbit/s connection under Linux This quote appears to only talks about a web server, not proxy.In the case of WCG, that has been implemented in full, to the extent that plain http connections are actively prohibited by HSTS (HTTP Strict Transport Security) - as we saw during the SSL certificate expiry on 09 Sep 2022. My experience is that a server receiving mainly https connection requests can service many, many fewer concurrent requests than a basic http server. Can anyone confirm that observation, and ideally quantize the effect? |
||
|
|
mdxi
Advanced Cruncher Joined: Dec 6, 2017 Post Count: 109 Status: Offline Project Badges: 
|
My experience is that a server receiving mainly https connection requests can service many, many fewer concurrent requests than a basic http server. Can anyone confirm that observation Is there additional overhead in servicing HTTPS vs HTTP? Yes. Would I characterize it as resulting in "many, many fewer requests"? Not on anything resembling modern hardware, no. Also, since haproxy was brought up, if you have anything of the sort (haproxy, an AWS ALB, a physical load balancer), you're probably doing TLS termination there, for both performance and simplicy reasons. and ideally quantize the effect? Not without knowing the back-end hardware and network topology, no.  |
||
|
|
sam6861
Advanced Cruncher Joined: Mar 31, 2020 Post Count: 107 Status: Offline Project Badges:
|
Can bypass HSTS by using chrome incognito mode, or delete all web browser cookies. Some websites redirect to HTTPS, but this download server don't redirect, unless a web browser redirect with HSTS cookies.
----------------------------------------http://download.worldcommunitygrid.org/boinc/...rp1_stats/generations.txt Unfortunately it appears both HTTP and HTTPS have had the same problems with slow speed and "no server available" errors, when problems happened a few days ago with lots of new ARP1 tasks. Back then on certificate expired problem, the main page with HSTS bypass is just a proxy error, but download and uploads on HTTP did work. Can use BOINC client_state.xml to shut down BOINC and change download/upload location to HTTP or something, however new tasks goes to HTTPS. I believe HTTPS don't slow down much, unless the CPU do not have AES-NI instruction set. Poor HP mini 110-1000, Intel Atom N270 with no AES-NI can only download HTTPS at 9.41 MB/s with 100% CPU usage, but HTTP goes faster 12MB/s with less then 10% CPU usage. [Edit 1 times, last edit by sam6861 at Sep 27, 2022 10:14:43 PM] |
||
|
|
AgrFan
Senior Cruncher USA Joined: Apr 17, 2008 Post Count: 365 Status: Offline Project Badges: 
|
Interesting read ...
----------------------------------------Protect Servers with HAProxy Connection Limits and Queues https://www.haproxy.com/blog/protect-servers-...ection-limits-and-queues/ [Edit 1 times, last edit by AgrFan at Sep 28, 2022 2:21:17 AM] |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1928 Status: Offline Project Badges:
|
Well, a lot of discussion, all the while it seems what ever they could "tune" at Krembil/WCG's end seems to be working, even now that there are apparently OPNG WUs send out again.
----------------------------------------I was out pretty much all day and it seems all hosts that I was able to chek after dinner seem to be downloading, crunching, uploading and reporting WU from all projects (for me this is ARP1, MCM1, OPN1 and OPNG) just fine. So, while we still might wanna give it a day or two, it seems we have reached the end of this dark tunnel and see daylight again, which hopefully give the techs at WCG now time to look into the remaining outstanding issues... Ralf |
||
|
|
nivrip
Senior Cruncher North Yorkshire Joined: Sep 13, 2007 Post Count: 262 Status: Offline Project Badges:
|
, even now that there are apparently OPNG WUs send out again. Ralf I've been getting OPNG WUs for several weeks now. Am I the only one?
ЮРКШИР КРУНЧЕР
|
||
|
|
|