Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 98
Posts: 98   Pages: 10   [ Previous Page | 1 2 3 4 5 6 7 8 9 10 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 20643 times and has 97 replies Next Thread
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 2173
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

I have no confirmation on the subject but the 107 bytes downloaded before a download failed could only be the size of the error message corresponding to "connection unavailable" or "page not found"
You don't really read and try to understand most of the forum posts, don't you?

It is the complete text of the error message "503 Service Unavailable....."

And again, a 5xx http(s) error is a "server side" error message, send back from the server, in contrast to a 4xx error code, which is a "client side" error code and generated by the http(s) client itself. This is well define http(s)/Internet traffic behavior...

https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://httpwg.org/specs/rfc9110.html#status.503


Ralf
[Nov 9, 2022 10:29:02 PM]   Link   Report threatening or abusive post: please login first  Go to top 
HyperComputing
Advanced Cruncher
Joined: Aug 10, 2019
Post Count: 74
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

So in my apache config file all lines referring to error 4xx are useless ...
----------------------------------------
My GPU compute WUs as fast as I'm crushing bubble wrap.
[Nov 10, 2022 2:04:35 AM]   Link   Report threatening or abusive post: please login first  Go to top 
mxplm
Cruncher
Joined: Sep 12, 2009
Post Count: 13
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

Yes TPCBF, HyperComputing is right, the HTTP 4xx status codes are also sent by the server. E.g. the server decides that the client sent a bad request and returns that as an answer to the client.
[Nov 10, 2022 7:06:36 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

and no idea where that wcg "command" would be if it even exists, maybe I missed where it's referenced originally.
Well, PMH_UK literally said: "wcgresults -x". Have you ever tried using the Search function on the forum, AlmightyMoe? devilish Then try looking for wcgresults. It may work wonders realizing that you don't need the Retry Now button anymore, even when you are sleeping (using crontab)! laughing

I read that you use Linux, AlmightyMoe:
Using 2 linux installs
... so you might wanna give it a try.
[Nov 10, 2022 11:47:26 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 2173
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

Yes TPCBF, HyperComputing is right, the HTTP 4xx status codes are also sent by the server. E.g. the server decides that the client sent a bad request and returns that as an answer to the client.
Did you ever bother to read the links? RFC9110 (which replaced RFC2616) is a pretty good start...

Ralf
[Nov 10, 2022 4:49:07 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Greg_BE
Advanced Cruncher
Joined: May 9, 2016
Post Count: 124
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

But shouldn't this issue be solved by the project?
I mean they keep bragging about how the science did this and that.
Then comes the website being down, 30 minutes, oh no wait, longer and longer.
Then we keep fighting these transient errors.
This is getting really really old.

To get something like 5 tasks to load all their parts, I have to slam the server manually over and over until all the files get downloaded. I don't sit here waiting to see, I just check randomly if anything is downloading and mostly before I shut down for the night.
It took me 15 tries to get all the parts downloaded.
[Nov 10, 2022 7:40:25 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 2173
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

But shouldn't this issue be solved by the project?
I mean they keep bragging about how the science did this and that.
Then comes the website being down, 30 minutes, oh no wait, longer and longer.
Then we keep fighting these transient errors.
Web site&forum on one side and the BOINC facing servers should be (and in the past at IBM, were) completely different setups. One going down (be it underlying database issues, upgrade, maintenance) should not take down the other part...
And yes, now in the 6th month since resuming uploading WUs, this should be fixed by now. Unfortunately, due to the lack of communication, we don't know what is causing the delay in fixing this for good. It seems that at least a couple of times (just check Cyclops' overly optimistic announcement posts), they thought that this was done. But then it is impossible to tell if and how "the tech team" is getting its feedback. It certainly doesn't provide any feedback about what the actual issues are and what they a trying to do to fix it. So that they can get qualified feedback.
Instead, over and over again, people keep riding on non-issues and/or come up with yet new ways to work around those issues FOR THEM, AND THEM ALONE, with some of their remedies making things more likely even worse... sad
This is getting really really old.
Yes, it does.

Ralf
[Nov 10, 2022 8:05:58 PM]   Link   Report threatening or abusive post: please login first  Go to top 
HyperComputing
Advanced Cruncher
Joined: Aug 10, 2019
Post Count: 74
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

To get something like 5 tasks to load all their parts, I have to slam the server manually over and over until all the files get downloaded. I don't sit here waiting to see, I just check randomly if anything is downloading and mostly before I shut down for the night.
It took me 15 tries to get all the parts downloaded.


you can create a crontask with "/usr/bin/boinccmd --network_available" to do it for you.
no need to be root for this command
with a delay of 10 seconds all my download list is done in 15 minutes
unfortunately all my OPNG list is also done in 15 minutes and my boinc client has to wait for the next list of WUs to download

and I don't do this to have nice statistics but to return the maximum of results
----------------------------------------
My GPU compute WUs as fast as I'm crushing bubble wrap.
[Nov 10, 2022 8:47:24 PM]   Link   Report threatening or abusive post: please login first  Go to top 
mxplm
Cruncher
Joined: Sep 12, 2009
Post Count: 13
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

Did you ever bother to read the links?

No, I didn't. This is basic stuff for my daily work, I don't need to look it up anymore.

RFC9110 (which replaced RFC2616) is a pretty good start...

Thanks for the reference, it says that status codes are sent by the server, and that hasn't changed since HTTP 1.0 in 1996.

First some basics:
An HTTP "client" is a program that establishes a connection to a server for the purpose of sending one or more HTTP requests. An HTTP "server" is a program that accepts connections in order to service HTTP requests by sending HTTP responses.


Regarding status codes in general:
The status code of a response is a three-digit integer code [...]

Note: There are no "request status codes" or "client status codes" in HTTP.

And for the client-side error:
The 4xx (Client Error) class of status code indicates that the client seems to have erred. Except when responding to a HEAD request, the server SHOULD send [...] an explanation of the error situation, and whether it is a temporary or permanent condition.


Note also that man HTTP 4xx status codes cannot logically be given by the client:
- HTTP 401 Unauthorized: The server decides who is allowed in.
- HTTP 404 Not Found: How would a client know if a server has a certain site/resource without asking it?

If you still don't believe me, google it. You will find the same info everywhere.

Back to topic:
On my machine, a script checks if there are unfinished transfers every few seconds, no matter if they are stalled. If so, the script calls boinccmd --file_transfer http://www.worldcommunitygrid.org/ <random_transfer_file> retry. If the maximum amount of transfers (default=2) are already in progress, this does nothing except clearing any project back-off timer, and putting a random file next in line to be tried. If there are less than 2 downloads running and there are stalled ones, the retry reactivates one or two of them.

Without the script, I often don't get my WUs in time before the ones I have are done and my CPU is idle. This is mostly because the back-off timers quickly jump to waiting hours after a few failed download attempts. Also, while downloads are stalled the client won't ask for new work. With my script, the downloads are processed slowly but steadily and I never run out of WUs. Being aware that flooding the WCG servers with too many requests is not nice, I analyze the BOINC logs (file_xfer_debug option active) and the client does an average 2.5 requests per minute while downloading WUs, and none otherwise. I think for 32 threads running WUs that's a tolerable number. Around 50% of the requests are successful, meaning that on average each download fails only once before finishing successfully.
[Nov 10, 2022 11:04:47 PM]   Link   Report threatening or abusive post: please login first  Go to top 
HyperComputing
Advanced Cruncher
Joined: Aug 10, 2019
Post Count: 74
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Issue with task downloading

Downloads are 2 files by 2 files
under normal conditions a download is done in 1 second
under current conditions it takes 3 seconds for a download to be in error
after 4 downloads in error all the others files go on pending time

the "boinccmd --network_available" command allows the boinc client to resume where it left off without waiting for the pending time to expire
it takes 6 seconds for the whole list to be pending, so a 10 second delay for the boinccmd command isn't too short

4 requests every 10 seconds vs. 1 every second, I don't think the server will crash for that.

The goal is to return the most results even with a buggy server
There are already very competent teams who are dealing with the various problems of downloading and slowness of the site.
----------------------------------------
My GPU compute WUs as fast as I'm crushing bubble wrap.
[Nov 11, 2022 12:26:31 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 98   Pages: 10   [ Previous Page | 1 2 3 4 5 6 7 8 9 10 | Next Page ]
[ Jump to Last Post ]
Post new Thread