Uploading finished results, is now getting lots of transient HTTP errors. Rinse and repeat, uploading fails.

World Community Grid 2025-10-31 19:42:28 Started upload of MCM1_0241444_3782_0_r160218636_0
World Community Grid 2025-10-31 19:42:28 Started upload of MCM1_0241444_2608_1_r1968019595_0
World Community Grid 2025-10-31 19:42:29 Temporarily failed upload of MCM1_0241444_3782_0_r160218636_0: transient HTTP error
World Community Grid 2025-10-31 19:42:29 Backing off 00:08:52 on upload of MCM1_0241444_3782_0_r160218636_0
World Community Grid 2025-10-31 19:42:29 Temporarily failed upload of MCM1_0241444_2608_1_r1968019595_0: transient HTTP error
World Community Grid 2025-10-31 19:42:29 Backing off 00:28:55 on upload of MCM1_0241444_2608_1_r1968019595_0

Perfect timing,

I've not really noticed any upload issues until just now - annoying because if they don't clear away they can't be reported so there's a chance that new work can't be collected...

During the current upload issues I've turned on file_xfer_debug to look at the full upload and download paths. I don't know how the fanouts are divided across filestores or servers, but it might be of interest...

I also seem to get [fatal] download errors at one or two brief intervals each day (and I mean brief!):

• On 2025-10-29 I had 9 errors between 15:52 UTC and 15:55 UTC on one system and one at 18:41 UTC on another system;
  
• On 2025-10-30 I had one at 14:59 UTC and 5 between 16:06 and 16:11 UTC (spread over two systems);
  
• So far today (2025-10-31) I've had one at 13:57 UTC and three between 18:12 and 18:15 UTC
  

I'd really like to know what is causing this -- is it the server(s) or the proxy (or...)?

As I download an average of over 30 tasks an hour I see these as an inconvenience rather than a real annoyance -- however, someone processing far fewer tasks a day might well have a different view if they get several failures!

Cheers - Al.

[Edited to reflect the sudden appearance of multiple upload errors...]

Right -- further to my previous post , I've been watching the attempted uploads and downloads on my systems for the last few hours. I've not had a download error since I turned the extra logging on, but I've seen at least one attempted upload for every currently stuck task...

If you look at the URL path offered as part of the file_xfer_debug logging, its path part is of the form

    /boinc/upload/???/file_upload_handler

or

    /boinc/download/???/some_file_name

where the ??? will be 1, 2 or 3 hexadecimal characters, and represents the fanout directory.

There are 1024 potential directories, so ??? will range from 0 to 3ff.

Checking activity since turning the extra logging on, I've seen upload failures for items in the range (hex) 16b to 1fd, successes up to 10c and from 246 upwards.

I seem to recall something that suggested there were 6 potential groupings, and if they are sequential they might be either:

• 5 groups of 170 and 1 of 174 -- 0..a9, aa..153, 154..1fd, 1fe..2a7, 2a8..351, 352..3ff
  
• 4 groups of 171 and 2 of 170 -- 0..aa, ab..155, 156..200, 201..2ab, 2ac..355, 356..3ff
  

Downloads and uploads for a task will have the same fanout number...

If this guess happens to be right, all the failures would fall in the third, highlighted group, implying that only 1/6 of their service is unavailable. The majority of uploads will succeed.

Interestingly, I've just spotted a download with 1a4, so that group doesn't currently seem to be a problem for downloads -- however, I suspect that task will get stuck trying to upload when it has completed!...

I don't know whether the above is of any use -- provided "in case"...

Cheers - Al.

P.S. I used "get" in a sentence somewhere and preview got "Forbidden"; a slight reorganisation and it previewed -- hey, ho...

1ac - looks like in your fail range

https://upload.worldcommunitygrid.org/boinc/upload/1ac/file_upload_handler