Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 
World Community Grid Forums
  Category: Support
  Forum: BOINC Agent Support
  Thread: Transient HTTP Errors, the continuing story...

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 9
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 1929 times and has 8 replies Next Thread
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Transient HTTP Errors, the continuing story...
Let's start with some excuses for the lenghty message, but to have a better grasp of the situation I need to offer some snippets from the Event Log.
That said, to recap, my second machine 'b' wasn't able to upload to WCG, nor download from WCG anymore at the beginning of this week - the latest upload/download that really finished was this one:
31-Jul-2016 19:05:59 [World Community Grid] Finished upload of OET1_0003903_x4G9Zp_rig_69493_0_r1527338780_0
I wasn't there to give 'b' a kick (metaphorically speaking of course, we don't kick machines, we pat them on the shoulders), so we skip 24 hours:
01-Aug-2016 18:51:00 [---] Project communication failed: attempting access to reference site
01-Aug-2016 18:51:00 [World Community Grid] Temporarily failed download of cc6ff69341b2eff8dac7dd17e679dc0d.job: transient HTTP error
01-Aug-2016 18:51:00 [World Community Grid] Backing off 05:07:59 on download of cc6ff69341b2eff8dac7dd17e679dc0d.job
01-Aug-2016 18:51:00 [World Community Grid] Temporarily failed download of 1ac91c13e91724d799f2298cade697aa.zip: transient HTTP error
01-Aug-2016 18:51:00 [World Community Grid] Backing off 04:04:01 on download of 1ac91c13e91724d799f2298cade697aa.zip
01-Aug-2016 18:51:02 [---] Internet access OK - project servers may be temporarily down.

Then my girlfriend got home and tried a Project Update:
01-Aug-2016 19:03:55 [World Community Grid] update requested by user
01-Aug-2016 19:03:56 [World Community Grid] Sending scheduler request: Requested by user.
01-Aug-2016 19:03:56 [World Community Grid] Reporting 2 completed tasks
01-Aug-2016 19:03:56 [World Community Grid] Not requesting tasks: some download is stalled
01-Aug-2016 19:04:22 [---] Project communication failed: attempting access to reference site
01-Aug-2016 19:04:22 [World Community Grid] Scheduler request failed: SSL connect error
01-Aug-2016 19:04:24 [---] Internet access OK - project servers may be temporarily down.
01-Aug-2016 19:05:32 [World Community Grid] Fetching scheduler list
01-Aug-2016 19:05:34 [World Community Grid] Master file download succeeded
01-Aug-2016 19:05:39 [World Community Grid] Sending scheduler request: To report completed tasks.
01-Aug-2016 19:05:39 [World Community Grid] Reporting 2 completed tasks
01-Aug-2016 19:05:39 [World Community Grid] Not requesting tasks: some download is stalled
01-Aug-2016 19:06:04 [---] Project communication failed: attempting access to reference site
01-Aug-2016 19:06:04 [World Community Grid] Scheduler request failed: SSL connect error
01-Aug-2016 19:06:05 [---] Internet access OK - project servers may be temporarily down.
01-Aug-2016 19:08:05 [World Community Grid] Sending scheduler request: To report completed tasks.
01-Aug-2016 19:08:05 [World Community Grid] Reporting 2 completed tasks
01-Aug-2016 19:08:05 [World Community Grid] Not requesting tasks: some download is stalled
01-Aug-2016 19:08:30 [---] Project communication failed: attempting access to reference site
01-Aug-2016 19:08:30 [World Community Grid] Scheduler request failed: SSL connect error
01-Aug-2016 19:08:31 [---] Internet access OK - project servers may be temporarily down.

To no avail. So she restarted the BOINC service.
First off, what happened before that, how many "transient HTTP errors" were going on?
25-Jul-2016 04:07:32 [World Community Grid] Temporarily failed download of 59461610cda84f3921e83f26d4ef717c.zip: transient HTTP error
25-Jul-2016 06:23:44 [World Community Grid] Temporarily failed download of a9122a63040bb03ded58ad953338454b.job: transient HTTP error
25-Jul-2016 07:40:08 [World Community Grid] Temporarily failed download of adfbc0aff8999eb383dfe8092cc528af.job: transient HTTP error
25-Jul-2016 08:41:17 [World Community Grid] Temporarily failed download of 89a0c465e3cf746ba475737e05047950.zip: transient HTTP error
25-Jul-2016 11:54:00 [World Community Grid] Temporarily failed download of 6a735ce7d808f07c567fa1118f2ed94d.zip: transient HTTP error
25-Jul-2016 14:31:58 [World Community Grid] Temporarily failed download of 092fcbe82eb268602acb13ab9a7cc4d6.zip: transient HTTP error
25-Jul-2016 16:52:20 [World Community Grid] Temporarily failed download of a4ae0fc5626d6a5b685c1ee4099fdbf7.job: transient HTTP error
25-Jul-2016 17:55:34 [World Community Grid] Temporarily failed download of 7dfb3b952f0c76f6ebf10dc6d53ef3c7.zip: transient HTTP error
25-Jul-2016 18:57:18 [World Community Grid] Temporarily failed download of 6ce8e16a3d1b637f5a9ab87002cdb132.zip: transient HTTP error
25-Jul-2016 20:13:53 [World Community Grid] Temporarily failed download of b61fd69a3fb235e6451c4fa1b2416efa.zip: transient HTTP error
25-Jul-2016 21:21:34 [World Community Grid] Temporarily failed download of c040218d3f824e25f7169c321a377889.job: transient HTTP error
25-Jul-2016 22:30:03 [World Community Grid] Temporarily failed download of 13a7f07c120c25af609f229bdbacb408.zip: transient HTTP error
25-Jul-2016 23:39:21 [World Community Grid] Temporarily failed download of 8f52e6ba825e96415a6944545693d544.zip: transient HTTP error
26 July, same story, 18 transient HTTP errors.
27 July, same story, 24 transient HTTP errors.
28 July, same story and worse: 164 transient HTTP errors.
29 July, same story and even worse: 229 transient HTTP errors.
30 July, same story and even more worse: 789 transient HTTP errors.
31 July, succeeded in both downloading and uploading only 2 (two!) files; 86 transient HTTP errors.
1 Aug, 16 transient HTTP errors, but no file transfers at all; then my girlfriend got home.

After the restart (/bin/systemctl restart boinc-client) no more transient HTTP errors, until ... today.
06-Aug-2016 03:13:34 [World Community Grid] Temporarily failed download of 83854fec7a5c07b7ac538a8363e68f58.zip: transient HTTP error
06-Aug-2016 03:15:10 [World Community Grid] Temporarily failed download of b26f7a2b4501311d0d9a8c9beb81f002.pdbqt: transient HTTP error
06-Aug-2016 04:15:53 [World Community Grid] Temporarily failed download of bc738010459817cdaaf4b3fc97819b31.zip: transient HTTP error
06-Aug-2016 05:29:15 [World Community Grid] Temporarily failed download of 8b583407b9572b2d2a37453503350ea8.zip: transient HTTP error
06-Aug-2016 06:31:48 [World Community Grid] Temporarily failed download of bea59d90fbb3f531861a4187f5dac6ca.zip: transient HTTP error
06-Aug-2016 07:37:02 [World Community Grid] Temporarily failed download of 7aa6f07603ca6ec0088077b82f880b02.zip: transient HTTP error
06-Aug-2016 08:47:33 [World Community Grid] Temporarily failed download of 78013f13c41b0ab1bb008cd1adfbfd16.zip: transient HTTP error
06-Aug-2016 09:52:33 [World Community Grid] Temporarily failed download of 06b3fab035ccad13186b0a7dbd59451a.job: transient HTTP error
06-Aug-2016 10:57:18 [World Community Grid] Temporarily failed download of d82bf2349b244b6bca22f9fa09a36ae2.job: transient HTTP error
06-Aug-2016 12:08:56 [World Community Grid] Temporarily failed download of eddd9e86e056b56645bfb6aa2e8c9080.zip: transient HTTP error
06-Aug-2016 13:13:50 [World Community Grid] Temporarily failed download of 416ab7c9c61e70da59fffc2be127ca19.zip: transient HTTP error
06-Aug-2016 13:14:18 [World Community Grid] Temporarily failed download of 8850cfdf9e8965a442d7773d2d5cbbdf.job: transient HTTP error

I noticed some kind of pattern: after the initial connection to WCG, two downloads were started and one of them would fail with "transient HTTP error", catching up later on:
06-Aug-2016 03:12:53 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 03:12:53 [World Community Grid] Reporting 8 completed tasks
06-Aug-2016 03:12:53 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 03:13:08 [World Community Grid] Scheduler request completed: got 8 new tasks
06-Aug-2016 03:13:10 [World Community Grid] Started download of e4b0c42a03c599fb245248d8d5cc1341.job
06-Aug-2016 03:13:10 [World Community Grid] Started download of 83854fec7a5c07b7ac538a8363e68f58.zip
06-Aug-2016 03:13:34 [---] Project communication failed: attempting access to reference site
06-Aug-2016 03:13:34 [World Community Grid] Finished download of e4b0c42a03c599fb245248d8d5cc1341.job
06-Aug-2016 03:13:34 [World Community Grid] Temporarily failed download of 83854fec7a5c07b7ac538a8363e68f58.zip: transient HTTP error
06-Aug-2016 03:13:34 [World Community Grid] Backing off 00:02:53 on download of 83854fec7a5c07b7ac538a8363e68f58.zip
06-Aug-2016 03:13:34 [World Community Grid] Started download of a63aee90db7f7b35a322d172b3285c54.pdbqt
06-Aug-2016 03:13:34 [World Community Grid] Started download of 633ce3c62ef439b857596f88932c1362.job
06-Aug-2016 03:13:46 [---] Internet access OK - project servers may be temporarily down.
06-Aug-2016 03:13:46 [World Community Grid] Finished download of a63aee90db7f7b35a322d172b3285c54.pdbqt
06-Aug-2016 03:13:46 [World Community Grid] Finished download of 633ce3c62ef439b857596f88932c1362.job
06-Aug-2016 03:13:46 [World Community Grid] Started download of 64a89eb2c03c5ab48a9f56ea58b14d91.zip
06-Aug-2016 03:13:46 [World Community Grid] Started download of e06e2c4bd87ce8267f390c618c1f905e.pdbqt
06-Aug-2016 03:13:59 [World Community Grid] Finished download of 64a89eb2c03c5ab48a9f56ea58b14d91.zip
06-Aug-2016 03:13:59 [World Community Grid] Finished download of e06e2c4bd87ce8267f390c618c1f905e.pdbqt
06-Aug-2016 03:13:59 [World Community Grid] Started download of 08adfcca81992cf18687b488adbcc977.job
06-Aug-2016 03:13:59 [World Community Grid] Started download of 4e8009c2ec1e92d7d0956b4456e2255a.zip
06-Aug-2016 03:14:12 [World Community Grid] Finished download of 08adfcca81992cf18687b488adbcc977.job
06-Aug-2016 03:14:12 [World Community Grid] Finished download of 4e8009c2ec1e92d7d0956b4456e2255a.zip
06-Aug-2016 03:14:12 [World Community Grid] Started download of 290a89b947847406d1749602c992c9f3.pdbqt
06-Aug-2016 03:14:12 [World Community Grid] Started download of cae66c1049e8bc022bbc2c7988a5bd85.job
06-Aug-2016 03:14:24 [World Community Grid] Finished download of 290a89b947847406d1749602c992c9f3.pdbqt
06-Aug-2016 03:14:24 [World Community Grid] Started download of 7ed85237e9670e8358ba7536f96c8779.zip

06-Aug-2016 04:15:11 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 04:15:11 [World Community Grid] Reporting 4 completed tasks
06-Aug-2016 04:15:11 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 04:15:26 [World Community Grid] Scheduler request completed: got 4 new tasks
06-Aug-2016 04:15:28 [World Community Grid] Started download of bd0ae6eb9eb682759460ea74b7cc3aa8.job
06-Aug-2016 04:15:28 [World Community Grid] Started download of bc738010459817cdaaf4b3fc97819b31.zip
06-Aug-2016 04:15:53 [---] Project communication failed: attempting access to reference site
06-Aug-2016 04:15:53 [World Community Grid] Finished download of bd0ae6eb9eb682759460ea74b7cc3aa8.job
06-Aug-2016 04:15:53 [World Community Grid] Temporarily failed download of bc738010459817cdaaf4b3fc97819b31.zip: transient HTTP error
06-Aug-2016 04:15:53 [World Community Grid] Backing off 00:03:49 on download of bc738010459817cdaaf4b3fc97819b31.zip

06-Aug-2016 05:28:32 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 05:28:32 [World Community Grid] Reporting 8 completed tasks
06-Aug-2016 05:28:32 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 05:28:47 [World Community Grid] Scheduler request completed: got 8 new tasks
06-Aug-2016 05:28:49 [World Community Grid] Started download of a6425280bf74b4258090e9e67785e488.job
06-Aug-2016 05:28:49 [World Community Grid] Started download of 8b583407b9572b2d2a37453503350ea8.zip
06-Aug-2016 05:29:15 [---] Project communication failed: attempting access to reference site
06-Aug-2016 05:29:15 [World Community Grid] Finished download of a6425280bf74b4258090e9e67785e488.job
06-Aug-2016 05:29:15 [World Community Grid] Temporarily failed download of 8b583407b9572b2d2a37453503350ea8.zip: transient HTTP error

06-Aug-2016 06:31:05 [World Community Grid] Reporting 6 completed tasks
06-Aug-2016 06:31:05 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 06:31:20 [World Community Grid] Scheduler request completed: got 6 new tasks
06-Aug-2016 06:31:22 [World Community Grid] Started download of cf7f055d58032a178f4bc716c046951e.job
06-Aug-2016 06:31:22 [World Community Grid] Started download of bea59d90fbb3f531861a4187f5dac6ca.zip
06-Aug-2016 06:31:48 [---] Project communication failed: attempting access to reference site
06-Aug-2016 06:31:48 [World Community Grid] Finished download of cf7f055d58032a178f4bc716c046951e.job
06-Aug-2016 06:31:48 [World Community Grid] Temporarily failed download of bea59d90fbb3f531861a4187f5dac6ca.zip: transient HTTP error

06-Aug-2016 07:36:18 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 07:36:18 [World Community Grid] Reporting 6 completed tasks
06-Aug-2016 07:36:18 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 07:36:34 [World Community Grid] Scheduler request completed: got 6 new tasks
06-Aug-2016 07:36:36 [World Community Grid] Started download of fc13cdca988d5fc434709281370fce54.job
06-Aug-2016 07:36:36 [World Community Grid] Started download of 7aa6f07603ca6ec0088077b82f880b02.zip
06-Aug-2016 07:37:02 [---] Project communication failed: attempting access to reference site
06-Aug-2016 07:37:02 [World Community Grid] Finished download of fc13cdca988d5fc434709281370fce54.job
06-Aug-2016 07:37:02 [World Community Grid] Temporarily failed download of 7aa6f07603ca6ec0088077b82f880b02.zip: transient HTTP error

06-Aug-2016 08:46:48 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 08:46:48 [World Community Grid] Reporting 7 completed tasks
06-Aug-2016 08:46:48 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 08:47:04 [World Community Grid] Scheduler request completed: got 7 new tasks
06-Aug-2016 08:47:06 [World Community Grid] Started download of 65bf5941c4cb4b2ed0a9ef12b60de932.job
06-Aug-2016 08:47:06 [World Community Grid] Started download of 78013f13c41b0ab1bb008cd1adfbfd16.zip
06-Aug-2016 08:47:33 [---] Project communication failed: attempting access to reference site
06-Aug-2016 08:47:33 [World Community Grid] Finished download of 65bf5941c4cb4b2ed0a9ef12b60de932.job
06-Aug-2016 08:47:33 [World Community Grid] Temporarily failed download of 78013f13c41b0ab1bb008cd1adfbfd16.zip: transient HTTP error

etc.
06-Aug-2016 13:13:05 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 13:13:05 [World Community Grid] Reporting 8 completed tasks
06-Aug-2016 13:13:05 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 13:13:21 [World Community Grid] Scheduler request completed: got 8 new tasks
06-Aug-2016 13:13:23 [World Community Grid] Started download of 2d71ee739409f3af62b8ecaaaff57f24.job
06-Aug-2016 13:13:23 [World Community Grid] Started download of 416ab7c9c61e70da59fffc2be127ca19.zip
06-Aug-2016 13:13:50 [---] Project communication failed: attempting access to reference site
06-Aug-2016 13:13:50 [World Community Grid] Finished download of 2d71ee739409f3af62b8ecaaaff57f24.job
06-Aug-2016 13:13:50 [World Community Grid] Temporarily failed download of 416ab7c9c61e70da59fffc2be127ca19.zip: transient HTTP error
----------------------------------------
[Edit 5 times, last edit by adriverhoef at Aug 6, 2016 10:19:11 PM]
[Aug 6, 2016 4:15:10 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: Transient HTTP Errors, the continuing story...
I decided to tweak <max_file_xfers_per_project> in cc_config.xml and change it from 2 into 1.
In the next run of the scheduler request the transient HTTP error wasn't there.
So I decided to tweak <max_file_xfers_per_project> once more and change it into 8. Wait until scheduler request...
Then it appeared that BOINC was as handicapped as a fly with only one wing and a pain in its stomach, as hardly any upload/download succeeded at all:
06-Aug-2016 15:37:45 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 15:37:45 [World Community Grid] Reporting 10 completed tasks
06-Aug-2016 15:37:45 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 15:38:02 [World Community Grid] Scheduler request completed: got 10 new tasks
06-Aug-2016 15:38:04 [World Community Grid] Started download of 7577bbad3f8fc65261fa9d7b94b0d64a.job
06-Aug-2016 15:38:04 [World Community Grid] Started download of 2add410a6c4873a8079284224a838cb0.zip
06-Aug-2016 15:38:04 [World Community Grid] Started download of ffd352e5331ffc2c8cae8ee72be026b0.pdbqt
06-Aug-2016 15:38:04 [World Community Grid] Started download of 9ac6ec63c7e2d8964c08acdef45e46b6.job
06-Aug-2016 15:38:04 [World Community Grid] Started download of 7bd7531a4795755191d6ef69c69415b9.zip
06-Aug-2016 15:38:04 [World Community Grid] Started download of 31469da021fd7b0896f1e846f8c7e97a.pdbqt
06-Aug-2016 15:38:04 [World Community Grid] Started download of ff9f6fe822561bc46a47141ef5df7e5e.job
06-Aug-2016 15:38:04 [World Community Grid] Started download of 1324b6033765a5aa66f3deb3287e465b.zip
06-Aug-2016 15:39:52 [---] Project communication failed: attempting access to reference site
06-Aug-2016 15:39:52 [World Community Grid] Finished download of 7577bbad3f8fc65261fa9d7b94b0d64a.job
06-Aug-2016 15:39:52 [World Community Grid] Temporarily failed download of 2add410a6c4873a8079284224a838cb0.zip: transient HTTP error
06-Aug-2016 15:39:52 [World Community Grid] Backing off 00:03:48 on download of 2add410a6c4873a8079284224a838cb0.zip
06-Aug-2016 15:39:52 [World Community Grid] Temporarily failed download of ffd352e5331ffc2c8cae8ee72be026b0.pdbqt: transient HTTP error
06-Aug-2016 15:39:52 [World Community Grid] Backing off 00:02:50 on download of ffd352e5331ffc2c8cae8ee72be026b0.pdbqt
06-Aug-2016 15:39:52 [World Community Grid] Temporarily failed download of 9ac6ec63c7e2d8964c08acdef45e46b6.job: transient HTTP error
06-Aug-2016 15:39:52 [World Community Grid] Backing off 00:03:32 on download of 9ac6ec63c7e2d8964c08acdef45e46b6.job
06-Aug-2016 15:39:52 [World Community Grid] Temporarily failed download of 7bd7531a4795755191d6ef69c69415b9.zip: transient HTTP error
06-Aug-2016 15:39:52 [World Community Grid] Backing off 00:03:41 on download of 7bd7531a4795755191d6ef69c69415b9.zip
06-Aug-2016 15:39:52 [World Community Grid] Temporarily failed download of 31469da021fd7b0896f1e846f8c7e97a.pdbqt: transient HTTP error
06-Aug-2016 15:39:52 [World Community Grid] Backing off 00:02:29 on download of 31469da021fd7b0896f1e846f8c7e97a.pdbqt
06-Aug-2016 15:39:52 [World Community Grid] Temporarily failed download of ff9f6fe822561bc46a47141ef5df7e5e.job: transient HTTP error
06-Aug-2016 15:39:52 [World Community Grid] Backing off 00:03:01 on download of ff9f6fe822561bc46a47141ef5df7e5e.job
06-Aug-2016 15:39:52 [World Community Grid] Temporarily failed download of 1324b6033765a5aa66f3deb3287e465b.zip: transient HTTP error
06-Aug-2016 15:39:52 [World Community Grid] Backing off 00:03:28 on download of 1324b6033765a5aa66f3deb3287e465b.zip
06-Aug-2016 15:39:52 [World Community Grid] Started download of ff346719cd0b6e0ff4ef8fd96e71260b.pdbqt
06-Aug-2016 15:39:52 [World Community Grid] Started download of 6ed94b8fd3a27569daa97c783bfad712.job
06-Aug-2016 15:39:52 [World Community Grid] Started download of c5fd9823e4e9585bb67f41097035eedc.zip
06-Aug-2016 15:39:52 [World Community Grid] Started download of 74c6905f8428d670bac60c4dcf984495.pdbqt
06-Aug-2016 15:39:52 [World Community Grid] Started download of 9f82d75e53d9e808dd12fcf2de04b7d5.job
06-Aug-2016 15:39:52 [World Community Grid] Started download of 95ef1a0c174590e36cba1a115c55a7e5.zip
06-Aug-2016 15:39:52 [World Community Grid] Started download of 2d31f0ae091a2d1a1e1b49371d908356.pdbqt
06-Aug-2016 15:39:52 [World Community Grid] Started download of 3964e2166f9c5bd7e374cad4379df61c.job
06-Aug-2016 15:41:25 [World Community Grid] Finished download of ff346719cd0b6e0ff4ef8fd96e71260b.pdbqt
06-Aug-2016 15:41:25 [World Community Grid] Started download of 168d48e247297d58fa1f070c3d870b3a.zip
06-Aug-2016 15:41:32 [---] Internet access OK - project servers may be temporarily down.
06-Aug-2016 15:41:32 [World Community Grid] Temporarily failed download of c5fd9823e4e9585bb67f41097035eedc.zip: transient HTTP error
06-Aug-2016 15:41:32 [World Community Grid] Backing off 00:02:36 on download of c5fd9823e4e9585bb67f41097035eedc.zip
06-Aug-2016 15:41:32 [World Community Grid] Temporarily failed download of 74c6905f8428d670bac60c4dcf984495.pdbqt: transient HTTP error
06-Aug-2016 15:41:32 [World Community Grid] Backing off 00:02:02 on download of 74c6905f8428d670bac60c4dcf984495.pdbqt
06-Aug-2016 15:41:32 [World Community Grid] Temporarily failed download of 9f82d75e53d9e808dd12fcf2de04b7d5.job: transient HTTP error
06-Aug-2016 15:41:32 [World Community Grid] Backing off 00:02:29 on download of 9f82d75e53d9e808dd12fcf2de04b7d5.job
06-Aug-2016 15:41:32 [World Community Grid] Temporarily failed download of 95ef1a0c174590e36cba1a115c55a7e5.zip: transient HTTP error
06-Aug-2016 15:41:32 [World Community Grid] Backing off 00:02:47 on download of 95ef1a0c174590e36cba1a115c55a7e5.zip
06-Aug-2016 15:41:32 [World Community Grid] Temporarily failed download of 2d31f0ae091a2d1a1e1b49371d908356.pdbqt: transient HTTP error
06-Aug-2016 15:41:32 [World Community Grid] Backing off 00:02:39 on download of 2d31f0ae091a2d1a1e1b49371d908356.pdbqt
06-Aug-2016 15:41:32 [World Community Grid] Temporarily failed download of 3964e2166f9c5bd7e374cad4379df61c.job: transient HTTP error
06-Aug-2016 15:41:32 [World Community Grid] Backing off 00:02:11 on download of 3964e2166f9c5bd7e374cad4379df61c.job
06-Aug-2016 15:41:32 [World Community Grid] Started download of fec30f130dfc8ff13c928e8d7be07a38.pdbqt
06-Aug-2016 15:41:32 [World Community Grid] Started download of 22e608e55a2857f35a65604297c9749f.job
06-Aug-2016 15:41:32 [World Community Grid] Started download of 1103523a3faaf9fdf153ff118d949ff3.zip
06-Aug-2016 15:41:32 [World Community Grid] Started download of 45afae7e89ee77df90ecbd6df1462bf0.pdbqt
06-Aug-2016 15:41:32 [World Community Grid] Started download of 04b55c6e67e3fad2d17ea3d0a2cf00b9.job
06-Aug-2016 15:41:32 [World Community Grid] Started download of 2b26342cac1b5a2a88ff2bebca75dad8.zip
06-Aug-2016 15:42:30 [World Community Grid] Finished download of 6ed94b8fd3a27569daa97c783bfad712.job
06-Aug-2016 15:42:30 [World Community Grid] Started download of d730f2fb96301f5d020b7f8f7636fb94.pdbqt
06-Aug-2016 15:42:38 [---] Project communication failed: attempting access to reference site
06-Aug-2016 15:42:38 [World Community Grid] Finished download of 168d48e247297d58fa1f070c3d870b3a.zip
06-Aug-2016 15:42:38 [World Community Grid] Finished download of fec30f130dfc8ff13c928e8d7be07a38.pdbqt
06-Aug-2016 15:42:38 [World Community Grid] Temporarily failed download of 1103523a3faaf9fdf153ff118d949ff3.zip: transient HTTP error
06-Aug-2016 15:42:38 [World Community Grid] Backing off 00:03:20 on download of 1103523a3faaf9fdf153ff118d949ff3.zip
06-Aug-2016 15:42:38 [World Community Grid] Temporarily failed download of 45afae7e89ee77df90ecbd6df1462bf0.pdbqt: transient HTTP error
06-Aug-2016 15:42:38 [World Community Grid] Backing off 00:03:25 on download of 45afae7e89ee77df90ecbd6df1462bf0.pdbqt
06-Aug-2016 15:42:38 [World Community Grid] Temporarily failed download of 04b55c6e67e3fad2d17ea3d0a2cf00b9.job: transient HTTP error
06-Aug-2016 15:42:38 [World Community Grid] Backing off 00:03:34 on download of 04b55c6e67e3fad2d17ea3d0a2cf00b9.job
06-Aug-2016 15:42:38 [World Community Grid] Temporarily failed download of 2b26342cac1b5a2a88ff2bebca75dad8.zip: transient HTTP error
06-Aug-2016 15:42:38 [World Community Grid] Backing off 00:03:46 on download of 2b26342cac1b5a2a88ff2bebca75dad8.zip
06-Aug-2016 15:42:38 [World Community Grid] Started download of ac0bda0c9e617b8b8cf0dac5789886e0.job
06-Aug-2016 15:42:38 [World Community Grid] Started download of cbc7dcb17f5040cbd75232b8cab93820.zip
06-Aug-2016 15:42:38 [World Community Grid] Started download of 8a1d0855bf73c390c7c53b212929f4b9.pdbqt
06-Aug-2016 15:42:38 [World Community Grid] Started download of be9b74f6c36b2e8eb70396a46315af77.job
06-Aug-2016 15:42:38 [World Community Grid] Started download of 929fae3f785541a45ecc5cf0ad1021f8.zip
06-Aug-2016 15:42:38 [World Community Grid] Started download of bd025569775e576ea09ee27618748c5d.pdbqt
06-Aug-2016 15:43:17 [---] Internet access OK - project servers may be temporarily down.
06-Aug-2016 15:43:17 [World Community Grid] Finished download of 22e608e55a2857f35a65604297c9749f.job
06-Aug-2016 15:43:44 [---] Project communication failed: attempting access to reference site
06-Aug-2016 15:43:44 [World Community Grid] Started download of 2add410a6c4873a8079284224a838cb0.zip
06-Aug-2016 15:43:44 [World Community Grid] Finished download of d730f2fb96301f5d020b7f8f7636fb94.pdbqt
06-Aug-2016 15:43:44 [World Community Grid] Finished download of ac0bda0c9e617b8b8cf0dac5789886e0.job
06-Aug-2016 15:43:44 [World Community Grid] Temporarily failed download of 929fae3f785541a45ecc5cf0ad1021f8.zip: transient HTTP error
06-Aug-2016 15:43:44 [World Community Grid] Backing off 00:03:42 on download of 929fae3f785541a45ecc5cf0ad1021f8.zip
06-Aug-2016 15:43:44 [World Community Grid] Temporarily failed download of bd025569775e576ea09ee27618748c5d.pdbqt: transient HTTP error
06-Aug-2016 15:43:44 [World Community Grid] Backing off 00:02:49 on download of bd025569775e576ea09ee27618748c5d.pdbqt
06-Aug-2016 15:44:07 [---] Internet access OK - project servers may be temporarily down.
06-Aug-2016 15:44:07 [World Community Grid] Started download of ffd352e5331ffc2c8cae8ee72be026b0.pdbqt
06-Aug-2016 15:44:07 [World Community Grid] Started download of 9ac6ec63c7e2d8964c08acdef45e46b6.job
06-Aug-2016 15:44:07 [World Community Grid] Started download of 7bd7531a4795755191d6ef69c69415b9.zip
06-Aug-2016 15:44:07 [World Community Grid] Started download of 31469da021fd7b0896f1e846f8c7e97a.pdbqt
06-Aug-2016 15:44:07 [World Community Grid] Temporarily failed download of cbc7dcb17f5040cbd75232b8cab93820.zip: transient HTTP error
06-Aug-2016 15:44:07 [World Community Grid] Backing off 00:03:31 on download of cbc7dcb17f5040cbd75232b8cab93820.zip
06-Aug-2016 15:44:07 [World Community Grid] Temporarily failed download of 8a1d0855bf73c390c7c53b212929f4b9.pdbqt: transient HTTP error
06-Aug-2016 15:44:07 [World Community Grid] Backing off 00:03:34 on download of 8a1d0855bf73c390c7c53b212929f4b9.pdbqt
06-Aug-2016 15:44:07 [World Community Grid] Started upload of OET1_0003944_x4GVE_rig_47528_0_r1359594804_0
06-Aug-2016 15:45:15 [---] Project communication failed: attempting access to reference site
06-Aug-2016 15:45:15 [World Community Grid] Finished download of be9b74f6c36b2e8eb70396a46315af77.job
06-Aug-2016 15:45:15 [World Community Grid] Temporarily failed upload of OET1_0003944_x4GVE_rig_47525_0_r2132575028_0: transient HTTP error
06-Aug-2016 15:45:15 [World Community Grid] Backing off 00:03:03 on upload of OET1_0003944_x4GVE_rig_47525_0_r2132575028_0
06-Aug-2016 15:45:16 [---] Internet access OK - project servers may be temporarily down.
06-Aug-2016 15:45:16 [World Community Grid] Finished download of 2add410a6c4873a8079284224a838cb0.zip
06-Aug-2016 15:45:16 [World Community Grid] Finished download of ffd352e5331ffc2c8cae8ee72be026b0.pdbqt
06-Aug-2016 15:45:16 [World Community Grid] Temporarily failed download of 9ac6ec63c7e2d8964c08acdef45e46b6.job: transient HTTP error
06-Aug-2016 15:45:16 [World Community Grid] Backing off 00:04:00 on download of 9ac6ec63c7e2d8964c08acdef45e46b6.job
06-Aug-2016 15:45:16 [World Community Grid] Temporarily failed download of 7bd7531a4795755191d6ef69c69415b9.zip: transient HTTP error
06-Aug-2016 15:45:16 [World Community Grid] Backing off 00:06:32 on download of 7bd7531a4795755191d6ef69c69415b9.zip
06-Aug-2016 15:45:16 [World Community Grid] Temporarily failed download of 31469da021fd7b0896f1e846f8c7e97a.pdbqt: transient HTTP error
06-Aug-2016 15:45:16 [World Community Grid] Backing off 00:05:00 on download of 31469da021fd7b0896f1e846f8c7e97a.pdbqt
06-Aug-2016 15:45:16 [World Community Grid] Temporarily failed upload of OET1_0003944_x4GVE_rig_47528_0_r1359594804_0: transient HTTP error
06-Aug-2016 15:45:16 [World Community Grid] Backing off 00:02:47 on upload of OET1_0003944_x4GVE_rig_47528_0_r1359594804_0

Disastrous behaviour, I would say, so I tweaked <max_file_xfers_per_project> once more and set it to 1 again.
06-Aug-2016 15:45:54 [---] Re-reading cc_config.xml
06-Aug-2016 15:45:54 [---] Config: use all coprocessors
06-Aug-2016 15:45:54 [---] log flags: file_xfer, sched_ops, task, unparsed_xml
Result for the moment:
06-Aug-2016 16:45:30 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 16:45:30 [World Community Grid] Reporting 16 completed tasks
06-Aug-2016 16:45:30 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 16:45:47 [World Community Grid] Scheduler request completed: got 16 new tasks
06-Aug-2016 16:45:49 [World Community Grid] Started download of 6a550ae6bbccc7e8d0bbf6ca72a70a86.job
06-Aug-2016 16:46:05 [World Community Grid] Finished download of 6a550ae6bbccc7e8d0bbf6ca72a70a86.job
06-Aug-2016 16:46:05 [World Community Grid] Started download of 2b54505fec0a5aad928109e557f9542a.zip
06-Aug-2016 16:46:06 [World Community Grid] Finished download of 2b54505fec0a5aad928109e557f9542a.zip
06-Aug-2016 16:46:06 [World Community Grid] Started download of 0d8539d60675bc4df020e084ef954dbf.pdbqt
06-Aug-2016 16:46:07 [World Community Grid] Finished download of 0d8539d60675bc4df020e084ef954dbf.pdbqt
etc., and one hour later:
06-Aug-2016 17:51:46 [World Community Grid] Sending scheduler request: To report completed tasks.
06-Aug-2016 17:51:46 [World Community Grid] Reporting 19 completed tasks
06-Aug-2016 17:51:46 [World Community Grid] Requesting new tasks for CPU
06-Aug-2016 17:52:04 [World Community Grid] Computation for task OET1_0003944_x4GVE_rig_56868_0 finished
06-Aug-2016 17:52:04 [World Community Grid] Starting task OET1_0003944_x4GVE_rig_88314_0
06-Aug-2016 17:52:04 [World Community Grid] Scheduler request completed: got 19 new tasks
06-Aug-2016 17:52:06 [World Community Grid] Started upload of OET1_0003944_x4GVE_rig_56868_0_r1717794166_0
06-Aug-2016 17:52:06 [World Community Grid] Started download of 8b1484a142eb1d96ec200f60b6c339de.job
06-Aug-2016 17:52:37 [---] Project communication failed: attempting access to reference site
(Yeah, well maybe we can't have an upload and a download together at the same time anymore now that I've set <max_file_xfers_per_project> to 1.)
06-Aug-2016 17:52:37 [World Community Grid] Temporarily failed upload of OET1_0003944_x4GVE_rig_56868_0_r1717794166_0: transient HTTP error
06-Aug-2016 17:52:37 [World Community Grid] Backing off 00:03:03 on upload of OET1_0003944_x4GVE_rig_56868_0_r1717794166_0
06-Aug-2016 17:52:37 [World Community Grid] Finished download of 8b1484a142eb1d96ec200f60b6c339de.job
06-Aug-2016 17:52:37 [World Community Grid] Started download of f10ea17dfd75ac114638d7303b62898f.zip
06-Aug-2016 17:52:38 [World Community Grid] Finished download of f10ea17dfd75ac114638d7303b62898f.zip
06-Aug-2016 17:52:38 [World Community Grid] Started download of 396aa88ef01505e68499986727752215.pdbqt
06-Aug-2016 17:52:39 [---] Internet access OK - project servers may be temporarily down.
06-Aug-2016 17:52:39 [World Community Grid] Finished download of 396aa88ef01505e68499986727752215.pdbqt
06-Aug-2016 17:52:39 [World Community Grid] Started download of 6d40ce2935814d246a54d59248db7627.job
06-Aug-2016 17:52:41 [World Community Grid] Finished download of 6d40ce2935814d246a54d59248db7627.job
etc.

I'm hoping though that the transient HTTP errors are mostly over now.
[Aug 6, 2016 4:15:39 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: Transient HTTP Errors, the continuing story...
By the way, before my girlfriend restarted the BOINC service, she also enabled <unparsed_xml> in cc_config.xml, as suggested by SekeRob. Up until now I haven't seen anything rearing its ugly head in living colour though.
[Aug 6, 2016 5:01:23 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Transient HTTP Errors, the continuing story...
Hi adriverhoef.
Your not the only one having problems here, I've been seeing uploads, downloads & reporting fail for the last couple of days. As well as the web site being slow or unreachable at times plus being logged out a few times a day, it just seems to be getting worse I don't know if it's just to many crunchers here or what. I'm not someone that changes files or anything and I'm not having problems anywhere else, so I don't think that it's my end.
Hopefully they can fix whatever problems are, here's hoping.

edit / New day same problems, like so.
Sun 07 Aug 2016 08:13:55 AEST | World Community Grid | Computation for task ZIKA_000052042_x2whx_DV_NS3hlcsDelMn_PRwasBnd_0497_0 finished
Sun 07 Aug 2016 08:13:55 AEST | World Community Grid | Starting task ZIKA_000052044_x2whx_DV_NS3hlcsDelMn_PRwasBnd_0112_0
Sun 07 Aug 2016 08:13:57 AEST | World Community Grid | Started upload of ZIKA_000052042_x2whx_DV_NS3hlcsDelMn_PRwasBnd_0497_0_r2070741365_0
Sun 07 Aug 2016 08:14:03 AEST | World Community Grid | Finished upload of ZIKA_000052042_x2whx_DV_NS3hlcsDelMn_PRwasBnd_0497_0_r2070741365_0
Sun 07 Aug 2016 08:14:12 AEST | World Community Grid | update requested by user
Sun 07 Aug 2016 08:15:01 AEST | | Project communication failed: attempting access to reference site
Sun 07 Aug 2016 08:15:01 AEST | World Community Grid | Scheduler request failed: Timeout was reached
Sun 07 Aug 2016 08:15:03 AEST | | Internet access OK - project servers may be temporarily down.
----------------------------------------
[Edit 2 times, last edit by Former Member at Aug 6, 2016 10:19:48 PM]
[Aug 6, 2016 9:58:21 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: Transient HTTP Errors, the continuing story...
Thanks, P . P . L .

This just in:
07-Aug-2016 01:03:11 [World Community Grid] Error reported by file upload server: Message from server: We are currently experiencing high load and are temporarily deferring your file upload request. Your client will automatically try again later.
("transient upload error")
[Aug 6, 2016 11:08:16 PM]   Link   Report threatening or abusive post: please login first  Go to top 
uplinger
Former World Community Grid Tech
Joined: May 23, 2005
Post Count: 3952
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Transient HTTP Errors, the continuing story...
We are working to offload some of the issues we are seeing to a swift object storage. The first tests of this have been the slideshow images for all projects as well as the latest HST1 beta test. We are seeing good results and hope this will help decrease the download issues members are seeing.

Thanks,
-Uplinger
[Aug 12, 2016 7:49:33 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: Transient HTTP Errors, the continuing story...
There was a blown fuse fifteen days ago while I wasn't at home and my computer (running Fedora 23, IPv6) had to be rebooted after I came home, five days later. That was 10 days ago.
The BOINC service on my computer was also restarted automatically ten days ago and has been running continuously since then:
$ ps -fuboinc
UID        PID  PPID  C STIME TTY          TIME CMD
boinc     1987     1  3 Aug24 ?        08:02:16 /usr/bin/boinc_client --daemon --start_delay 1
...
One week later of continuously running BOINC the first transient HTTP error started to appear and one day later the errors started piling up:
$ grep transient stdoutdae.txt | head
01-Sep-2016 06:37:19 [World Community Grid] Temporarily failed upload of OET1_0004042_x3MX2_rig_84313_0_r1458046998_0: transient HTTP error
02-Sep-2016 06:51:20 [World Community Grid] Temporarily failed upload of OET1_0004051_x3MX5p_rig_17527_0_r973133033_0: transient HTTP error
02-Sep-2016 09:17:28 [World Community Grid] Temporarily failed upload of OET1_0004051_x3MX5p_rig_0734_0_r1277429079_0: transient HTTP error
02-Sep-2016 10:30:01 [World Community Grid] Temporarily failed upload of OET1_0004051_x3MX5p_rig_55220_0_r1532999125_0: transient HTTP error
02-Sep-2016 11:18:24 [World Community Grid] Temporarily failed upload of OET1_0004051_x3MX5p_rig_97954_0_r56293886_0: transient HTTP error
02-Sep-2016 11:31:27 [World Community Grid] Temporarily failed upload of OET1_0004051_x3MX5p_rig_98839_0_r1280233199_0: transient HTTP error
02-Sep-2016 12:35:11 [World Community Grid] Temporarily failed upload of OET1_0004051_x3MX5p_rig_59599_0_r25770343_0: transient HTTP error
02-Sep-2016 12:43:32 [World Community Grid] Temporarily failed upload of OET1_0004051_x3MX5p_rig_6359_0_r1316046314_0: transient HTTP error
02-Sep-2016 13:13:06 [World Community Grid] Temporarily failed download of 6784a17ed9602377a8ed4fc5d4af8018.job: transient HTTP error
02-Sep-2016 13:16:50 [World Community Grid] Temporarily failed upload of OET1_0004052_x3MX5p_rig_12811_0_r1663156814_0: transient HTTP error
... "but still they come" ...
03-Sep-2016 14:15:07 [World Community Grid] Temporarily failed upload of OET1_0004057_x3MX5p_rig_87006_0_r1324975285_0: transient HTTP error
03-Sep-2016 14:19:11 [World Community Grid] Temporarily failed upload of OET1_0004057_x3MX5p_rig_87006_0_r1324975285_0: transient HTTP error
03-Sep-2016 14:21:35 [World Community Grid] Temporarily failed upload of OET1_0004057_x3MX5p_rig_86698_0_r939911575_0: transient HTTP error
03-Sep-2016 14:30:38 [World Community Grid] Temporarily failed upload of OET1_0004057_x3MX5p_rig_87003_0_r2003114573_0: transient HTTP error

I decided to enable 'http_debug' in the Event Log today and saw this flying by:
03-Sep-2016 13:56:46 [World Community Grid] Started upload of OET1_0004057_x3MX5p_rig_91336_0_r1116164963_0
03-Sep-2016 13:56:47 [World Community Grid] [http] [ID#7539] Info:  Hostname in DNS cache was stale, zapped
03-Sep-2016 13:56:47 [World Community Grid] [http] [ID#7539] Info:    Trying 198.20.8.241...
03-Sep-2016 13:56:47 [World Community Grid] [http] [ID#7539] Info:  Connected to grid.worldcommunitygrid.org (198.20.8.241) port 443 (#2669)
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:    CAfile: /etc/pki/tls/certs/ca-bundle.crt
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:    CApath: none
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:  ALPN/NPN, server did not agree to a protocol
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:  SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:  Server certificate:
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:      subject: CN=*.worldcommunitygrid.org,O=International Business Machines Corporation,L=Armonk,ST=New York,C=US
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:      start date: Jun 23 00:00:00 2015 GMT
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:      expire date: Aug 21 23:59:59 2017 GMT
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:      common name: *.worldcommunitygrid.org
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:      issuer: CN=thawte SSL CA - G2,O="thawte, Inc.",C=US
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: POST /boinc/wcg_cgi/file_upload_handler HTTP/1.1
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: Host: grid.worldcommunitygrid.org
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.6.22)
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: Accept: */*
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: Accept-Encoding: deflate, gzip
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: Content-Type: application/x-www-form-urlencoded
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: Accept-Language: en_GB
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: Content-Length: 298
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Sent header to server: 
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:  We are completely uploaded and fine 
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Received header from server: HTTP/1.1 200 OK
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Received header from server: Date: Sat, 03 Sep 2016 11:57:02 GMT
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Received header from server: Vary: Accept-Encoding
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Received header from server: Content-Encoding: gzip
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Received header from server: X-Frame-Options: SAMEORIGIN
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Received header from server: Content-Length: 75
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Received header from server: Content-Type: text/plain
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Received header from server: 
03-Sep-2016 13:57:02 [World Community Grid] [http] [ID#7539] Info:  Connection #2669 to host grid.worldcommunitygrid.org left intact
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Info:  Found bundle for host grid.worldcommunitygrid.org: 0x558948c97280
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Info:  Re-using existing connection! (#2669) with host grid.worldcommunitygrid.org
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Info:  Connected to grid.worldcommunitygrid.org (198.20.8.241) port 443 (#2669)
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: POST /boinc/wcg_cgi/file_upload_handler HTTP/1.1
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: Host: grid.worldcommunitygrid.org
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.6.22)
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: Accept: */*
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: Accept-Encoding: deflate, gzip
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: Content-Type: application/x-www-form-urlencoded
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: Accept-Language: en_GB
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: Content-Length: 13959
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: Expect: 100-continue
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Sent header to server: 
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: HTTP/1.1 100 Continue
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Info:  We are completely uploaded and fine 
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: HTTP/1.1 200 OK
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: Date: Sat, 03 Sep 2016 11:57:03 GMT
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: Vary: Accept-Encoding
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: Content-Encoding: gzip
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: X-Frame-Options: SAMEORIGIN
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: Content-Length: 60
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: Content-Type: text/plain
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Received header from server: 
03-Sep-2016 13:57:03 [World Community Grid] [http] [ID#7539] Info:  Connection #2669 to host grid.worldcommunitygrid.org left intact
03-Sep-2016 13:57:04 [World Community Grid] Finished upload of OET1_0004057_x3MX5p_rig_91336_0_r1116164963_0 
03-Sep-2016 14:14:37 [World Community Grid] Computation for task OET1_0004057_x3MX5p_rig_87006_0 finished
03-Sep-2016 14:14:37 [World Community Grid] Starting task OET1_0004061_x3MX5p_rig_12384_0
03-Sep-2016 14:14:40 [World Community Grid] Started upload of OET1_0004057_x3MX5p_rig_87006_0_r1324975285_0
03-Sep-2016 14:14:40 [World Community Grid] [http] [ID#7540] Info:  Connection 2668 seems to be dead!
03-Sep-2016 14:14:40 [World Community Grid] [http] [ID#7540] Info:  Closing connection 2668
03-Sep-2016 14:14:40 [World Community Grid] [http] [ID#7540] Info:  Connection 2669 seems to be dead!
03-Sep-2016 14:14:40 [World Community Grid] [http] [ID#7540] Info:  Closing connection 2669
03-Sep-2016 14:14:40 [World Community Grid] [http] [ID#7540] Info:  Connection 2667 seems to be dead!
03-Sep-2016 14:14:40 [World Community Grid] [http] [ID#7540] Info:  Closing connection 2667
03-Sep-2016 14:14:41 [World Community Grid] [http] [ID#7540] Info:    Trying 198.20.8.241...
03-Sep-2016 14:14:41 [World Community Grid] [http] [ID#7540] Info:  Connected to grid.worldcommunitygrid.org (198.20.8.241) port 443 (#2670)
03-Sep-2016 14:15:06 [World Community Grid] [http] [ID#7540] Info:    CAfile: /etc/pki/tls/certs/ca-bundle.crt
03-Sep-2016 14:15:06 [World Community Grid] [http] [ID#7540] Info:    CApath: none
03-Sep-2016 14:15:06 [World Community Grid] [http] [ID#7540] Info:  NSS error -5961 (PR_CONNECT_RESET_ERROR) 
03-Sep-2016 14:15:06 [World Community Grid] [http] [ID#7540] Info:  TCP connection reset by peer
03-Sep-2016 14:15:06 [World Community Grid] [http] [ID#7540] Info:  Closing connection 2670
03-Sep-2016 14:15:06 [World Community Grid] [http] HTTP error: SSL connect error
03-Sep-2016 14:15:07 [---] Project communication failed: attempting access to reference site
03-Sep-2016 14:15:07 [---] [http] HTTP_OP::init_get(): http://www.google.com/
03-Sep-2016 14:15:07 [World Community Grid] Temporarily failed upload of OET1_0004057_x3MX5p_rig_87006_0_r1324975285_0: transient HTTP error
03-Sep-2016 14:15:07 [World Community Grid] Backing off 00:03:36 on upload of OET1_0004057_x3MX5p_rig_87006_0_r1324975285_0
...
I decided to google for "NSS error 5961 (PR_CONNECT_RESET_ERROR)" and found http://superuser.com/questions/916077/what-is...61-pr-connect-reset-error
"This error from NSS is the same error you get with openssl (errno=104: ECONNRESET). This simply means, that the peer or some middlebox in between (firewall) is terminating the connection."
"Since the site is reachable from my place I would suggest, that there is a firewall on your site blocking the connection. The behavior is fairly typical for DPI firewalls in that the initial TCP connection is allowed but once you send the first data (ClientHello from TLS handshake) it will determine if your access is allowed by policy and let it pass or deny it by injecting a TCP RST."

So I wondered: Is the site reachable from my place?
$ curl -v grid.worldcommunitygrid.org:443
* Rebuilt URL to: grid.worldcommunitygrid.org:443/
* Trying 198.20.8.241...
* Connected to grid.worldcommunitygrid.org (198.20.8.241) port 443 (#0)
> GET / HTTP/1.1
> Host: grid.worldcommunitygrid.org:443
> User-Agent: curl/7.43.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
$ openssl s_client -connect grid.worldcommunitygrid.org:443 -msg
CONNECTED(00000003)
>>> TLS 1.2 [length 0005]
16 03 01 00 c4
>>> TLS 1.2 Handshake [length 00c4], ClientHello
01 00 00 c0 03 03 95 89 84 17 b9 76 5b 7e e5 86
...
0f 00 01 01
<<< ??? [length 0005]
16 03 03 13 02
<<< TLS 1.2 Handshake [length 0057], ServerHello
02 00 00 53 03 03 00 00 00 21 e2 89 6b 5e ee 28
...
01 00 ff 01 00 01 00
<<< TLS 1.2 Handshake [length 113a], Certificate
0b 00 11 36 00 11 33 00 04 ee 30 82 04 ea 30 82
...
5a e1 60 dc 64 23 3c 42 2e 4d
depth=3 C = ZA, ST = Western Cape, L = Cape Town, O = Thawte Consulting cc, OU = Certification Services Division, CN = Thawte Premium Server CA, emailAddress = premium-server@thawte.com
verify return:1
depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2006 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA
verify return:1
depth=1 C = US, O = "thawte, Inc.", CN = thawte SSL CA - G2
verify return:1
depth=0 C = US, ST = New York, L = Armonk, O = International Business Machines Corporation, CN = *.worldcommunitygrid.org
verify return:1
<<< TLS 1.2 Handshake [length 016d], ServerKeyExchange
0c 00 01 69 03 00 18 61 04 09 49 92 32 8e 9b a7
...
8a a1 da 45 6a 26 19 83 25 af 87 7c 76
<<< TLS 1.2 Handshake [length 0004], ServerHelloDone
0e 00 00 00
>>> ??? [length 0005]
16 03 03 00 66
>>> TLS 1.2 Handshake [length 0066], ClientKeyExchange
10 00 00 62 61 04 d0 5e 1e 84 88 5c 1a fa c6 7a
...
b7 79 75 2a 0d 7a
>>> ??? [length 0005]
14 03 03 00 01
>>> TLS 1.2 ChangeCipherSpec [length 0001]
01
>>> ??? [length 0005]
16 03 03 00 28
>>> TLS 1.2 Handshake [length 0010], Finished
14 00 00 0c f3 fc f6 35 84 18 d1 f8 54 c4 09 ec
<<< ??? [length 0005]
14 03 03 00 01
<<< TLS 1.2 ChangeCipherSpec [length 0001]
01
<<< ??? [length 0005]
16 03 03 00 28
<<< TLS 1.2 Handshake [length 0010], Finished
14 00 00 0c 58 2f 1d 41 67 8d 8f e2 ac c1 fa d1
---
Certificate chain
0 s:/C=US/ST=New York/L=Armonk/O=International Business Machines Corporation/CN=*.worldcommunitygrid.org
i:/C=US/O=thawte, Inc./CN=thawte SSL CA - G2
1 s:/C=US/O=thawte, Inc./CN=thawte SSL CA - G2
i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
2 s:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/emailAddress=premium-server@thawte.com
3 s:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/emailAddress=premium-server@thawte.com
i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/emailAddress=premium-server@thawte.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE6jCCA9KgAwIBAgIQNXoRJKPf4lxku9WckEN/ATANBgkqhkiG9w0BAQsFADBB
...
Zue/Fo2NxwPOtLQh57A=
-----END CERTIFICATE-----
subject=/C=US/ST=New York/L=Armonk/O=International Business Machines Corporation/CN=*.worldcommunitygrid.org
issuer=/C=US/O=thawte, Inc./CN=thawte SSL CA - G2
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 4922 bytes and written 359 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: B711000072496F686BE3925433861B3888B2955E5858585821C4CA570000010B
Session-ID-ctx:
Master-Key: 8F073371E3BE8C6C2AB01E6AEE494C4CDEEC70D405254B3D9CB8498C2D107495C498934066D4D8DBB23013512A08DB1F
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1472906273
Timeout : 300 (sec)
Verify return code: 0 (ok)
---

I've cut away many, many lines of output from the output of the 'openssl' command.
Since the site grid.worldcommunitygrid.org is reachable from my place, I guess there's no firewall or router blocking the road on my side.

So far so good, or ...?

Sometimes they will upload fine ...
03-Sep-2016 14:54:53 [World Community Grid] Started upload of OET1_0004057_x3MX5p_rig_88869_0_r1876110144_0
03-Sep-2016 14:54:55 [World Community Grid] Finished upload of OET1_0004057_x3MX5p_rig_88869_0_r1876110144_0
... and six minutes later they will fail ...
03-Sep-2016 15:00:49 [World Community Grid] Computation for task OET1_0004057_x3MX5p_rig_92355_0 finished
03-Sep-2016 15:00:49 [World Community Grid] Starting task OET1_0004061_x3MX5p_rig_3467_0
03-Sep-2016 15:00:51 [World Community Grid] Started upload of OET1_0004057_x3MX5p_rig_92355_0_r1414086952_0
03-Sep-2016 15:00:52 [World Community Grid] [http] [ID#7552] Info: Connection 2685 seems to be dead!
03-Sep-2016 15:00:52 [World Community Grid] [http] [ID#7552] Info: Closing connection 2685
03-Sep-2016 15:00:52 [World Community Grid] [http] [ID#7552] Info: Connection 2686 seems to be dead!
03-Sep-2016 15:00:52 [World Community Grid] [http] [ID#7552] Info: Closing connection 2686
03-Sep-2016 15:00:52 [World Community Grid] [http] [ID#7552] Info: Connection 2684 seems to be dead!
03-Sep-2016 15:00:52 [World Community Grid] [http] [ID#7552] Info: Closing connection 2684
03-Sep-2016 15:00:52 [World Community Grid] [http] [ID#7552] Info: Trying 198.20.8.241...
03-Sep-2016 15:00:52 [World Community Grid] [http] [ID#7552] Info: Connected to grid.worldcommunitygrid.org (198.20.8.241) port 443 (#2687)
03-Sep-2016 15:01:18 [World Community Grid] [http] [ID#7552] Info: CAfile: /etc/pki/tls/certs/ca-bundle.crt
03-Sep-2016 15:01:18 [World Community Grid] [http] [ID#7552] Info: CApath: none
03-Sep-2016 15:01:18 [World Community Grid] [http] [ID#7552] Info: NSS error -5961 (PR_CONNECT_RESET_ERROR)
03-Sep-2016 15:01:18 [World Community Grid] [http] [ID#7552] Info: TCP connection reset by peer
03-Sep-2016 15:01:18 [World Community Grid] [http] [ID#7552] Info: Closing connection 2687
03-Sep-2016 15:01:18 [World Community Grid] [http] HTTP error: SSL connect error
03-Sep-2016 15:01:19 [---] Project communication failed: attempting access to reference site
03-Sep-2016 15:01:19 [---] [http] HTTP_OP::init_get(): http://www.google.com/
03-Sep-2016 15:01:19 [World Community Grid] Temporarily failed upload of OET1_0004057_x3MX5p_rig_92355_0_r1414086952_0: transient HTTP error
03-Sep-2016 15:01:19 [World Community Grid] Backing off 00:03:19 on upload of OET1_0004057_x3MX5p_rig_92355_0_r1414086952_0
03-Sep-2016 15:01:20 [---] [http] [ID#0] Info: Trying 2a00:1450:4013:c01::69...
03-Sep-2016 15:01:20 [---] [http] [ID#0] Info: Connected to www.google.com (2a00:1450:4013:c01::69) port 80 (#2688)
03-Sep-2016 15:01:20 [---] [http] [ID#0] Sent header to server: GET / HTTP/1.1
03-Sep-2016 15:01:20 [---] [http] [ID#0] Sent header to server: Host: www.google.com
03-Sep-2016 15:01:20 [---] [http] [ID#0] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.6.22)
03-Sep-2016 15:01:20 [---] [http] [ID#0] Sent header to server: Accept: */*
03-Sep-2016 15:01:20 [---] [http] [ID#0] Sent header to server: Accept-Encoding: deflate, gzip
03-Sep-2016 15:01:20 [---] [http] [ID#0] Sent header to server: Content-Type: application/x-www-form-urlencoded
03-Sep-2016 15:01:20 [---] [http] [ID#0] Sent header to server: Accept-Language: en_GB
03-Sep-2016 15:01:20 [---] [http] [ID#0] Sent header to server:
03-Sep-2016 15:01:20 [---] [http] [ID#0] Received header from server: HTTP/1.1 302 Found
03-Sep-2016 15:01:20 [---] [http] [ID#0] Received header from server: Cache-Control: private
03-Sep-2016 15:01:20 [---] [http] [ID#0] Received header from server: Content-Type: text/html; charset=UTF-8
03-Sep-2016 15:01:20 [---] [http] [ID#0] Received header from server: Location: http://www.google.nl/?gfe_rd=cr&ei=oMnKV7POEoef-wbGypa4CQ
03-Sep-2016 15:01:20 [---] [http] [ID#0] Received header from server: Content-Length: 258
03-Sep-2016 15:01:20 [---] [http] [ID#0] Received header from server: Date: Sat, 03 Sep 2016 13:01:20 GMT
03-Sep-2016 15:01:20 [---] [http] [ID#0] Received header from server:
03-Sep-2016 15:01:20 [---] [http] [ID#0] Info: Ignoring the response-body
03-Sep-2016 15:01:20 [---] [http] [ID#0] Info: Connection #2688 to host www.google.com left intact
03-Sep-2016 15:01:20 [---] [http] [ID#0] Info: Issue another request to this URL: 'http://www.google.nl/?gfe_rd=cr&ei=oMnKV7POEoef-wbGypa4CQ'
03-Sep-2016 15:01:21 [---] [http] [ID#0] Info: Trying 2a00:1450:4013:c01::5e...
03-Sep-2016 15:01:21 [---] [http] [ID#0] Info: Connected to www.google.nl (2a00:1450:4013:c01::5e) port 80 (#2689)
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server: GET /?gfe_rd=cr&ei=oMnKV7POEoef-wbGypa4CQ HTTP/1.1
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server: Host: www.google.nl
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.6.22)
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server: Accept: */*
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server: Accept-Encoding: deflate, gzip
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server: Referer: http://www.google.com/
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server: Content-Type: application/x-www-form-urlencoded
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server: Accept-Language: en_GB
03-Sep-2016 15:01:21 [---] [http] [ID#0] Sent header to server:
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: HTTP/1.1 200 OK
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: Date: Sat, 03 Sep 2016 13:01:21 GMT
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: Expires: -1
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: Cache-Control: private, max-age=0
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: Content-Type: text/html; charset=ISO-8859-1
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: Content-Encoding: gzip
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: Server: gws
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: Content-Length: 4695
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: X-XSS-Protection: 1; mode=block
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: X-Frame-Options: SAMEORIGIN
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server: Set-Cookie: NID=85=HSnXdp11bcuALJtOP17Pzhb7C6n8qUY8DEvhIjeIN53LaUXNYxAJc-cqFkFh3o6UYR4X7rRRrMzM-T-uv93XtYQkbX1D-cr1twVAwgqdFpDqOXWzKhoJbkrqEQJ4kqMu; expires=Sun, 05-Ma
r-2017 13:01:21 GMT; path=/; domain=.google.nl; HttpOnly
03-Sep-2016 15:01:21 [---] [http] [ID#0] Received header from server:
03-Sep-2016 15:01:21 [---] [http] [ID#0] Info: Connection #2689 to host www.google.nl left intact
03-Sep-2016 15:01:21 [---] Internet access OK - project servers may be temporarily down.

One solution I know is to restart the BOINC service, but when I'm not at home, there's no easy way to tell if HTTP errors start piling up. What's more, restarting the BOINC service is only fighting symptoms, not tackling the real problem (why transient HTTP errors start piling up) that I'm trying to solve.
[Sep 3, 2016 1:20:55 PM]   Link   Report threatening or abusive post: please login first  Go to top 
SekeRob
Master Cruncher
Joined: Jan 7, 2013
Post Count: 2741
Status: Offline
Reply to this Post  Reply with Quote 
Re: Transient HTTP Errors, the continuing story...
Create a crontab job at an interval of your buffer depth, e.g. 2 days, to do

sudo service boinc-client restart

or whatever script call actions this (above applies to my ubuntu/gianfranco build).

Edit: Not too often as each boot loses some progress on running jobs falling back at resume to last checkpoint.
----------------------------------------
[Edit 1 times, last edit by SekeRob* at Sep 3, 2016 1:34:26 PM]
[Sep 3, 2016 1:32:51 PM]   Link   Report threatening or abusive post: please login first  Go to top 
dango
Senior Cruncher
Joined: Jul 27, 2009
Post Count: 307
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Transient HTTP Errors, the continuing story...
i am getting that errors only on 1 VM, but yeah, better is no error :)
[Sep 3, 2016 1:45:41 PM]   Link   Report threatening or abusive post: please login first  Go to top 
[ Jump to Last Post ]
Post new Thread