Occasionally the whole or portions of the BOINC server daemon functions are suspended by the WCG techs or the system itself to perform maintenance on the filing system [GPFS SAN storage infrastructure] running for instance a fsck file check that can take a long time. The message (an error 503) could also be because the automated monitoring system thinks the momentary server load is too high at which time the least essential parts are temporarily put on hold. This will generate a set of messages on the connecting clients such as the below where task downloading and Ready to Report clearing continue to execute normally:

3601 WCG 15-6-2010 7:51:10 Started upload of CMD2_0568-CLIC5A.clustersOccur-2P3W_A.clustersOccur_2_54113_58960_0_0
3602 WCG 15-6-2010 7:51:11 [error] Error reported by file upload server: Maintenance underway: file uploads are temporarily disabled.
3603 WCG 15-6-2010 7:51:11 Temporarily failed upload of CMD2_0568-CLIC5A.clustersOccur-2P3W_A.clustersOccur_2_54113_58960_0_0: transient upload error
3604 WCG 15-6-2010 7:51:11 Backing off 1 min 0 sec on upload of CMD2_0568-CLIC5A.clustersOccur-2P3W_A.clustersOccur_2_54113_58960_0_0
3605 WCG 15-6-2010 7:51:11 Finished download of 29b4fd3674d32aebe54bbd2c03019077.dat.gzb
3606 WCG 15-6-2010 7:51:11 Finished download of 2138672f19447d7d22d6ace41a2530b5.pdb.gzb
3607 WCG 15-6-2010 7:51:11 Finished download of b37e58277d4024573131822bd2425042.pdb.gzb
3608 WCG 15-6-2010 7:51:11 Started download of 171a9579faa2141bbf8b84e06e539f59.dat.gzb
3609 WCG 15-6-2010 7:51:14 Finished download of 171a9579faa2141bbf8b84e06e539f59.dat.gzb
3610 WCG 15-6-2010 7:52:11 Started upload of CMD2_0568-CLIC5A.clustersOccur-2P3W_A.clustersOccur_2_54113_58960_0_0
3611 WCG 15-6-2010 7:52:21 Finished upload of CMD2_0568-CLIC5A.clustersOccur-2P3W_A.clustersOccur_2_54113_58960_0_0
3612 WCG 15-6-2010 7:52:23 [sched_op_debug] Starting scheduler request
3613 WCG 15-6-2010 7:52:23 Sending scheduler request: To report completed tasks.
3614 WCG 15-6-2010 7:52:23 Reporting 1 completed tasks, not requesting new tasks
3615 WCG 15-6-2010 7:52:23 [sched_op_debug] CPU work request: 0.00 seconds; 0 idle CPUs
3616 WCG 15-6-2010 7:52:28 Scheduler request completed

When the client receives such an instruction it is backing off by an amount of time that will increase as it repeats. This in such a way that when the situation normalizes, not all clients that need to upload result files will NOT do this simultaneous causing other bottleneck issues.

This is an automated process which does NOT require member intervention and is best left alone. If uploads are attempted too often using the Retry Now or Update functions, the client might conclude that the problem is permanent and cancel the result(s).

Please let the process run it's course and read the active support forums should the situation persist for hours.

Related topics:

Server Structure Information


Return to the Start Here FAQ index