World Community Grid - View Thread - Getting "finish file present too long" error for every WU when HDD is busy

World Community Grid Forums

Category: Completed Research

Forum: Microbiome Immunity Project

Thread: Getting "finish file present too long" error for every WU when HDD is busy

Quick Go »

No member browsing this thread

Thread Status: Active
Total posts in this thread: 6

[ ]

Author

This topic has been viewed 2218 times and has 5 replies

trivium
Cruncher
Joined: May 25, 2011
Post Count: 3
Status: Offline
Project Badges:

1 year badge for Human Proteome Folding - Phase 2

180 day badge for Help Fight Childhood Cancer

180 day badge for Help Cure Muscular Dystrophy - Phase 2

180 day badge for The Clean Energy Project - Phase 2

1 year badge for Computing for Clean Water

1 year badge for Drug Search for Leishmaniasis

1 year badge for GO Fight Against Malaria

50 year badge for Mapping Cancer Markers

14 day badge for Uncovering Genome Mysteries

2 year badge for Outsmart Ebola Together

2 year badge for FightAIDS@Home - Phase 2

10 year badge for Microbiome Immunity Project

90 day badge for Africa Rainfall Project

5 year badge for OpenPandemics - COVID-19


Getting "finish file present too long" error for every WU when HDD is busy

Hello everyone, I want to report about problem with MIP workunits.
I have my personal home server (based on DAppNode project), and have a full Etherum blockchain on the same disk with BOINC.

High disk usage prevents BOINC to finish the work and send results. Each WU runs several hours, but then finishes with an error. There is a huge waste of computing time. What can I do to help BOINC process to finish the jobs?
Is there any config values that I can change to solve this (excluding process priority)?

List of failed jobs: https://www.worldcommunitygrid.org/ms/viewBoi...By=sentTime&pageNum=1

Hardware:
- Double Intel(R) Xeon(R) CPU L5630 @ 2.13GHz
- 12 GB RAM (ECC)

[Dec 31, 2019 12:31:35 AM]

Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7846
Status: Offline
Project Badges:

2 year badge for Human Proteome Folding - Phase 2

14 day badge for Help Cure Muscular Dystrophy

2 year badge for Discovering Dengue Drugs - Together

2 year badge for Nutritious Rice for the World

14 day badge for The Clean Energy Project

10 year badge for Help Fight Childhood Cancer

90 day badge for Influenza Antiviral Drug Search

2 year badge for Help Cure Muscular Dystrophy - Phase 2

45 day badge for Discovering Dengue Drugs - Together - Phase 2

2 year badge for The Clean Energy Project - Phase 2

2 year badge for Computing for Clean Water

5 year badge for Drug Search for Leishmaniasis

5 year badge for GO Fight Against Malaria

2 year badge for Computing for Sustainable Water

200 year badge for Mapping Cancer Markers

5 year badge for Uncovering Genome Mysteries

20 year badge for Outsmart Ebola Together

10 year badge for FightAIDS@Home - Phase 2

100 year badge for Smash Childhood Cancer

2 year badge for Africa Rainfall Project

100 year badge for OpenPandemics - COVID-19


Re: Getting "finish file present too long" error for every WU when HDD is busy

First off, only you can see the information in the link you provided.
Secondly, I have a suggestion on the busy harddrive problem. You could install a second drive and have all of your BOINC files and work on a second drive. I am guessing that the blockchain work is monopolizing pretty much of all your existing drive I/O. A second drive would only have to be a small cheap drive if BOINC is the only thing on it.
Cheers

----------------------------------------

Sgt. Joe
*Minnesota Crunchers*

[Dec 31, 2019 2:18:19 AM]

trivium
Cruncher
Joined: May 25, 2011
Post Count: 3
Status: Offline
Project Badges:


Re: Getting "finish file present too long" error for every WU when HDD is busy

Thank you for clarification, I am posted my error log below.

I am noticed this problem more then a year ago, when I am installed BOINC at my personal web server. Several work units failed with this error every day. This was happened daily at 00:00, because my server's cron started at in this time (backups, log archiving, etc).

I think this problem relates to all servers running BOINC and having a HDD-intensive tasks from time to time.


<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
[2019-12-30 23:16:12:] :: BOINC:: Initializing ... ok.
[2019-12-30 23:16:12:] :: BOINC :: boinc_init()
INFO: result number = 0
BOINC:: Setting up shared resources ... ok.
BOINC:: Setting up semaphores ... ok.
BOINC:: Updating status ... ok.
BOINC:: Registering timer callback... ok.
BOINC:: Worker initialized successfully.
command: ../../projects/www.worldcommunitygrid.org/wcgrid_mip1_rosetta_7.16_x86_64-pc-linux-gnu -in::file::zip MIP1_databasev2.zip @./MIP1_00261809.flags -out::file::silent result_silent.out -run:jran 1498175442 -nstruct 8 -out::level 100 -run::no_scorefile true
Registering options..
Registered extra options.
Initializing broker options ...
Registered extra options.
Initializing core...
Initializing options.... ok
Options::initialize()
Options::adding_options()
Options::initialize() Check specs.
Options::initialize() End reached
Loaded options.... ok
Processed options.... ok
Initializing random generators... ok
Initialization complete.
Setting WU description ...
Unpacking zip data: ../../projects/www.worldcommunitygrid.org/mip1.MIP1_databasev2.zip
Setting database description ...
Setting up checkpointing ...
Setting up graphics native ...
set_shared_memory_fully_initialized ...
abrelax ...
abrelax.run
Setting up folding (abrelax) ...
Beginning folding (abrelax) ...
BOINC:: Worker startup.
Sequence Length = 147
Starting work on structure: _0001
Finished _0001 in 2118.58 seconds.
Starting work on structure: _0002
Finished _0002 in 2388.87 seconds.
Starting work on structure: _0003
Finished _0003 in 2166.31 seconds.
Starting work on structure: _0004
Finished _0004 in 1293.51 seconds.
Starting work on structure: _0005
[2019-12-31 1:38:13:] :: BOINC:: Initializing ... ok.
[2019-12-31 1:38:13:] :: BOINC :: boinc_init()
INFO: result number = 0
BOINC:: Setting up shared resources ... ok.
BOINC:: Setting up semaphores ... ok.
BOINC:: Updating status ... ok.
BOINC:: Registering timer callback... ok.
BOINC:: Worker initialized successfully.
command: ../../projects/www.worldcommunitygrid.org/wcgrid_mip1_rosetta_7.16_x86_64-pc-linux-gnu -in::file::zip MIP1_databasev2.zip @./MIP1_00261809.flags -out::file::silent result_silent.out -run:jran 1498175442 -nstruct 8 -out::level 100 -run::no_scorefile true
Registering options..
Registered extra options.
Initializing broker options ...
Registered extra options.
Initializing core...
Initializing options.... ok
Options::initialize()
Options::adding_options()
Options::initialize() Check specs.
Options::initialize() End reached
Loaded options.... ok
Processed options.... ok
Initializing random generators... ok
Initialization complete.
Setting WU description ...
Unpacking zip data: ../../projects/www.worldcommunitygrid.org/mip1.MIP1_databasev2.zip
Setting database description ...
Setting up checkpointing ...
Setting up graphics native ...
set_shared_memory_fully_initialized ...
abrelax ...
abrelax.run
Setting up folding (abrelax) ...
Beginning folding (abrelax) ...
BOINC:: Worker startup.
Starting work on structure: _0005
[2019-12-31 1:44:50:] :: BOINC:: Initializing ... ok.
[2019-12-31 1:44:50:] :: BOINC :: boinc_init()
INFO: result number = 0
BOINC:: Setting up shared resources ... ok.
BOINC:: Setting up semaphores ... ok.
BOINC:: Updating status ... ok.
BOINC:: Registering timer callback... ok.
BOINC:: Worker initialized successfully.
command: ../../projects/www.worldcommunitygrid.org/wcgrid_mip1_rosetta_7.16_x86_64-pc-linux-gnu -in::file::zip MIP1_databasev2.zip @./MIP1_00261809.flags -out::file::silent result_silent.out -run:jran 1498175442 -nstruct 8 -out::level 100 -run::no_scorefile true
Registering options..
Registered extra options.
Initializing broker options ...
Registered extra options.
Initializing core...
Initializing options.... ok
Options::initialize()
Options::adding_options()
Options::initialize() Check specs.
Options::initialize() End reached
Loaded options.... ok
Processed options.... ok
Initializing random generators... ok
Initialization complete.
Setting WU description ...
Unpacking zip data: ../../projects/www.worldcommunitygrid.org/mip1.MIP1_databasev2.zip
Setting database description ...
Setting up checkpointing ...
Setting up graphics native ...
set_shared_memory_fully_initialized ...
abrelax ...
abrelax.run
Setting up folding (abrelax) ...
Beginning folding (abrelax) ...
BOINC:: Worker startup.
Starting work on structure: _0005
Finished _0005 in 1921.43 seconds.
Starting work on structure: _0006
Finished _0006 in 1488.72 seconds.
Starting work on structure: _0007
Finished _0007 in 1356.27 seconds.
Starting work on structure: _0008
Finished _0008 in 2178.2 seconds.
======================================================
DONE :: 8 structures in 14938.2 cpu seconds
======================================================
BOINC :: BOINC support services shutting down cleanly ...
03:42:19 (17248): called boinc_finish(0)

</stderr_txt>
]]>

Also, when I searched for a solution I am noticed same problems in other BOINC projects (like Rosetta, Milky Way and so on). Maybe this problem relates to BOINC core and I should create an issue in official BOINC repository?

What happens when work unit finishes with "finish file present too long" message? How long is "too long" - is it a BOINC platform limit or a work unit limit?

[Jan 2, 2020 2:05:22 PM]

Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7846
Status: Offline
Project Badges:


Re: Getting "finish file present too long" error for every WU when HDD is busy

You have provided a clue when you say this happens when you start a cron job doing logs, backup, etc. You could specify BOINC only to run say from 01:00 to 23:00. then your cron job would have the disk to itself and it would noy bother BOINC. An inelegant fix, but it might work for you.
Cheers

----------------------------------------

Sgt. Joe
*Minnesota Crunchers*

[Jan 2, 2020 2:39:47 PM]

TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 286
Status: Offline
Project Badges:

90 day badge for Nutritious Rice for the World

2 year badge for Help Fight Childhood Cancer

180 day badge for Discovering Dengue Drugs - Together - Phase 2

5 year badge for The Clean Energy Project - Phase 2

2 year badge for Drug Search for Leishmaniasis

10 year badge for Uncovering Genome Mysteries

10 year badge for Outsmart Ebola Together

20 year badge for FightAIDS@Home - Phase 2

20 year badge for Smash Childhood Cancer

20 year badge for Microbiome Immunity Project

20 year badge for Africa Rainfall Project

50 year badge for OpenPandemics - COVID-19


Re: Getting "finish file present too long" error for every WU when HDD is busy

These cron jobs (backups, log archiving, etc) - do all all run together or are they serialized so one finishes before the next starts. Can you reduce the priority of these cron jobs. Will take longer to run but have less impact on 'nornal' work. On the machines here routine house-keeping is spread throughout the whole night. The "nice" utilities from the coreutils package is used to reduce priority for disk intense house-keeping tasks.

----------------------------------------

Run Time Stats https://grassmere-productions.no-ip.biz/

[Jan 2, 2020 4:52:12 PM]

trivium
Cruncher
Joined: May 25, 2011
Post Count: 3
Status: Offline
Project Badges:


Re: Getting "finish file present too long" error for every WU when HDD is busy

Ok, if increasing a workunit finalization timeout is not an option, and BOINC doesn't have a config variable for it - I'll try to pause BOINC when system or HDD is busy. Thanks.

[Jan 4, 2020 5:40:45 AM]

[ ]