Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 38
Posts: 38   Pages: 4   [ Previous Page | 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 5881 times and has 37 replies Next Thread
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

Hi again.

It's 3:40 (AM), .... Sooo funny.
BOINC still reports disk usage as less than 40 MB (with WCG and SETI@Home together).

Hi again hoanqui,

with respect to the 40mb you see used in those pie charts of BOINC, those are the sizes of the project subdirectories where the jobs to do / not completed are stored. BOINC does not report usage for:

1. The portions used by the tasks from the Swapfile/Virtual Memory. It only reports this at start up of what's available.
2. The Temporary store used by the files after each checkpoint.
3. AFAIK, the portions used in the progress slot subdirectories (anyone knows about this.... slot0 slot1 etc?)

If you monitor the process manager and compare to the Table provided, you are observing at time 300-450mb Swapfile use. With HPF2 the actual needs are about 200mb swapfile, PER JOB, so the wall can be hit if you have concurrent jobs going. If you've set the value % of what's allowed to be used from RAM, VM or storage, you will hit the wall.

There are some internal values in a job e.g. FAAH has itself limited to 500mb (last i heard). If it hits that limit it will hang and report this in the message log.

As yet i do not discover which exact BOINC version you run (assume it to be 5.10 considering the Local pref ref.), but the 5.10 parameter management is, to put it mildly 'up dung creek" depending on the point release used. Not even what you see in local prefs thru a remote connection can be trusted to be correct at this time of writing thru 5.10.18. WCG has refrained from declaring any 5.10.x version recommended and continues with 5.8 offering on it's download page. WCG also asked Berkeley to hold back on sending out new release recommendations. Both 5.10.7 and 5.10.13 are premature.... they work for many, but not in sufficient hi numbers for others due all it's parm complications and bugs.

So let me tell you what works for me on a C2D

a. Use at most 5gb disk storage (it never will unless you get into HDC jobs on a quad where sizes can be 1gb+ per job)
b. Leave at least 1gb free on disk (i got plenty more though, but swapfiles can grow)
c. Use at most 75% of total disk space. (can get finicky if real space is tight)
d. Use at most 75% of swap file (allow VM to freely expand in the OS settings itself)

I've not had but 1 issue with any present project, but the strange none % progress on HPF2, which in 99% of the cases on a suspend/release of the WCG project (in the project tab) makes it hop across the barrier it sat looping before. If there are no messages on HPF2, then that is a possibility of happening.

don't get upset.... we'll get thru it for a good cause, but accept that 5.10 is not for the greater crunching populous yet.

cheers
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
----------------------------------------
[Edit 1 times, last edit by Sekerob at Aug 15, 2007 8:56:16 AM]
[Aug 15, 2007 8:52:14 AM]   Link   Report threatening or abusive post: please login first  Go to top 
hoanqui
Cruncher
Joined: Aug 14, 2007
Post Count: 15
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

Lawrence,

I mention HD usage because Sekerob's response about the problem was to check the minimum requirements of the projects, specifically the HD limit I had set in my prefs (200 MB). So I upped the limit to 2 GB. (as you see, it didn't help)

I'm using the latest recommended BOINC version for OS X (5.10.7). As I said in my first post, I'm using local prefs (defined in the Simple View of BOINCManager), and the limits are right now 2 GB HD, 30% CPU, don't work when on battery, work after 1 minute idle.
[Aug 15, 2007 10:11:41 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

Suggest you switch to Advance View and the Advance Menu, Local Prefs for greater detail settings and following my recommendation in my post earlier. The Simple view I think only controls the storage and not things like the VM settings.

ttyl
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Aug 15, 2007 10:30:22 AM]   Link   Report threatening or abusive post: please login first  Go to top 
hoanqui
Cruncher
Joined: Aug 14, 2007
Post Count: 15
Status: Offline
Reply to this Post  Reply with Quote 
sad Re: HPF2 CPU throtteling + reset problems


with respect to the 40mb you see used in those pie charts of BOINC, those are the sizes of the project subdirectories where the jobs to do / not completed are stored. BOINC does not report usage for:

1. The portions used by the tasks from the Swapfile/Virtual Memory. It only reports this at start up of what's available.
2. The Temporary store used by the files after each checkpoint.
3. AFAIK, the portions used in the progress slot subdirectories (anyone knows about this.... slot0 slot1 etc?)


Anyway, HD free space has never decreased noticeably. When the problem reappeared last night, I had more than 2.7 GB free, and they are still free now.
So I can't find a link with the problem.


If you monitor the process manager and compare to the Table provided, you are observing at time 300-450mb Swapfile use. With HPF2 the actual needs are about 200mb swapfile, PER JOB, so the wall can be hit if you have concurrent jobs going.


I think having a preferences-set limit of 2 GB, and 2.7 GB free on the HD, discards that possiblity.


If you've set the value % of what's allowed to be used from RAM, VM or storage, you will hit the wall.


Just for completeness, here I recap once again my prefs, local and otherwise:
Local: 2 GB, 30% CPU, 1 minute idle, no working while on battery
Web (are they used? who knows? not the normal user, that's for sure): no more than 50% CPU, no more than 75% of total mem, 3 mins idle, at most 1 CPU, switch apps every 120 mins, no more than .3 GB HD, at least .5 GB free HD, no more than 75% HD, write to disk at most every 99, stay in memory, connect to network every 3 days.



There are some internal values in a job e.g. FAAH has itself limited to 500mb (last i heard). If it hits that limit it will hang and report this in the message log.


The log never reported more than the oh-so-useful "zero status, no file, may need to reset".



As yet i do not discover which exact BOINC version you run (assume it to be 5.10 considering the Local pref ref.),


5.10.7 (latest recommended for OS X)

Not even what you see in local prefs thru a remote connection can be trusted to be correct at this time of writing thru 5.10.18. WCG has refrained from declaring any 5.10.x version recommended and continues with 5.8 offering on it's download page.


Interesting. The problem is, I already was using BOINC when I found WCG months ago, so I didn't even realize that they had a different BOINC recommendation.

The other (and I feel much bigger) part of the problem, of course, is the lack of any sanity checking - or at least, the lack of any reporting of the results of whatever sanity checking. "Hey, user, we'll try to run but you should be aware that this BOINC release is not an officially supported one by WCG; if something goes wrong, you should yadda yadda yadda".
EVEN IF I had gotten into BOINC thru WCG, I would have surely updated my BOINC installation as soon as I had discovered that a new version was available. Which is what normal users do, and in fact is what they are usually told to do. So sanity checking AND REPORTING gets mandatory - and its lack, crippling. Which is the case.


WCG also asked Berkeley to hold back on sending out new release recommendations. Both 5.10.7 and 5.10.13 are premature.... they work for many, but not in sufficient hi numbers for others due all it's parm complications and bugs.


Shame on them. And shame on WCG for not letting me know without me having to go out of my way.


So let me tell you what works for me on a C2D

a. Use at most 5gb disk storage (it never will unless you get into HDC jobs on a quad where sizes can be 1gb+ per job)


As said, 2.7 GB free, 2 GB prefs-limited. So: check.


b. Leave at least 1gb free on disk (i got plenty more though, but swapfiles can grow)


Check.


c. Use at most 75% of total disk space. (can get finicky if real space is tight)


Check.


d. Use at most 75% of swap file (allow VM to freely expand in the OS settings itself)


There is no VM setting in OS X that I can think of, and anyway, I set 75% of memory in the web's prefs page. So, check.


I've not had but 1 issue with any present project, but the strange none % progress on HPF2, which in 99% of the cases on a suspend/release of the WCG project (in the project tab) makes it hop across the barrier it sat looping before. If there are no messages on HPF2, then that is a possibility of happening.


I have tried that, and anyway I guess that the effects of that are a subset of what a project reset does, which I did four times yesterday. So... check, and Not Working For Me.


don't get upset.... we'll get thru it for a good cause, but accept that 5.10 is not for the greater crunching populous yet.


I do feel that this problem asks for some upsetting :P. Someone, somewhere, is not doing his/her homework.

Just for the record: I'm keeping WCG suspended on my computer, as nothing new has been suggested that could improve the situation, and WCG-induced CPU maxing-out has managed to blaze a direct trail for getting into my nerves :P.
(and I'm not downgrading BOINC, since anyway good old SETI@Home is happily crunching away).
[Aug 15, 2007 11:02:02 AM]   Link   Report threatening or abusive post: please login first  Go to top 
hoanqui
Cruncher
Joined: Aug 14, 2007
Post Count: 15
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

Suggest you switch to Advance View and the Advance Menu, Local Prefs for greater detail settings and following my recommendation in my post earlier. The Simple view I think only controls the storage and not things like the VM settings.

ttyl


Nice. The last I knew was that local prefs were only available through the Simple View - that was the only reason I used to switch out of the Advanced View.

So I recap the prefs, as seen from Advanced View:
CPU:Do work after 1 min idle, not on batts, not while in use; switch apps every 60 mins, at most 16 processors, at most 30% CPU
Network:Connect about ever .1 days, additional work buffer .25 days; everything else off
Disk: use at most 2 GB, leave free at least .1 GB, at most 50% total disk space, write to disk at most every 60 secs, at most 75% swap file
Memory: at most 50% when in use, at most 90% when idle, do not leave in memory

Those are not the prefs I would most like to be in effect (the network values are downright weird), but anyway they still don't seem to explain anything.
[Aug 15, 2007 11:16:41 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

From the early days of 5.10 CA's (Community Advisors) have been vocal on WCG not recommending 5.10. The WCG download page now explicitly states the version it offers:

We can't win them all, even if going out of our way to help users going. Someone with advanced MAC experience will be able to tell how to address the VM side. Here's a oogled post of someone even putting his swapfile on a dedicated partition and commenting on a 80mb default: http://www.xlr8yourmac.com/OSX/OS_x_swapfile_sizes.html
http://text.dslreports.com/faq/allthingsmac/6_OSX

strange thing is, i think you are the first to report issues that are linked to disk space on a Mac (I'm not sure now if it is the problem, but we can only work from eliminating possibilities). That OS with it's strong GUI is probably less needing looking under the hood.
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Aug 15, 2007 11:32:48 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

hoanqui, 5.10.7 has numerous preference related bugs. In particular, the Advanced View DOES NOT SHOW THE CORRECT PREFERENCES. Worse, if you then click "Okay", it saves these incorrect preferences, overwriting anything you set elsewhere. And that's why 5.10 isn't ready for release yet.

Please don't reset WCG any more. Clearly that isn't fixing your problem, so there's no point. Instead, please will you try just letting the work unit continue.

You should be aware that there are periods when the BOINC throttle is not applied - in particular during checkpointing. The throttle works by suspending and resuming the application very frequently, and the application can't be suspended during checkpointing. If checkpointing is taking a very long time, then that may need looking into.
[Aug 15, 2007 12:16:52 PM]   Link   Report threatening or abusive post: please login first  Go to top 
hoanqui
Cruncher
Joined: Aug 14, 2007
Post Count: 15
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

From the early days of 5.10 CA's (Community Advisors) have been vocal on WCG not recommending 5.10. The WCG download page now explicitly states the version it offers:


I see that WCG, in its downloads page, offers a 5.8.x version.
I can't see anywhere else (except in these forums, that is) any remote hint that the user SHOULD NOT update to the latest officially recommended BOINC version, as stated in the official BOINC homepage.
How exactly am I supposed to know that?
This should be made EXPLICIT, in-your-face, multiple times and in many ways: in the WCG info web pages, the WCG prefs web pages, and the BOINC logs.


We can't win them all, even if going out of our way to help users going. Someone with advanced MAC experience will be able to tell how to address the VM side. Here's a oogled post of someone even putting his swapfile on a dedicated partition and commenting on a 80mb default: http://www.xlr8yourmac.com/OSX/OS_x_swapfile_sizes.html
http://text.dslreports.com/faq/allthingsmac/6_OSX


Oh, please.
Just for the record, in case some Mac OS X user comes here and sees these posts:
THERE IS NO REASON to touch your VM, except in VERY specific cases. In the earliest versions of OS X, it could SOMETIMES make sense. But it's been a long time (OS X 10.2 era) since I last saw anyone talking about swap partitions and the like for OS X.
By the way, the first page you link to confirm that: they refer to OS X 10.1 (circa 2001).
(the second link doesn't even mention virtual memory)

In case you want some more context, my OS X with it standard VM settings (standard because there is no official way to change them) is using right now a 3GB swap file, of which more than 2.5 GB are reported as in use by the OS. OS X grows and shrinks its swap files as needed; it starts with few MB at boot and I've seen it grow past 5 GB and back under 2GB as the OS sees fit.
For completeness sake again, I have 3 GB RAM, in a Macbook Core2Duo. I have right now 2.7 GB of free HD space. When freshly booted, the HD free space is about 6 GB; the VM grows in that free space.


strange thing is, i think you are the first to report issues that are linked to disk space on a Mac (I'm not sure now if it is the problem, but we can only work from eliminating possibilities).


That's it, we indeed can't be sure if that is the problem.

And since there is absolutely no hint about what could exactly be the problem, any user with remotely similar symptoms could be having it.

And as I said, I have been suffering this problem for weeks before coming here to look for some solution. Other users could just be suffering the CPU peaks and don't
realize, or don't bother to look for the reason, or don't know how to! Because, as said, there is no hint, even in BOINC's logs, that HPF2 is causing CPU extreme usage. Only that "it exited with no files". Useful!
[Aug 15, 2007 12:52:16 PM]   Link   Report threatening or abusive post: please login first  Go to top 
hoanqui
Cruncher
Joined: Aug 14, 2007
Post Count: 15
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

hoanqui, 5.10.7 has numerous preference related bugs. In particular, the Advanced View DOES NOT SHOW THE CORRECT PREFERENCES. Worse, if you then click "Okay", it saves these incorrect preferences, overwriting anything you set elsewhere. And that's why 5.10 isn't ready for release yet.


OK, just in case I'm deactivating those local preferences and going back to the webpage ones.


Please don't reset WCG any more. Clearly that isn't fixing your problem, so there's no point. Instead, please will you try just letting the work unit continue.


Yes, I noticed that.
But again, I was just following the terribly limited information I had about the problem: "the project may need to be reset". Yeah.

About "letting the work unit continue": that means spurious CPU usage peaks that last minutes. That means impeded computer usage, and fan blowing hard for minutes. That means the noise waking me up in the night because of some problem that I have no information to solve.
So that means WCG / HPF2 (haven't decided yet) are off my computer.


You should be aware that there are periods when the BOINC throttle is not applied - in particular during checkpointing. The throttle works by suspending and resuming the application very frequently, and the application can't be suspended during checkpointing. If checkpointing is taking a very long time, then that may need looking into.


I can report lots of instances of >1 minute periods of 100% CPU usage, which seemingly always end up with that "no file, may need reset" log message.
The happening of those peaks seems random.
[Aug 15, 2007 1:08:32 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: HPF2 CPU throtteling + reset problems

The BOINC CPU throttle of 30% means it runs 3 seconds at 100% and pauses for 7 seconds.... on all projects, both 5.8 and 5.10. At some point the throttling itself caused problems in the science, with the only way avoiding offered was to set it at 100%.

Like Didactylos already mentioned, during checkpointing the throttle is ignored as disk i/o takes priority.

As for the original message:
Task lf432_00028_11 exited with zero status but no 'finished' file
If this happens repeatedly you may need to reset the project.

One known cause is the system clock being synchronized by the system. If that happens middle of the science, particularly backward, BOINC considers a timekeeping problem and tries to return a project to the last checkpoint. Just wonder if the throttling alternation 3:7 and the system time keeping might have something to do with this. Very few i've heard of set it to 30%, considering that the processes use pure idsle time and are at lowest priority i.e. even at 100% should hardly impair use. Full tilt is known to cause the fan to run all the time, but another Mac user reported that 90% was enough for that not to happen.
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
----------------------------------------
[Edit 2 times, last edit by Sekerob at Aug 15, 2007 1:25:21 PM]
[Aug 15, 2007 1:12:51 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 38   Pages: 4   [ Previous Page | 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread