Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 13
Posts: 13   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 2893 times and has 12 replies Next Thread
Mgruben
Advanced Cruncher
Joined: May 26, 2013
Post Count: 94
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
CEP2 Work Unit Slot Size (in GB)

Hey all,

Trying to figure out how much space a CEP2 WU can possibly take up while it is crunching, since I've personally seen one go from 400MB to 850MB in a matter of minutes (then shoot back down), and I've also seen one crunching away at 1.1GB.

It's relevant for me since I'm hoping to run these units on (read: I've mounted my /slots directory to) a RAMdisk, but I need some worst-case scenario figures so I avoid running out of space and crashing/aborting the WU.

Tl;dr: do CEP2 WUs get bigger than 1.1GB when crunching, or is that their max size (final form, for DBZ fans)?

Edit: spelling better, one edit at a time.
----------------------------------------

----------------------------------------
[Edit 1 times, last edit by Mgruben at Aug 20, 2013 8:41:11 PM]
[Aug 20, 2013 8:40:19 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Paul Schlaffer
Senior Cruncher
USA
Joined: Jun 12, 2005
Post Count: 279
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)

The system requirements are 2gb per work unit.

In my case I bought a used 50nm Intel X25-E slc SSD on line. It's much cheaper than ram, and is handling 16 simultaneous work units. With a 2000 terabyte endurance rating it will be good for more than 8 years.
----------------------------------------
“Where an excess of power prevails, property of no sort is duly respected. No man is safe in his opinions, his person, his faculties, or his possessions.” – James Madison (1792)
[Aug 21, 2013 2:59:26 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)

Here are the official system requirements - 1 GB RAM for CEP2 and 2 GB disk : https://secure.worldcommunitygrid.org/help/viewTopic.do?shortName=minimumreq

Some important background information : The researchers run large molecules that use up to 8 GB RAM on their own linux grid at the lab. They are picking small molecules to send to us that can run on 32-bit compiled code. An improved version of the code is being developed. The current talk is that we will continue to have a 32-bit small version of the improved program but will also have the option of running a 64-bit version for large (8 GB) jobs.

So what does our current code require? I am assuming that molecule picking is not an exact science so some assigned molecules probably exceed the limits. I have no idea how the code enforces limits. If I were doing the coding, I would simply set a 1 GB limit on the largest array and return an error as soon as a molecule tried to exceed it. This would be a loose interpretation of a 1 GB limit since the RAM would include smaller arrays, program code, etc. So a max usage of 1.1 GB would be optimistic but 2 GB seems unreasonably large. If testing showed more than 1.25 GB, I would reduce the max allowed array size.

I do not know how the actual programmers implemented the limit, but if they are like me the limit is probably 1GB plus a bit. After all, virtual memory is supposed to make life easier for programmers by allowing us some slop compared to the really tight coding necessary in the 1960's.

Lawrence
----------------------------------------
[Edit 1 times, last edit by Former Member at Aug 21, 2013 5:17:48 AM]
[Aug 21, 2013 5:14:21 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Eric_Kaiser
Veteran Cruncher
Germany (Hessen)
Joined: May 7, 2013
Post Count: 1047
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)

I appreciate a 64-bit version for large jobs. Please improve checkpointing too.
----------------------------------------

[Aug 21, 2013 10:51:52 AM]   Link   Report threatening or abusive post: please login first  Go to top 
CandymanWCG
Senior Cruncher
Romania
Joined: Dec 20, 2010
Post Count: 421
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)

hi Mcgruber,

Here are my 2 cents on the matter: I have also successfully been running CEP2 on a standard (free) RAMDisk partition of just 4GB, with 2 concurrent tasks plus 2 WUs from the other projects. Never ran into trouble. From time to time I would even let S@H and S@H Beta run wild with a WU (a 5th) and still no issues.

I hope this helps.

Cheers!
----------------------------------------
Knowledge is limited. Imagination encircles the world! - Albert Einstein



[Aug 21, 2013 10:58:43 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Mgruben
Advanced Cruncher
Joined: May 26, 2013
Post Count: 94
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)

All great information everyone, thanks for the replies.

I did notice the 2048MB Disk Space and 1024MB RAM space per CEP2 WU under system requirements; I'm fine with allowing (1024x4)MB RAM for 4 concurrent CEP2 WUs, though I had sort of assumed that at least some of the 2048MB Disk Space per WU resided outside of the /slots directory while crunching, and hence I would only need somewhat less than (2048x4)MB RAMdisk for 4 concurrent CEP2 WUs.

I guess if I want to cut it close and only allocate an 8GB RAMdisk to hold 4 concurrent CEP2 WUs, that's my call haha, but I've been warned.

<aside>
I bought 4GB ram when I first built my crunching rig, before I had aspirations of RAMdisking.
After I discovered the ability to RAMdisk, I quicky found out that I'd need more RAM sticks if I wanted to run multiple CEP2s, but could only find my RAM flavor in 8GB units.
In sum, as it happens I'll only have 12GB RAM unless I want to replace the 3-week-old 4GB RAM stick with an 8GB unit, so it looks like I'll have the bare minimum needed to RAMdisk 4 concurrent CEP2 WUs (4GB RAM, 8GB RAMdisk).
I'm sort of ignoring system overhead and other things that might be going on in the system that would tip me over the crucial RAM usage figures, so my plot for 4 CEP2 WUs may be ultimately ill-fated, but we'll see how things work out.
</aside>

Thanks again all
----------------------------------------

[Aug 23, 2013 12:26:24 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Mgruben
Advanced Cruncher
Joined: May 26, 2013
Post Count: 94
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)


I have also successfully been running CEP2 on a standard (free) RAMDisk partition of just 4GB, with 2 concurrent tasks plus 2 WUs from the other projects. Never ran into trouble.


Are there any other WUs which use more than a few MBs of disk space while crunching? Every time I check the slots folder size of FAAH or FAAHv or SN2S I usually see maybe 10MB or something, and so I usually ignore these as negligible.

Reading your statement, I assume you're tapping 2.2GB or so with the (CEP2)x2 in a worst-case scenario, to which the other WUs add maybe 0.02GB. Given that I don't know exactly the working size max of the CEP2 WUs though, it could also be the case that your CEP2 WUs peg 3.9GB, for a total of maybe 3.95GB conservatively (this seems way too high, though logically still possible).

I suppose this could always change in the future though.
----------------------------------------

[Aug 23, 2013 12:32:47 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)

My plot for 4 CEP2 WUs may be ultimately ill-fated, but we'll see how things work out.


I think you'll be fine. Most of the CEP processes I run seem to hover around 200-300MB.. they may spike here and there but I'd think that the likelihood of exceeding max RAM would be pretty low. However, if you've configured BOINC to consume max 50% while in use (or idle for that matter), you might run into some WU stoppage:) I might be wrong, but I don't think the min. requirements listed actually take those configuration values into consideration.
[Aug 23, 2013 12:37:04 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Mgruben
Advanced Cruncher
Joined: May 26, 2013
Post Count: 94
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)

Update:

I configured a script to record my RAMdisk's usage (my /slots folder resides on my RAMdisk) every 15 minutes for 10 hours while one CEP2 (and 3 other different WCG Projects) WU was running.

The highest I got (unfortunately the WU lasted 11:40, while I was only measure for 10:00 of that) was 1.3GB used. Assuming that this usage mostly if not all attributable to the single CEP2 WU, it looks like 1.3GB may be the functional max, so that 1.5GB is a conservative CEP2 WU estimate while crunching.

That said, I'm extrapolating from 40 polling points over a single WU without controlling for the other WCG projects running on my box, so while I'll be running with the 1.5GB number, I caution others against doing so without checking /slot size for yourself.
----------------------------------------

----------------------------------------
[Edit 1 times, last edit by Mgruben at Aug 23, 2013 12:52:20 PM]
[Aug 23, 2013 12:51:50 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Jim1348
Veteran Cruncher
USA
Joined: Jul 13, 2009
Post Count: 1066
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CEP2 Work Unit Slot Size (in GB)

I guess if I want to cut it close and only allocate an 8GB RAMdisk to hold 4 concurrent CEP2 WUs, that's my call haha, but I've been warned.

I ran 4 concurrent jobs on a quad-core with a 5120 MB ramdisk for a year without problems. On rare occasion (maybe once a month as I recall) I would get a message that a work unit would not download because of insufficient disk space, but that did not interfere with anything running. This was with 8 GB total memory, so the rest was for the running of the applications.

Now I run 7 concurrent jobs on an i7-3770 with a 12 GB ramdisk with no problems or messages (16 GB total memory).

If you have problems, I doubt that it will be due to insufficient disk space, though it is of course possible that the work units and their working requirements get larger over time.

But I have now bought an additional 16 GB of memory for the new work units and am awaiting their arrival.
[Aug 23, 2013 1:10:40 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 13   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread