Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 35
Posts: 35   Pages: 4   [ Previous Page | 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 6200 times and has 34 replies Next Thread
Yarensc
Advanced Cruncher
USA
Joined: Sep 24, 2011
Post Count: 136
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

This is the BOINC settings the CEP scientists recommend for optimizing this science, the most relevant (I think) are the memory settings (7 and 8) which talk about making sure you have enough RAM for the whole science to run there.
http://static.molecularspace.org/uploads/2013..._CEP2_Custom_Settings.pdf

One reason the disk usage might be so high is the VM usage SekeRob mentioned above. In windows 7 you can check the 'real' memory usage (not just what's shown in task manager) by opening 'resource monitor' from the Performance tab in task manager and going to the memory tab. The two main things you'd want to look at are the 'Hard Faults/sec' column (how often that process runs out of RAM and has to store things on the disk) and the 'Working Set' column, which is how much total memory the task is using (RAM + what's been shoved off to the disk). If the Working Set number is a lot bigger than the amount of memory task manager says the process is using, it means a lot of that app is being stored on disk, which could account for some of your 100+GB of disk usage.
[Aug 24, 2015 7:34:56 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

Use an HDD instead of an SSD if you're going to do CEP2. Any write speed advantage a new SSD has compared to a short stroked HDD, is more or less gone when the SSD fills up and begins to overwrite old data. SSDs are not ready for write-intensive tasks or virtualization, at least not until XPoint is paired with NAND in future SSDs, sometimes late next year.


I'm crunching 12 threads OET in an Ubuntu guest located on the first HDD (C:) and 12 threads CEP2 in a W7 host on a second HDD, without any problems. Since I put the BOINC Data file on a second HDD and short stroked that HDD down to 2,5%, the previous problem I had with resetting CEP2 wus is now gone. The computer needs 10W more now, compared to when I only had one HDD and did 24 threads OET, but that's a small price to pay for a greener planet. Let's just hope Bao Group gears the algorithms towards finding low resistance, carbon based molecules with band gaps suitable for photovoltaic cells, instead of carbon based LEDs. She is after all sitting on LG's science advisory board--the biggest OLED manufacturer on the planet--so there's room for doubt ...
[Aug 24, 2015 9:42:31 PM]   Link   Report threatening or abusive post: please login first  Go to top 
ExtraTerrestrial Apes
Cruncher
Joined: Nov 7, 2009
Post Count: 12
Status: Offline
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

Thanks for all those replies, guys!

I may have fixed the worst part of my setup: use 50% of memory while PC in use, 90% otherwise. I've had the setting like this for years without giving it much thought. Might explain why I saw little memory usage / lot's of free memory when watching the tasks.. let's see how it works with 80%.

And I really like the idea of symbolic links for the files pushed into the slot directories. that would be a cool feature to include into BOINC, so all projects benefit. Call it "lazy / deferred write" and only copy & modify files if the application wants to modify them. That should seldomly or never happen, as the apps probably generate new result files for each WU anyway.

MrS
----------------------------------------
ExtraTerrestrial Apes - Scanning for our furry friends since Jan 2002
[Aug 25, 2015 8:31:12 PM]   Link   Report threatening or abusive post: please login first  Go to top 
ExtraTerrestrial Apes
Cruncher
Joined: Nov 7, 2009
Post Count: 12
Status: Offline
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

A quick notice of what I'm seeing, after allowing BOINC to use up prety much all my main memory. WCG CEP2 executable:

- uses 100 - 110 MB accordsing to Task Manager
- has a working set of 100 - 110 MB according to Ressource Monitor
- shows disk activity every few seconds, often in the 0.x MB/s range but also frequently about 6 MB/s for a few seconds
- SSDReady shows projected 160 GB/day (normal would be 33 for my system

4.3 GB of 8 GB main memory are used. During the observation time the WU went from ~4% to 6% - not sure if things change later on. The rest of the config is: Win 8.1 64 & BOINC 7.6.6 (not sure what else might matter).

For now I would conclude: "I don't think I'm seeing the behaviour I should have seen according to your comments". And "if CEP2 wants more memory, it's not getting it or not taking it".

MrS
----------------------------------------
ExtraTerrestrial Apes - Scanning for our furry friends since Jan 2002
[Aug 26, 2015 8:25:17 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

The analysis tool of choice amongst BOINC insiders is Process Explorer [PE] and Process Monitor [PM] by SystemInternals, division products of MS. They got a whole suite of [free] tools to dissect what's going on. For CEP2 the one to look at is wcgrid_cep2_qchem_prod_win32.exe.7.00 which after each checkpoint exits and starts a new job in the series of maximum 9 8 [most of the times the logic calls it quits after 6 jobs have run]. The Disk and network tab of the process is just showing zero for me, except when a job finishes. Now 17 minutes into job #3 (4 really), 385MB VM + 98MB peak RAM use is showing. With PM setting a process filter for 'begins with' wcgrid_cep2 reveals the 6700 or so links to the input/output files used by CEP2. Narrow the operations filter to WriteFile and it shows hundreds upon hundreds of files open, then proceed to filter that on 'Show file system activity'. See nothing exciting what I've not seen before with this science.
----------------------------------------
[Edit 1 times, last edit by Former Member at Aug 26, 2015 10:23:55 PM]
[Aug 26, 2015 10:02:14 PM]   Link   Report threatening or abusive post: please login first  Go to top 
enels
Senior Cruncher
Joined: Apr 25, 2008
Post Count: 286
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

Would putting the BOINC data files on a USB 3.0 thumb drive help?
[Aug 26, 2015 10:20:35 PM]   Link   Report threatening or abusive post: please login first  Go to top 
ExtraTerrestrial Apes
Cruncher
Joined: Nov 7, 2009
Post Count: 12
Status: Offline
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

Would putting the BOINC data files on a USB 3.0 thumb drive help?

No, not really. Those drives are cheap (at least the small ones), but for a good reason: the flash used for them is usually of lower quality and has less endurance than full SSDs. Plus the controller is usually less advanced and has less DRAM cache to organize things cleverly (to keep write amplification in check). All in all: if a flash based drive is going to get "hammered" with writes it should either be an Enterprise model made for that (would be too expensive for me) or a relatively cheap mainstream drive. In that case it doesn't matter as much if it only lasts a few years.

Others have suggested to use some old HDD instead. That is good advice, as long as its not a too loud drive and can fit physically (won't e.g. in a laptop). 2.5" drives are especially energy efficient. They'd consume about 2W on average, which under 24/7 operation would cost about 4€/year in Germany.

MrS
----------------------------------------
ExtraTerrestrial Apes - Scanning for our furry friends since Jan 2002
[Aug 27, 2015 7:45:26 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7846
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

Would putting the BOINC data files on a USB 3.0 thumb drive help?

I have done this thrice, not only BOINC, but the OS (Linux Mint) also. The first time I used the cheapest no name drive I could find, I think it was $3.99 for an 8 gig drive. It lasted about 4 months and then froze up. The second time I used a slightly better drive, $5.99 for a 16g (PNY) drive( prices have come down lately). That one lasted about 10 months and then froze with the message it had turned into a read only drive. Currently I have another 16g drive (SanDisk) running (about 2 months). They have been run on a Dell 2650 server with 8 cores running 24/7. All have been USB2.0. I am quite sure they can't use much electricity. Even though they fail eventually, I am satisfied with the performance. Been running mainly MCM1 over this time period.

Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
----------------------------------------
[Edit 1 times, last edit by Sgt.Joe at Aug 27, 2015 11:56:24 PM]
[Aug 27, 2015 11:52:16 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

Once an SSD has filled less than all pages in a block, it usually needs to erase the whole block before it can write more data to its pages. This slows it down and causes more wear on the SSD than the actual amount of data BOINC needs to write when checkpointing CEP2. SSDs are great for all projects, except CEP2 (UGM?).



Extremetech


Look what happens when the garbage collection process becomes overwhelmed, after about 30 minutes of CEP2-like writing. A total latency collapse. The SSD becomes slower than a standard HDD.



StorageReview
----------------------------------------
[Edit 1 times, last edit by Former Member at Aug 28, 2015 5:06:37 AM]
[Aug 28, 2015 3:37:16 AM]   Link   Report threatening or abusive post: please login first  Go to top 
ExtraTerrestrial Apes
Cruncher
Joined: Nov 7, 2009
Post Count: 12
Status: Offline
Reply to this Post  Reply with Quote 
Re: excessive disk transfers

TBMS, your data is solid. I only disagree at one point:
Look what happens when the garbage collection process becomes overwhelmed, after about 30 minutes of CEP2-like writing.

The writing performed there is constantly hammering the drive with 4k writes, at the maximum performance the drive can sustain. There's not a single break for the controller to recover. That's not what I see during CEP2. It often writes, yes, but that's a small amount of data every few seconds. And then a few 10 MB maybe once a minute (not measured). There is a lot of idle time in between, hence any half-decent SSD (not the JMicron from 2009) won't have a problem with that load. Except that it has to "pay" for the writes, of course.

An some more updates:
A WU has finished with the setting of more available RAM. It increased credits per hour by a factor of 2 compared to the previous one! I don't know how statistically valid those numbers are, though (just a single sample for each setting).

The current one is between 24 and 26% and still shows 110-120 MB usage in task manager, whereas the working set has increased to 210 - 220 MB. Overall memory load still 54%, with 80% allowed for BOINC.

MrS
----------------------------------------
ExtraTerrestrial Apes - Scanning for our furry friends since Jan 2002
[Aug 28, 2015 9:52:37 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 35   Pages: 4   [ Previous Page | 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread