| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Completed Research
Forum: Microbiome Immunity Project
Thread: Errors when running MIP from a ram disk |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 11
|
|
| Author |
|
|
phytell
Cruncher Joined: Sep 8, 2014 Post Count: 39 Status: Offline |
Hi all,
As you can probably guess from the subject, I've taken to running a lot of WCG work from ram disks, especially when the WUs need to unpack archives, as is the case with MIP. Specifically, I'll create a junction from boinc/slots to say, s:/slots (I'm running on Windows), and watch as i/o performance goes through the roof. This strategy has worked amazingly well for everything from CEP2 to SCC, but is suddenly resulting in output file not found errors for MIP. Investigation seems to point to an issue with relative pathnames, but I can't narrow it any further than that. Is anybody else trying this same strategy, and if so, do you have any tips that could help me resolve the problem? Some of my machines are running on both older and slower HDDs, which means they really, really need to be working in ram instead. |
||
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges: 
|
I often use Primo Ramdisk Pro for the BOINC DATA folder when running Windows. It has an option to create the junction points that might work better, though I think it is just a GUI for the standard Windows junction points and may not solve it.
----------------------------------------But I think they allow a trial version if you want to try it for free. I don't need it for slow HDDs, since I am all on SSD, but I do it to protect the SSD from the high writes of some projects. Another advantage with the "Pro" version at least is that it allows the use of hidden memory. Since I am on Win7 Home Premium 64-bit, I am normally limited to the use of 16 GB memory. But the Primo Ramdisk Pro allows the ramdisk to be placed on the rest of the 32 GB that I have installed. [Edit 1 times, last edit by Jim1348 at Apr 16, 2018 3:51:50 PM] |
||
|
|
phytell
Cruncher Joined: Sep 8, 2014 Post Count: 39 Status: Offline |
Hi there,
----------------------------------------I'm glad to hear I'm not totally alone in ramdisk land. I'll rebuild the linkage again and make sure to use a directory junction this time; I believe the last attempt at least was a symbolic link. Thanks for the help; I'll post back here with my results as soon as the test machine completes a unit (three cheers for Windows xp!). Eta is somewhere around six to eight hours. The archives have extracted fine at least; my ramdisk has 54m used, which matches up with two MIP units. As a side note, I'm using Soft Perfect's ramdisk application. Its served me beautifully, and has no limits on ramdisk size as far as I know (obviously you're limited by file systems, but if you have enough ram for that to be an issue then we're in a whole other world). Its also (or was when I downloaded it) completely free. [Edit 1 times, last edit by xorg-21 at Apr 16, 2018 8:06:01 PM] |
||
|
|
phytell
Cruncher Joined: Sep 8, 2014 Post Count: 39 Status: Offline |
Alright, a result has completed. It errored out. Log follows:
Result Log Result Name: MIP1_ 00066105_ 0630_ 0-- <core_client_version>7.2.47</core_client_version> <![CDATA[ <stderr_txt> [standard text] Starting work on structure: _0001 Finished _0001 in 10393.9 seconds. ====================================================== DONE :: 1 structures in 10514.4 cpu seconds ====================================================== BOINC :: BOINC support services shutting down cleanly ... 18:56:10 (2656): called boinc_finish(0) </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>MIP1_00066105_0630_0_r1265021548_0</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]> Of course, the results work just fine when I don't junction slots to the ramdisk. |
||
|
|
David410
Cruncher Joined: Dec 16, 2017 Post Count: 27 Status: Offline Project Badges: 
|
First I want to say that I am not an expert, but these informations may be useful to you. As far as I know, most of the projects use soft links to get the data they need. And most of the useful files for the workunits running in the slots are found in projects/www.worldcommunitygrid.org. I think that the file that made your WU error out was in that directory, but for some reason BOINC didn't menage to get it in order to upload it. This is strange though, because in order to just start the WU you need files in that directory and in your computer the WU was completed, just not uploaded.
|
||
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges:
|
I am not quite sure what is going wrong. But you could just create the ramdisk, and then create the directory "BOINC" on the ramdisk. Then, instead of using junction points, you could just re-install BOINC and point it to that directory on the ramdisk.
----------------------------------------However, often times the registry remembers the entries to the old data directory location, and you have to manually hack them out of the registry if it is not a clean install (i.e., the first time BOINC is installed on that OS), or else BOINC gives errors during installation about not finding the directory. It would probably help to uninstall BOINC first, which might get rid of the old registry entries, but does not always do so. If there are still errors, it may be a problem with Soft Perfect; I tried it a long time ago and had some problems, though I don't remember what. Good luck. PS - There is also DATARam Ramdisk, which works well and is low cost, but does not have all the features of Primo Ramdisk. You might have to use an older version for XP compatibility though. That could be a problem with any of them now. [Edit 3 times, last edit by Jim1348 at Apr 17, 2018 1:10:43 PM] |
||
|
|
phytell
Cruncher Joined: Sep 8, 2014 Post Count: 39 Status: Offline |
I'd actually done something like that previously, having both /projects and /slots on the ramdisk. The difficulty with this is that work units are a lot more vulnerable to a crash, since checkpoint files are stored in /projects. You also need a lot more ram (my /projects directory has been over a gigabyte at times).
Its also worth noting that no other application (at least as far as I've been able to tell) has had this form of issue. Does anyone happen to know how much disk I/O MIP actually uses? While I'd really prefer to be able to use ramdisks in all cases, if the only major work occurs at the start of the unit, its not nearly as crucial as it would be if the WUs are like CEP2 used to be. |
||
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges:
|
Does anyone happen to know how much disk I/O MIP actually uses? While I'd really prefer to be able to use ramdisks in all cases, if the only major work occurs at the start of the unit, its not nearly as crucial as it would be if the WUs are like CEP2 used to be. Nothing is as awful as CEP2 (in this universe at any rate). In fact, that is why I started using a ramdisk to begin with. But since you ask, I just happen to have the BOINC Data folder on a ramdisk (DATARam), and started running two MIP on an i7-4771 machine (Win7 64-bit). After allowing it to stabilize for 30 minutes, the writes are around 180 MB/day, or 90 MB/day per work unit. That is nothing. The OS will use more than that. You should not need a ramdisk for that at all I would think. [Edit 1 times, last edit by Jim1348 at Apr 17, 2018 10:20:45 PM] |
||
|
|
phytell
Cruncher Joined: Sep 8, 2014 Post Count: 39 Status: Offline |
Thanks for the numbers. That, at least, makes me feel a bit better about the units running from HDD.
I think I'll just let MIP continue as it is, and see how future projects handle running from ram. It would be really nice if we could see some statistics like yours as part of the system requirements for a project; for instance "this project will require 100 mb project storage, 28mb per work unit, and require x I/O throughout the process of completing a single unit". I know there's some form of specifications, but they're usually so vague as to be useless, and make no mention of actual read/write intensity. Ironically enough, CEP2 was probably the closest to accurate that I've seen. |
||
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges:
|
It would be really nice if we could see some statistics like yours as part of the system requirements for a project; for instance "this project will require 100 mb project storage, 28mb per work unit, and require x I/O throughout the process of completing a single unit". Yes, that would be useful, though it may have to be contributed by volunteers. The admins could set up a standard format though, and make it a sticky. |
||
|
|
|