Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 10
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 1475 times and has 9 replies Next Thread
cqexbesd
Cruncher
Joined: Oct 13, 2008
Post Count: 14
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Storing Data In a Different Directory

Hi,

I want BOINC to store its data files in a separate location to its executables (including the downloaded executables for each project). Basically I have a large amount of space mounted no-exec and a much smaller amount of space where things can be exec'ed from. I tried the --dir argument but it seemed to move the per project executables as well.

I hoped to find someone having written up step by step instructions but my search skills have failed me...

Thanks,

Andrew
[Oct 5, 2012 7:20:57 PM]   Link   Report threatening or abusive post: please login first  Go to top 
cqexbesd
Cruncher
Joined: Oct 13, 2008
Post Count: 14
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

I should have said this is under Linux. Sorry!
[Oct 5, 2012 7:25:23 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Bearcat
Master Cruncher
USA
Joined: Jan 6, 2007
Post Count: 2803
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

Sek is the master of this subject but think it would be wise to clear your cache, uninstall, then reinstall to insure it installs properly. But Sek probably knows the commands to do this.
----------------------------------------
Crunching for humanity since 2007!

[Oct 5, 2012 8:19:55 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

Hi Andrew,

Installing the BOINC core client to any other place than standard as it takes about 20MB and is called on only once during startup is not what you want to do, but you could **. After that, all science applications and data come from the separate data_dir which on Ubuntu is default /var/lib/boinc-client. Separating the science application and data is *not* possible, but moving of the data_dir in past I've done by including an <options> line in the cc_config.xml called

<data_dir>location address</data_dir>

See manual: http://boinc.berkeley.edu/wiki/Cc_config.xml

If your search skills did not find anything in the Start Here FAQ forum, then those skills could do with some polishing. There's a topic on the SH index... Section 1 Sub M-9 [it's for Windows though FAIK].

--dir is BTW in effect the same [I've never used it], but as said, you can't split the science apps / project directory from the actual slot data. BOINC functioning depends on a hierarchical structure for anything it needs to build a task in a slot. That said, BOINC is open source, so you could compile your own, but for support you'd have to go to the developers. Their forum is http://boinc.berkeley.edu/dev/index.php

Edit: ** For a popular solution such as having a mobile install on a USB Drive. Got a USB 3.0 [on Ubuntu 12.04] for dual boot.
----------------------------------------
[Edit 1 times, last edit by Former Member at Oct 6, 2012 8:12:38 AM]
[Oct 6, 2012 7:33:02 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Ingleside
Veteran Cruncher
Norway
Joined: Nov 19, 2005
Post Count: 974
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

After that, all science applications and data come from the separate data_dir which on Ubuntu is default /var/lib/boinc-client.

(snip)

you can't split the science apps / project directory from the actual slot data. BOINC functioning depends on a hierarchical structure for anything it needs to build a task in a slot.

Well, under Windows atleast it's fairly easy to separate the slots-directory and the projects-directory, and it's strange if this doesn't also work under Linux...

Now, I'm not running Linux, but something like this should probably work in a command-window, to move the slots-directory.
Note, make sure BOINC is fully stopped before trying this:
mv /var/lib/boinc-client/slots /something/boinc_data/slots
mkdir /var/lib/boinc-client/slots
ln -s /something/boinc_data/slots /var/lib/boinc-client/slots
Note, apparently the ln-command wants things backward, so instead of source -> destinatination like for moving files and so on, it's destination <- from-point. rolling eyes

Also, as already noted I'm not running Linux, so before someone tries this I would recommend to make a backup, or finish all work on client before trying it.


Under Windows 7, the similar would be something like this:
1: Stop BOINC.
2: Uninstall BOINC, to not run into any access-rights-problems.
3: Move slots-directory from c:\programdata\boinc\slots to example d:\boinc and in the process creating d:\boinc\slots. (again due to access-rights)
4: Open a cmd-window, with "Run as Administrator"-privileges. Type this command:
mklink /d c:\programdata\boinc\slots d:\boinc\slots
5: If no error-message, re-install BOINC & re-start BOINC.

Note, c:\programdata\boinc\slots must not be present to use mklink to create the link to another directory.

Note, if a program isn't aware the physical location is different from the supplied path, it will incorrectly detect free disk-space. Meaning, in the example, if c:\ has 0.5 GB free disk space while d:\ has 1 TB+ free disk space, it's a good chance BOINC will give a "not enouth free disk space" to download a CEP2-task demanding 1 GB+ free space.

Similarly, if c:\ has 10 GB+ free space but D:\ has only 50 MB free space, BOINC will happily allow you to download the CEP2-task, but it will crash on startup giving most likely a different error-message than "out of disk-space".

As long as both c:\ and d:\ has enough free disk space, I'm not aware of any problems doing this. Afterall, even old dos-programs never heard about "software links" or "Reparse points" will happily run error-free.

Older windows-versions like windows-2000 did not include mklink, but these versions can use linkd.exe (chances are must download it) to make "Reparse points" instead of soft-links. Not sure if Vista also included mklink.


Now, if moving slots-directory to another hd would give any speed-advantage or not is another matter, but atleast it's doable for anyone wanting to test this out. Atleast in my experience, CEP2 still crapped-out even got a two-disk RAID just for the slots-directory to play with. sad
----------------------------------------


"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
----------------------------------------
[Edit 1 times, last edit by Ingleside at Oct 6, 2012 4:48:02 PM]
[Oct 6, 2012 4:44:41 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

Please be so kind if you say A, Ingleside, to finish the B part too. There is *no* advantage to split the project folders that hold the source science executable and dynamic task data i.e. a mishmash of data and science programs from the /slots. If you want to run it that way, fine. If you wish to proof you can split /projects/* from /slots just because... good luck. Just don't think anyone is helped by doing this. Consider that.

BTW, you can do symlinking on Linux. It was "borrowed" by Microsoft. Symlinking was tried on CEP2 and it miserably failed to produce valid results.
[Oct 6, 2012 6:31:51 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Ingleside
Veteran Cruncher
Norway
Joined: Nov 19, 2005
Post Count: 974
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

BTW, you can do symlinking on Linux. It was "borrowed" by Microsoft. Symlinking was tried on CEP2 and it miserably failed to produce valid results.

If my recollection isn't too fuzzy the corruption came from some users trying to be "clever" by linking individual files, forgetting that while two files could be the same at start of task, this didn't need to be the case later in the task...

Placing the whole slots-directory another location on the other hand works with no corruption on CEP2-tasks, atleast under Windows.

While putting the slots-directory on a separate hd did shave-off the initial startup-time of CEP2-tasks with some seconds, as already mentioned this didn't fix the 6466 reasons for CEP2 to still crap-out.
----------------------------------------


"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
----------------------------------------
[Edit 1 times, last edit by Ingleside at Oct 6, 2012 7:31:43 PM]
[Oct 6, 2012 7:24:55 PM]   Link   Report threatening or abusive post: please login first  Go to top 
cqexbesd
Cruncher
Joined: Oct 13, 2008
Post Count: 14
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

Thanks. When I said I couldn't find anything with a search I meant I couldn't find anything that solved my problem - not that I couldn't find how to move the data directory. Because the larger partition is marked noexec things will just fail if the per-project apps are stored on there. Unless you mean there is something else in the FAQ and I have missed it twice...

I realise I could make changes to the actual client and rebuild but I was hoping for a lower cost solution if possible (i.e. a pre-tested setting) - it has taken me months to find enough time to build the client in the first place just to run it on this machine.

I might look into symlinking next time I get the chance. If it takes the bulk of the data and the writes elsewhere it might be enough. I didn't want to resort to hacks until I knew there wasn't a proper method.

Thanks,

Andrew
[Oct 7, 2012 3:33:11 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

Andrew, if not already aware, when a task is set up in a /slots/xx the data task files + a copy of the science application is then copied there and processed from there. BOINC does this to ensure per-task integrity, so you can have as many of the same run concurrent in their own memory segment. If understanding correctly then your noexec partition would not be an option the way BOINC works.
[Oct 7, 2012 6:18:23 PM]   Link   Report threatening or abusive post: please login first  Go to top 
cqexbesd
Cruncher
Joined: Oct 13, 2008
Post Count: 14
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Storing Data In a Different Directory

Thanks, I wasn't aware but I just found out by trying it. Looks like what I wanted to do isn't possible without changing the source... sad

Thanks for your help!
[Oct 7, 2012 9:23:50 PM]   Link   Report threatening or abusive post: please login first  Go to top 
[ Jump to Last Post ]
Post new Thread