Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 
World Community Grid Forums
  Category: Completed Research
  Forum: The Clean Energy Project - Phase 2 Forum
  Thread: Reducing the gap from elapsed time to CPU time

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 37
Posts: 37   Pages: 4   [ Previous Page | 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 5981 times and has 36 replies Next Thread
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
There you are, ubuntu is half or more your problem with cep2, many posts on that v.v. the efficiency of this project on windows versus linux.
[Jun 15, 2014 5:45:19 PM]   Link   Report threatening or abusive post: please login first  Go to top 
branjo
Master Cruncher
Slovakia
Joined: Jun 29, 2012
Post Count: 1892
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
IDN lavaflow, Mac OS X is, similarly to Linix, the Unix clone and I have received only good2great utilization on it.

But, I am not arguing against your statement, because I have never used any Linux flavo(u)r and have not any experiences with it chicken

Cheers peace
----------------------------------------

Crunching@Home since January 13 2000. Shrubbing@Home since January 5 2006

----------------------------------------
[Edit 1 times, last edit by branjo at Jun 15, 2014 6:20:49 PM]
[Jun 15, 2014 6:15:06 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Mamajuanauk
Master Cruncher
United Kingdom
Joined: Dec 15, 2012
Post Count: 1900
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
QUestion Guys - what versions of Windows server support 4 processors?

I know windows 7 & 8 only support 2 processors, so only the server versions support 4p hardware setups, but which ones?
----------------------------------------
Mamajuanauk is the Name! Crunching is the Game!
[Jun 15, 2014 6:47:15 PM]   Link   Report threatening or abusive post: please login first  Go to top 
captainjack
Advanced Cruncher
Joined: Apr 14, 2008
Post Count: 147
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
Mamajuanauk,

First a few words about my build. I have a 6 core 12 thread Intel processor with 2 NVIDIA GPU's in the system. It is running Ubuntu desktop 14.04 and I used the Berkeley installer to install BOINC in my home directory. You will need to change the instructions to match the location of your BOINC installation and your number of threads.

Let's assume that you add a new drive and establish a mount point for the drive called "drive2" and that you want to create your second BOINC client on "drive2". So here goes:

Create a new folder on drive2 called BOINC-31418.

From your primary BOINC folder, copy the "locale" folder from your primary BOINC folder to /drive2/BOINC-31418.

Copy the file named "ca-bundle.crt" from your primary BOINC folder to /drive2/BOINC-31418.

Copy the file named "gui_rpc_auth.cfg" from your primary BOINC folder to /drive2/BOINC-31418. The file contains a password to log onto the BOINC client. Edit the gui_rpc_auth.cfg file in /drive2/BOINC-31418 and change the password to something you can remember. Be sure to not add any extra characters or carriage returns.

Set up or modify the cc_config.xml in your primary BOINC folder to indicate the number of cpu's you want the primary BOINC to use and to <allow_multiple_clients>1</allow_multiple_clients>. My cc_config.xml for my primary BOINC folder is listed below.

<cc_config>
<log_flags>
<file_xfer>1</file_xfer>
<sched_ops>1</sched_ops>
<task>1</task>
<android_debug>0</android_debug>
<app_msg_receive>0</app_msg_receive>
<app_msg_send>0</app_msg_send>
<async_file_debug>0</async_file_debug>
<benchmark_debug>0</benchmark_debug>
<checkpoint_debug>0</checkpoint_debug>
<coproc_debug>0</coproc_debug>
<cpu_sched>0</cpu_sched>
<cpu_sched_debug>0</cpu_sched_debug>
<cpu_sched_status>0</cpu_sched_status>
<dcf_debug>0</dcf_debug>
<disk_usage_debug>0</disk_usage_debug>
<file_xfer_debug>0</file_xfer_debug>
<gui_rpc_debug>0</gui_rpc_debug>
<heartbeat_debug>0</heartbeat_debug>
<http_debug>0</http_debug>
<http_xfer_debug>0</http_xfer_debug>
<mem_usage_debug>0</mem_usage_debug>
<network_status_debug>0</network_status_debug>
<notice_debug>0</notice_debug>
<poll_debug>0</poll_debug>
<priority_debug>0</priority_debug>
<proxy_debug>0</proxy_debug>
<rr_simulation>0</rr_simulation>
<rrsim_detail>0</rrsim_detail>
<sched_op_debug>0</sched_op_debug>
<scrsave_debug>0</scrsave_debug>
<slot_debug>0</slot_debug>
<state_debug>0</state_debug>
<statefile_debug>0</statefile_debug>
<suspend_debug>0</suspend_debug>
<task_debug>0</task_debug>
<time_debug>0</time_debug>
<trickle_debug>0</trickle_debug>
<unparsed_xml>0</unparsed_xml>
<work_fetch_debug>0</work_fetch_debug>
</log_flags>
<options>
<abort_jobs_on_exit>0</abort_jobs_on_exit>
<allow_multiple_clients>1</allow_multiple_clients>
<allow_remote_gui_rpc>1</allow_remote_gui_rpc>
<client_version_check_url>http://boinc.berkeley.edu/download.php?xml=1</client_version_check_url>
<client_new_version_text></client_new_version_text>
<client_download_url>http://boinc.berkeley.edu/download.php</client_download_url>
<disallow_attach>0</disallow_attach>
<dont_check_file_sizes>0</dont_check_file_sizes>
<dont_contact_ref_site>0</dont_contact_ref_site>
<exit_after_finish>0</exit_after_finish>
<exit_before_start>0</exit_before_start>
<exit_when_idle>0</exit_when_idle>
<fetch_minimal_work>0</fetch_minimal_work>
<fetch_on_update>0</fetch_on_update>
<force_auth>default</force_auth>
<http_1_0>0</http_1_0>
<http_transfer_timeout>300</http_transfer_timeout>
<http_transfer_timeout_bps>10</http_transfer_timeout_bps>
<max_event_log_lines>4000</max_event_log_lines>
<max_file_xfers>8</max_file_xfers>
<max_file_xfers_per_project>2</max_file_xfers_per_project>
<max_stderr_file_size>0</max_stderr_file_size>
<max_stdout_file_size>0</max_stdout_file_size>
<max_tasks_reported>0</max_tasks_reported>
<ncpus>10</ncpus>
<network_test_url>http://www.google.com/</network_test_url>
<no_alt_platform>0</no_alt_platform>
<no_gpus>0</no_gpus>
<no_info_fetch>0</no_info_fetch>
<no_priority_change>0</no_priority_change>
<os_random_only>0</os_random_only>
<proxy_info>
<socks_server_name></socks_server_name>
<socks_server_port>80</socks_server_port>
<http_server_name></http_server_name>
<http_server_port>80</http_server_port>
<socks5_user_name></socks5_user_name>
<socks5_user_passwd></socks5_user_passwd>
<http_user_name></http_user_name>
<http_user_passwd></http_user_passwd>
<no_proxy></no_proxy>
</proxy_info>
<rec_half_life_days>10.000000</rec_half_life_days>
<report_results_immediately>0</report_results_immediately>
<run_apps_manually>0</run_apps_manually>
<save_stats_days>30</save_stats_days>
<skip_cpu_benchmarks>0</skip_cpu_benchmarks>
<simple_gui_only>0</simple_gui_only>
<start_delay>0</start_delay>
<stderr_head>0</stderr_head>
<suppress_net_info>1</suppress_net_info>
<unsigned_apps_ok>0</unsigned_apps_ok>
<use_all_gpus>1</use_all_gpus>
<use_certs>0</use_certs>
<use_certs_only>0</use_certs_only>
<vbox_window>0</vbox_window>
</options>
</cc_config>


Now create a cc_config.xml for your BOINC-31418 folder and be sure that it says <allow_remote_gui_rpc>1</allow_remote_gui_rpc>. Note that this one is set up for 2 CPU's on my machine. Set the ncpus to the number you want allocated to your BOINC-31418 client.

<cc_config>
<log_flags>
<file_xfer>1</file_xfer>
<sched_ops>1</sched_ops>
<task>1</task>
<android_debug>0</android_debug>
<app_msg_receive>0</app_msg_receive>
<app_msg_send>0</app_msg_send>
<async_file_debug>0</async_file_debug>
<benchmark_debug>0</benchmark_debug>
<checkpoint_debug>0</checkpoint_debug>
<coproc_debug>0</coproc_debug>
<cpu_sched>0</cpu_sched>
<cpu_sched_debug>0</cpu_sched_debug>
<cpu_sched_status>0</cpu_sched_status>
<dcf_debug>0</dcf_debug>
<disk_usage_debug>0</disk_usage_debug>
<file_xfer_debug>0</file_xfer_debug>
<gui_rpc_debug>0</gui_rpc_debug>
<heartbeat_debug>0</heartbeat_debug>
<http_debug>0</http_debug>
<http_xfer_debug>0</http_xfer_debug>
<mem_usage_debug>0</mem_usage_debug>
<network_status_debug>0</network_status_debug>
<notice_debug>0</notice_debug>
<poll_debug>0</poll_debug>
<priority_debug>0</priority_debug>
<proxy_debug>0</proxy_debug>
<rr_simulation>0</rr_simulation>
<rrsim_detail>0</rrsim_detail>
<sched_op_debug>0</sched_op_debug>
<scrsave_debug>0</scrsave_debug>
<slot_debug>0</slot_debug>
<state_debug>0</state_debug>
<statefile_debug>0</statefile_debug>
<suspend_debug>0</suspend_debug>
<task_debug>0</task_debug>
<time_debug>0</time_debug>
<trickle_debug>0</trickle_debug>
<unparsed_xml>0</unparsed_xml>
<work_fetch_debug>0</work_fetch_debug>
</log_flags>
<options>
<abort_jobs_on_exit>0</abort_jobs_on_exit>
<allow_multiple_clients>1</allow_multiple_clients>
<allow_remote_gui_rpc>1</allow_remote_gui_rpc>
<client_version_check_url>http://boinc.berkeley.edu/download.php?xml=1</client_version_check_url>
<client_new_version_text></client_new_version_text>
<client_download_url>http://boinc.berkeley.edu/download.php</client_download_url>
<disallow_attach>0</disallow_attach>
<dont_check_file_sizes>0</dont_check_file_sizes>
<dont_contact_ref_site>0</dont_contact_ref_site>
<exit_after_finish>0</exit_after_finish>
<exit_before_start>0</exit_before_start>
<exit_when_idle>0</exit_when_idle>
<fetch_minimal_work>0</fetch_minimal_work>
<fetch_on_update>0</fetch_on_update>
<force_auth>default</force_auth>
<http_1_0>0</http_1_0>
<http_transfer_timeout>300</http_transfer_timeout>
<http_transfer_timeout_bps>10</http_transfer_timeout_bps>
<max_event_log_lines>4000</max_event_log_lines>
<max_file_xfers>8</max_file_xfers>
<max_file_xfers_per_project>2</max_file_xfers_per_project>
<max_stderr_file_size>0</max_stderr_file_size>
<max_stdout_file_size>0</max_stdout_file_size>
<max_tasks_reported>0</max_tasks_reported>
<ncpus>2</ncpus>
<network_test_url>http://www.google.com/</network_test_url>
<no_alt_platform>0</no_alt_platform>
<no_gpus>1</no_gpus>
<no_info_fetch>0</no_info_fetch>
<no_priority_change>0</no_priority_change>
<os_random_only>0</os_random_only>
<proxy_info>
<socks_server_name></socks_server_name>
<socks_server_port>80</socks_server_port>
<http_server_name></http_server_name>
<http_server_port>80</http_server_port>
<socks5_user_name></socks5_user_name>
<socks5_user_passwd></socks5_user_passwd>
<http_user_name></http_user_name>
<http_user_passwd></http_user_passwd>
<no_proxy></no_proxy>
</proxy_info>
<rec_half_life_days>10.000000</rec_half_life_days>
<report_results_immediately>0</report_results_immediately>
<run_apps_manually>0</run_apps_manually>
<save_stats_days>30</save_stats_days>
<skip_cpu_benchmarks>0</skip_cpu_benchmarks>
<simple_gui_only>0</simple_gui_only>
<start_delay>0</start_delay>
<stderr_head>0</stderr_head>
<suppress_net_info>1</suppress_net_info>
<unsigned_apps_ok>0</unsigned_apps_ok>
<use_all_gpus>0</use_all_gpus>
<use_certs>0</use_certs>
<use_certs_only>0</use_certs_only>
<vbox_window>0</vbox_window>
</options>
</cc_config>


To be sure that everything is configured correctly, you should shut down BOINC and start it back up again.

Now you should be able to start up the second BOINC client. There are a couple of ways to do that. Option 1 is to open a terminal session and enter the following command "/home/MyName/BOINC/run_client --gui_rpc_port 31418 --dir /drive2/BOINC-31418 --detach". You will need to change "/home/MyName/BOINC" to the location of your primary BOINC installation (wherever run_client is located). If you used the Ubuntu installer, you might need to use a sudo in front of that command. Option 2 is to put that command into a file, make the file executable, then run the file.

Client number 2 should now be running. You can look in the System Monitor and check to make sure that two processes named boinc are running. To access the second boinc client from boinc manager,
Open BOINC Manager
Click on "Advanced"
Click on "Select Computer"
A dialog should open that asks for a computer name and a password. Put "127.0.0.1:31418" in for the computer name and put in the password that you set up in the gui_rpc_auth.cfg file.
Click on "Enter".

You should now be looking at your second BOINC client. You should be able to click on "Advanced" then "Event Log" and see the startup log entries. From there you should be able to attach to WCG and start running tasks.

To attach boinc manager back to your primary boinc client, click on "Advanced", "Select Computer", then for host name enter "localhost", it should automatically pick up the password, then click on enter.

For each additional BOINC client that you want to create, increment the 31418 by 1 for the folder name and the startup command.

I've tried to remember all this from some cryptic notes and observations on my Ubuntu box. If you run into any trouble, let me know and I will try to help out. As usual, it probably takes longer to read the instructions than it takes to set it up. And yes, I have CEP2 tasks running in boinc client number 2 as we speak.

Hope that helps,
captainjack
[Jun 15, 2014 8:04:52 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Mamajuanauk
Master Cruncher
United Kingdom
Joined: Dec 15, 2012
Post Count: 1900
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
Many thanks captainjack, I'll give it a go, I think one client per processor is what I'll be after. I'll give it a go creating ad additional one to start with though...

One questions for now - how do these clients appear in WCG device page? All as one on the machine or as their own entry?

Thanks again!
----------------------------------------
Mamajuanauk is the Name! Crunching is the Game!



----------------------------------------
[Edit 1 times, last edit by Mamajuanauk at Jun 15, 2014 8:41:50 PM]
[Jun 15, 2014 8:39:46 PM]   Link   Report threatening or abusive post: please login first  Go to top 
captainjack
Advanced Cruncher
Joined: Apr 14, 2008
Post Count: 147
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
You should have some WCG tasks running in the primary client before you attach client number 2 to WCG. When you attach client number 2, it should give you a message in the event log that "World Community Grid | Generated new computer cross-project ID: (a bunch of alphanumeric characters)". Then each boinc client should be reported separately. My two boinc clients are reported as 3072512 and 3074606.

Good luck on your adventure.
[Jun 15, 2014 9:08:51 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Paul Schlaffer
Senior Cruncher
USA
Joined: Jun 12, 2005
Post Count: 279
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
Hi Mamajuana,

As mentioned above CEP is a highly IO intensive application. If the system is only running on a single hard drive, I would be surprised if its not generating errors. At the beginning of the WU a large number of small files are generated. This is dragging down your processing time. You can reasonably mitigate this in two ways:

1. Stripe several hard drives in RAID 0, increasing the number until IO response, queue, and disk active time come down to a reasonable level. Your disk active time should never be 100%.
2. Use an SSD, preferably an enterprise class like the Intel DC series or in my case a used X25-E. If I was running this many units I would go this route.

For comparison, with my setup the delta between my CPU and "total" time is 4 minutes per WU. Before the SSD I had these running on a 5 drive RAID 5 array. This was working fine however I also use this system to work. The SSD lowered the disk activity metrics more and has been working great. I think you would be fine with a single enterprise class SSD. Also keep in mind you should have 2Gb of HD space and 1Gb of memory per WU.

Edit: I think the average on the strike-throughed portion is off. Please ignore at this time.
----------------------------------------
“Where an excess of power prevails, property of no sort is duly respected. No man is safe in his opinions, his person, his faculties, or his possessions.” – James Madison (1792)
----------------------------------------
[Edit 3 times, last edit by Paul Schlaffer at Jun 16, 2014 2:16:18 AM]
[Jun 15, 2014 9:16:23 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Paul Schlaffer
Senior Cruncher
USA
Joined: Jun 12, 2005
Post Count: 279
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
QUestion Guys - what versions of Windows server support 4 processors?

I know windows 7 & 8 only support 2 processors, so only the server versions support 4p hardware setups, but which ones?


Windows Server "standard" will support 4 sockets. You can also download it from Microsoft to try before buying a key.
----------------------------------------
“Where an excess of power prevails, property of no sort is duly respected. No man is safe in his opinions, his person, his faculties, or his possessions.” – James Madison (1792)
[Jun 15, 2014 9:23:30 PM]   Link   Report threatening or abusive post: please login first  Go to top 
OldChap
Veteran Cruncher
UK
Joined: Jun 5, 2009
Post Count: 978
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
Mama, re: Paul's comment....

2. Use an SSD, preferably an enterprise class like the Intel DC series or in my case a used X25-E. If I was running this many units I would go this route.

I did my research for another disk intensive project and concluded that the ssd I would try is a Seagate 600 Pro 200. The 100, 200 and 400GB versions are actually bigger but with "built in" over-provisioning. From a speed point of view this has the effect of allowing the drive to maintain good throughput even under the heaviest loads whilst still allowing smart to do it's thing.

This drive is guaranteed for 5 years or 520TB which in effect means it is designed for ~285GB a day for 5 years which shows its roots in the enterprise sector.

In terms of outright speed there are maybe half a dozen or so drives that are faster even when 3/4 full of data (a condition I feel is more relevant to those of us that work our machines) but of those even the drive with the second best TBW expectation (Intel 730) comes in a distant second on life expectancy of 127.5TB or 70GB/day for 5 years. normal consumer drives seem to top out around 44GB/day.

I conclude that these drives (not the 120 240 480 versions) are a niche product well suited to hard work.

EDIT: Running a mix of projects my linux box (2p 16c 32t) loses about 0.2 hours on a 6 hour wu
my Win7 box (1p 8c 16t) loses around 0.5 hours on a 9 hour wu. Both boxes use ssd's and same cpu's.

I was told don't bother with trying to change priorities because it does not obey anyway.
----------------------------------------

----------------------------------------
[Edit 1 times, last edit by OldChap at Jun 15, 2014 10:42:27 PM]
[Jun 15, 2014 10:24:26 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Mamajuanauk
Master Cruncher
United Kingdom
Joined: Dec 15, 2012
Post Count: 1900
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Reducing the gap from elapsed time to CPU time
Many thanks OC, that drive looks impressive! However, I have several spare drive available, so I'm going to try splitting the 4 CPU's to 4 separate clients as per captainjacks suggestion above, if that doesn't improve things, then I'll give a new drive a try

Many thanks to all who have contributed to this thread with comments and advice, as I have two of these machines, please don't stop with the suggestions, I can try them on the other machine...

Your assistance is, as always very much appreciated!
----------------------------------------
Mamajuanauk is the Name! Crunching is the Game!
[Jun 16, 2014 7:31:51 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 37   Pages: 4   [ Previous Page | 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread