Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 45
Posts: 45   Pages: 5   [ Previous Page | 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 6045 times and has 44 replies Next Thread
Acibant
Advanced Cruncher
USA
Joined: Apr 15, 2020
Post Count: 126
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

Is there a reason to prioritise one project over another like 4 ARP and 2 MCM1 (or any other combination) instead of the default 1 APR and unlimited everything else in the default profile.
Because ARP is beefy you have the need to keep it to half of threads at most but beyond that it's really your preference. You might push for badges or you might have preferred projects.

I want to compensate for the lower-power machines that also might be memory-limited so I do as many ARP as possible. Conversely, I am no longer doing any OPN because they have a limit to how many work units they can handle per day and it's more efficient for them to be done on a GPU and not a CPU. So limiting CPU demand lets them issue more GPU work units versus CPU. (None of the other projects are GPU-enabled.) It also seems like MCM gives more points per hour of processing than OPN in my experience. So those factors lead me to do half-ARP/half-MCM with HST also opted-in (but I don't bother with limits on it since it's so rare as it is).
----------------------------------------

[Jul 31, 2021 5:15:39 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Mike.Gibson
Ace Cruncher
England
Joined: Aug 23, 2007
Post Count: 12146
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

I should add that app_config.xml needs to be installed on each machine individually if you want to use it on that machine.

The profile settings can differ from machine to machine, but you only have 4 different profiles available.

Mike
[Jul 31, 2021 9:09:34 PM]   Link   Report threatening or abusive post: please login first  Go to top 
BobbyB
Veteran Cruncher
Canada
Joined: Apr 25, 2020
Post Count: 603
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

It seems I am kind of doing the same thing but by fluke. I restricted ARP from 2 machine because I feel they are not up to snuff (lower-power machines) and gave them to the Ryzen. I really have no GPU worth talking about in this context.

I figured app_config.xml was for each machine. I guess I could up the ARP to 4.

Without an app_config.xml I have observed 3 ARPs running. They just bubble up to the top. So when I examine the sample XML above I presume the profile allows the number of WU to be downloaded (5 ARP) but only 4 concurrent running in app_config.xml.

So my profile for the Ryzen should be 5 or more ARPs and leave the rest alone but concurrent specified as 4 in the XML.

I could use trial and error for all this but will learn it faster here with people who have been there.
[Jul 31, 2021 11:35:14 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Acibant
Advanced Cruncher
USA
Joined: Apr 15, 2020
Post Count: 126
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

So my profile for the Ryzen should be 5 or more ARPs and leave the rest alone but concurrent specified as 4 in the XML.
Correct, if you only want one extra ARP work unit in the queue. Note that if you are opted in to other projects (and you will be if you are only using half your threads for ARP) and you don't specify limits either for concurrent or maximum downloaded units for those other projects, the client could then legitimately just download and process only units for those and ignore ARP. It of course depends on the WCG servers and the work being handed out at the moment the client requests more so something to keep in mind. These are maximums being specified and not minimums.

If you do cap MCM for instance the client will be forced to download some ARP to try to get to your specified "cache" value in the online preferences. This brings up a quirk where if there are any projects you have not opted into, if your caps for the ones you are doing are seen by the client as not being enough to get to that cache value, it will go and download from those projects you don't normally work on to get more of a backlog unless you uncheck "If there is no work available for the project(s) I have selected above, please send me work from another project."

The client can also hold onto several tasks that it has uploaded but not yet reported. These do unfortunately seem to count against the limits you set online for downloaded tasks (but not for max concurrent). So if you really ride the knife's edge with just one extra work unit, it may be worthwhile to specify the "report_results_immediately" option to keep from losing all backlog as soon as just one completes. I only do that for ARP since they take so long to process. I let the client naturally batch together task reporting for shorter ones like MCM so the servers aren't constantly getting hit with those (but this can require a larger download buffer to avoid unused threads waiting for new work).
   <app>
<name>arp1</name>
<max_concurrent>4</max_concurrent>
<report_results_immediately/>
</app>

Sorry if I'm getting too into the weeds here. It all depends on just how much control and micromanagement you want to do.
----------------------------------------

[Aug 1, 2021 1:22:00 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Mike.Gibson
Ace Cruncher
England
Joined: Aug 23, 2007
Post Count: 12146
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

Even the slower machines can do ARP, so long as they are not so slow as to exceed the deadline. Just don't use more than half the threads on each. If they take more than half the deadline then the profile for that machine should not have an extra unit. If they take more than 2 days they will not classify as 'reliable' but that just means you will get units from recent generations and not stragglers.

If you have 8 threads available then you can run 4 ARP at a time. Scale up or down according to the threads on the machine.

ARP will always rise to the top because their units take much longer than the others (apart from HST).

If you have more than 1 spare ARP unit then your machine might not classify as 'reliable' because the time that you have the units will go up. You don't need any more because they are readily available at the moment.

If you have more than 4 machines then run similar machines on the same profile.

Mike
[Aug 1, 2021 1:33:07 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Acibant
Advanced Cruncher
USA
Joined: Apr 15, 2020
Post Count: 126
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

The profile settings can differ from machine to machine, but you only have 4 different profiles available.
There's actually a bit of a clever "hack" to get around this. I've written up a Google Doc describing it for you.
----------------------------------------

[Aug 1, 2021 1:38:07 AM]   Link   Report threatening or abusive post: please login first  Go to top 
BobbyB
Veteran Cruncher
Canada
Joined: Apr 25, 2020
Post Count: 603
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

Mike.Gibson
Even the slower machines can do ARP, so long as they are not so slow as to exceed the deadline. Just don't use more than half the threads on each. If they take more than half the deadline then the profile for that machine should not have an extra unit. If they take more than 2 days they will not classify as 'reliable' but that just means you will get units from recent generations and not stragglers.
The machines are only 4 threads which is why I restrict them. Am looking at replacing 1 for something better. The other is Windows and it reboots after an update even though I "told" it to ask before booting. The rest are Ubuntu and do nothing by themself.
[Aug 1, 2021 3:52:19 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2089
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

On this system I have 6 threads (recently upped from 3) dedicated to ARP with a cache of only just one more. Their turnaround time varies from 10 to 17 hours, with a clear mean of 12 hours. In this way I can finish 12 ARP1-tasks per day by returning one ARP1-result each 2 hours.
----------------------------------------
[Edit 2 times, last edit by adriverhoef at Aug 1, 2021 2:45:42 PM]
[Aug 1, 2021 2:43:54 PM]   Link   Report threatening or abusive post: please login first  Go to top 
BobbyB
Veteran Cruncher
Canada
Joined: Apr 25, 2020
Post Count: 603
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

Acibant:
There's actually a bit of a clever "hack" to get around this. I've written up a Google Doc describing it for you.
WOW you sure went to great lengths for this hack.

OK so I upped to 4 ARPS. Will check resulting stats.
----------------------------------------
[Edit 1 times, last edit by BobbyB at Aug 1, 2021 4:00:21 PM]
[Aug 1, 2021 3:33:06 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Jim1348
Veteran Cruncher
USA
Joined: Jul 13, 2009
Post Count: 1066
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: How long does validation take

Each CPU is different, but Mike's advice to use only half the cores is the best starting point. On my Ryzen 3600 (Ubuntu 20.04.2) I restrict it to four cores with an app_config.xml (or a second BOINC instance in some cases). That chip has a lot of nice cache that the ARP's like.

It then depends on what is running on the other cores, and how much cache they take. At the moment, the other eight cores are running SiDock (CurieMarieDock), and the ARP's are taking a little over 11 hours. I think Win10 will be a little longer for the ARP's. The CM Dock take a bigger hit on Windows. You could use MCM with ARP and do well on either OS.
----------------------------------------
[Edit 3 times, last edit by Jim1348 at Aug 1, 2021 6:34:21 PM]
[Aug 1, 2021 6:29:33 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 45   Pages: 5   [ Previous Page | 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread