| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Support
Forum: GPU Support Forum
Thread: HCC GPU - Thread Priority should be increased (to 6) to not starve the GPU |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 28
|
|
| Author |
|
|
Jacob Klein
Cruncher Joined: May 31, 2007 Post Count: 28 Status: Offline Project Badges: 
|
knreed / WCG HCC Developers:
I use one of my video cards to work on the Help Conquer Cancer (HCC) GPU application. And I use <gpu_usage>0.5</gpu_usage> to do 2 tasks on 1 GPU. But I have a problem with its current implementation. Basically, using Process Explorer, I see the following for the wcg hcc1 executable: - when I use an app_config that specifies <cpu_usage>1</cpu_usage>, the base priority is 4, and the priority of the actual thread using the CPU is 1. - when I use an app_config that specifies <cpu_usage>0.999</cpu_usage>, the base priority is 6, and the priority of the actual thread using the CPU is 1. Now, because of the way I have GPUGrid GPU apps configured, sometimes my CPU is overloaded (I'll have 10 apps trying to use only 8 CPUs; 2 GPUGrid instances, 6 CPU apps, and 2 HCC1 instances). When this happens, guess what happens? The HCC1 instances, because of only having 1 as their thread priority, end up getting virtually NO CPU. And then the GPU Load goes down, and task performance suffers. I propose that you do what GPUGrid appears to have done. Namely these 2 things: 1) always put your process priority to 6 (instead of only on fractional GPU usage, since this is a GPU app) 2) always put your thread priority to 6 (to ensure that, when the system is overloaded, the GPU does not get starved) Can you please look into doing this? Thanks! Jacob |
|||
|
||||
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
What settings are you using for GPUGrid and what was the motiviation for those settings? I want to understand the whole scenario.
|
||
|
|
nanoprobe
Master Cruncher Classified Joined: Aug 29, 2008 Post Count: 2998 Status: Offline Project Badges: 
|
knreed / WCG HCC Developers: I use one of my video cards to work on the Help Conquer Cancer (HCC) GPU application. And I use <gpu_usage>0.5</gpu_usage> to do 2 tasks on 1 GPU. But I have a problem with its current implementation. Basically, using Process Explorer, I see the following for the wcg hcc1 executable: - when I use an app_config that specifies <cpu_usage>1</cpu_usage>, the base priority is 4, and the priority of the actual thread using the CPU is 1. - when I use an app_config that specifies <cpu_usage>0.999</cpu_usage>, the base priority is 6, and the priority of the actual thread using the CPU is 1. Now, because of the way I have GPUGrid GPU apps configured, sometimes my CPU is overloaded (I'll have 10 apps trying to use only 8 CPUs; 2 GPUGrid instances, 6 CPU apps, and 2 HCC1 instances). When this happens, guess what happens? The HCC1 instances, because of only having 1 as their thread priority, end up getting virtually NO CPU. And then the GPU Load goes down, and task performance suffers. I propose that you do what GPUGrid appears to have done. Namely these 2 things: 1) always put your process priority to 6 (instead of only on fractional GPU usage, since this is a GPU app) 2) always put your thread priority to 6 (to ensure that, when the system is overloaded, the GPU does not get starved) Can you please look into doing this? Thanks! Jacob FWIW, running 2 GPUGrid task on the same GPU will gain you pretty much nothing because, unless things have changed since I last ran those, 1 task pretty much maxes out your GPU. Second, GPUGrid tasks run on Nvidias' CUDA platform, which requires very little CPU power. The WCG GPU app runs on OpenCl which requires more of the CPU to be used because some the the functions that are required to run the app cannot be ported to the GPU.
In 1969 I took an oath to defend and protect the U S Constitution against all enemies, both foreign and Domestic. There was no expiration date.
  |
||
|
|
Jacob Klein
Cruncher Joined: May 31, 2007 Post Count: 28 Status: Offline Project Badges:
|
nanoprobe:
----------------------------------------You're free to do your own testing. I have here: http://www.gpugrid.net/forum_thread.php?id=3331 My results seem to indicate an increase of performance, when 2 GPUGrid tasks are running on the same GPU. Regards, Jacob [Edit 1 times, last edit by Jacob Klein at Apr 5, 2013 4:03:38 AM] |
||
|
|
Jacob Klein
Cruncher Joined: May 31, 2007 Post Count: 28 Status: Offline Project Badges:
|
What settings are you using for GPUGrid and what was the motiviation for those settings? I want to understand the whole scenario. knreed, Some of the GPUGrid tasks run a bit differently, on different GPU cards. So, for instance, if I get a "Long run" unit that runs on my GTX 660 Ti, because of the GPU architecture, they have the process utilize a full CPU core for that task. Yet, for that same application, if that same task runs on my GTX 460, because of that GPU architecture, the CPU process hardly uses any CPU at all (next to nothing!) For their short-run tasks, those tasks appear to use some of the CPU; not all, but not negligible. So, I believe they made the decision that they don't want to instruct BOINC to reserve CPU cores, yet they do want CPU to be readily available to feed any GPU task that needs it. I believe they accomplish this by setting their process priority higher-than-low and their CPU-instensive-thread priority at higher-than-low. Specifically, if Lowest=1, Low=4, then they choose 6, which I think is termed Below Normal. The end result is that the GPU is kept busy, because it is constantly fed, even if the number of CPU tasks saturates/exceeds all CPUs. The GPUGrid processes don't get "scheduled out", because of their higher priority. You can add their project to BOINC, and use ProcessExplorer to see Windows keep those processes in the CPU schedule. The WCG HCC tasks, on the other hand, appear to have the CPU-intensive-thread set at 1 (which is the same as most CPU tasks, though I'm positive I've seen some CPU tasks running at priority 4.), and so, even if I have my WCG HCC app_config set to <cpu_usage>1</cpu_usage>, ... Because I have GPUGrid tasks in the mix, the HCC tasks are competing against the CPU apps, and getting scheduled out, and then the GPU gets idle/useless. The scenario I'm looking to improve is: - 8 CPU machine - 1 GTX 660 Ti (running 2 GPUGrid tasks at 0.001 CPU 0.5 GPU, with thread priority usually 6 or 7) - 1 GTX 460 (running 2 HCC tasks at 1 CPU 0.5 GPU, with thread priority 1) - 6 CPU tasks (with thread priority usually 1, but sometimes 4) I'm asking you to change this, to increase the priority of that CPU-intensive-thread to **6**, so as to ensure the GPU can be kept busy. Will you please consider making this change? Thanks, Jacob [Edit 4 times, last edit by Jacob Klein at Apr 5, 2013 4:18:18 AM] |
||
|
|
nanoprobe
Master Cruncher Classified Joined: Aug 29, 2008 Post Count: 2998 Status: Offline Project Badges:
|
Nvidia driver support for OpenCl is horrible at best so increasing the thread priority will probably not help and with less than 30 days of GPU crunching left on the HCC project I doubt the techs will try to change things now. JMHO
----------------------------------------
In 1969 I took an oath to defend and protect the U S Constitution against all enemies, both foreign and Domestic. There was no expiration date.
|
||
|
|
nanoprobe
Master Cruncher Classified Joined: Aug 29, 2008 Post Count: 2998 Status: Offline Project Badges:
|
nanoprobe: You're free to do your own testing. I have here: http://www.gpugrid.net/forum_thread.php?id=3331 My results seem to indicate an increase of performance, when 2 GPUGrid tasks are running on the same GPU. Regards, Jacob Interesting. Like I said, I haven't run GPUGrid in a while.
In 1969 I took an oath to defend and protect the U S Constitution against all enemies, both foreign and Domestic. There was no expiration date.
|
||
|
|
Jacob Klein
Cruncher Joined: May 31, 2007 Post Count: 28 Status: Offline Project Badges:
|
knreed:
----------------------------------------I understand there isn't much HCC GPU work left, but... if you guys decide to add any more batches to HCC GPU, or you bring in another GPU project, would you please consider doing some testing with priorities, to confirm my results, and to make the change so we can get more work done? Your users would greatly appreciate it. nanoprobe: I know what I'm seeing, and the WCG HCC GPU's CPU processes are being context-switched out. I believe using a process priority of 6, and a CPU-intensive-thread priority of 6 or 7, will help to keep the GPU busy. You sure do like that reply button. [Edit 1 times, last edit by Jacob Klein at Apr 7, 2013 4:09:33 AM] |
||
|
|
nanoprobe
Master Cruncher Classified Joined: Aug 29, 2008 Post Count: 2998 Status: Offline Project Badges:
|
It has nothing to do with "liking" the reply button. I'm just trying to point out that Nvidia has horrible OpenCl driver support and until they decide to do something about that you'll never get constant full GPU load on any project that runs on the OpenCl platform. You can't compare the results you get at GPUGrid with what you get here. They will never come close to being the same on Nvidia cards for now no matter where the thread priority is set.
----------------------------------------
In 1969 I took an oath to defend and protect the U S Constitution against all enemies, both foreign and Domestic. There was no expiration date.
----------------------------------------[Edit 2 times, last edit by nanoprobe at Apr 7, 2013 12:12:30 PM] |
||
|
|
Jacob Klein
Cruncher Joined: May 31, 2007 Post Count: 28 Status: Offline Project Badges:
|
Nanoprobe:
----------------------------------------Most of the time, my system's CPUs are over-loaded with CPU tasks (which happens when one of the GPUs works on tasks that take fractional CPU). My testing of the World Community Grid's Help Conquer Cancer GPU app, running 2 tasks at the same time on my GTX 460, indicates the following: - On my over-loaded system, HCC GPU tasks on the GTX 460 result in GPU Load that fluctuates between 82%-98%, as the HCC GPU tasks are swapped in/out of the CPU queue. Yet the GPUGrid tasks on the other GPU, whose CPU portions run at the higher priorities, maintain a constant GPU Load of 99%. - When the system is not over-loaded (ie: I set BOINC to free 2 CPUs), the GTX 460's GPU Load stays steady at 97%-98%. Constant GPU Load. - I submit these finding as evidence of my claim that a change in the process priority, combined with a change of the cpu-intensive-thread's priority, will probably make a difference in attaining a steady GPU load, and thus getting more science work done more efficiently. - I encourage you to do your own testing, and back up your claims of the contrary, with actual test data. knreed: Looking forward to your response. Thanks, Jacob [Edit 2 times, last edit by Jacob Klein at Apr 7, 2013 4:41:58 PM] |
||
|
|
|