Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 29
Posts: 29   Pages: 3   [ Previous Page | 1 2 3 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 4919 times and has 28 replies Next Thread
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

Hello retsof,
HDC spikes up to 750 Mb

Rick Alther said that HDC occasionally spiked up to a bit more than 500 MB. So 750 MB should allow a normal XP system to run light programs such as a browser even while the HDC program is spiking without excessive paging. Of course, if you run hyperthreading and use BOINC to allow 2 threads to run, then you should have at least 1.25 GB to avoid excess paging.

Lawrence
[Jul 24, 2006 1:40:29 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Alther
Former World Community Grid Tech
United States of America
Joined: Sep 30, 2004
Post Count: 414
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

Since I've been receiving work units for Help Defeat Cancer, my PC goes nuts every 15 minutes (more or less). Something is writing or reading from disk to such an extent that all other tasks slowdown. Since this PC is my media PC, this is a problem. Is there a workaround for this?

What you are experiencing is the UD Agent checkpointing a large file. Help Defeat Cancer can write some very large files (130MB or more). This causes a problem with UD Agents because when we checkpoint with UD, it compresses and stream encrypts the files to be checkpointed to backup files. This causes the I/O to take a relatively long time to complete (it's about 7-9 times slower than straight I/O). This is the I/O lag you are seeing.

Knowing this, we "throttle" the I/O for the program. Instead of writing a lot of data at once, we write out the data in chunks and pause slightly between the chunks so that our application doesn't impact the user's machine with I/O. This also saves some memory. This makes a much larger difference in UD where all I/O is stream encrypted. While this I/O "throttle" (not to be confused in any way with the CPU throttle) certainly helps, there is nothing we can do when we checkpoint and UD backs the files up. That code is out of our hands.

What happens under the covers is this:
* The science app writes out files as it progresses through the code. All I/O in UD is stream encrypted and thus slow. The I/O read/written by the science app is "throttled" as described above.
* We reach a point in the code where we can checkpoint. For UD, we call their checkpoint routine. All it basically does is look at the list of files that are considered checkpoint files and backs them up. This is where you are seeing the I/O hit. For BOINC, we call our own checkpoint class which backs up the files for us. Under BOINC this is very quick compared to UD, so it should barely be noticable. After the files are copied and the state updated, we tell BOINC we've completed our checkpoint.

This answer is much longer than I anticpiated it would be, so I hope it's clear.
----------------------------------------
Rick Alther
Former World Community Grid Developer
[Jul 25, 2006 5:14:13 AM]   Link   Report threatening or abusive post: please login first  Go to top 
BlackSmurf68
Cruncher
Joined: Nov 18, 2004
Post Count: 4
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

Defrag didn't help. The pagefile is already in one fragment.
I'm running UD client and BeyondTV 4.3 only. Sometimes I pop open Firefox to do a little surfing. I noticed that when the disk goes nuts, the CPU usage drops from 60% to 20%. The pagefile used stays the same (around 630MB). I have a 7600gt gfx card so there are no shared memory issues.

Is it possible to store the checkpoint files in RAM or on a flash disk? If I could configure UD to store on another drive, that would probably fix my problem.
[Jul 26, 2006 1:44:55 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

i hope it's not one of those 'anti-virus' problems like with mcafee.

that virus scanner used to conflict with WCG a lot. i only found out about it by opening task manager and figuring out what else was taking CPU power.
[Jul 26, 2006 2:34:06 AM]   Link   Report threatening or abusive post: please login first  Go to top 
retsof
Former Community Advisor
USA
Joined: Jul 31, 2005
Post Count: 6824
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

Is it possible to store the checkpoint files in RAM or on a flash disk? If I could configure UD to store on another drive, that would probably fix my problem.
The most gain will be made by putting the Windows swap file on a separate drive from the UD files. That topic comes up now and then.

It seemed to be possible to run the UD interface on a flash drive and even swap it from computer to computer.

There's a "sneakernet" topic
on http://forum.grid.org concerning moving files from to the hard drive on one computer to the hard drive on another, advantageous when only one computer has a connection to the internet, and several others do not, but could be used for crunching. It works easiest when you have UDMonitor to manage the multiple slots on the internet computer.

I tried it with BOINC but it was bad news. It was more closely tied to the timestamp in the registry on one computer, and things became corrupted when it was on the other. It couldn't quite understand why I couldn't finish the workunit by a deadline of 1900.
----------------------------------------
SUPPORT ADVISOR
Work+GPU i7 8700 12threads
School i7 4770 8threads
Default+GPU Ryzen 7 3700X 16threads
Ryzen 7 3800X 16 threads
Ryzen 9 3900X 24threads
Home i7 3540M 4threads50%
----------------------------------------
[Edit 3 times, last edit by retsof at Jul 26, 2006 12:54:12 PM]
[Jul 26, 2006 12:44:13 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

Retsof, when desperate one looks at anything.....the timestamp point very very valid, to the extend that i disabled the WinXp clock sync function as it corrupted twice a WU in BOINC (latest official 5.4.9).

In UD that behaviour might be a bit more friendly...restart the WU, but could the clock synching be a factor?

The 33.33% is an observed one. Similar to HPF2, these present batches of FAAH are split in very large segments. Put a post out, seeing types existing of 6, 3, 2 and one resetting for even 90 to zero, suggesting a no segment at all ????? What ever is said about the frequent checkpointing, i wait with closing until it passes these percent hurdles just to be totally save......nothing so annoying than loosing 4 / 5 or more hours of crunch.

The Porting to a different computer...zip....copy....move pc ..unzip been successfully deployed by DPC members a few times without hearing of troubles.....lucked out.

Swapfile on a 2nd drive helped total system performance quite a bit. Also it sits in its private partion there of 2gb, so never has issues of fragmentation when on autosizing. PageDefrag it once a while, but usually it says its already in 1 space.
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Jul 26, 2006 1:12:04 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

It may or may not help, but in the WCG BOINC profile there is an option to save your work every 60 second (default). I've not figured the function, since the checkpoints of some project Work Units is sometimes hours apart and a restart would throw u back to the last checkpoint saved. Anyway, subjectively, things improved on my machine, setting that default to 600 seconds.
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
[Aug 2, 2006 7:21:17 PM]   Link   Report threatening or abusive post: please login first  Go to top 
AnimeStore.pl
Cruncher
Joined: Dec 26, 2004
Post Count: 2
Status: Offline
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

HDC seems to send the disk into a frenzy every 15 minutes or so and I can't understand why. For now, I'll just turn off HDC unless/until I hear the issue has been solved.

Same here. It's impossible to work on my computer with something like that. I'm ignoring this project. sad
[Aug 31, 2006 7:36:44 AM]   Link   Report threatening or abusive post: please login first  Go to top 
BlackSmurf68
Cruncher
Joined: Nov 18, 2004
Post Count: 4
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

The machine I'm running this on is my media PC. I still wanted to run WCG because contributing to help defeat diseases is important to me. I decided to buy a 1GB flash card and install the software there thinking that would resolve the disk "freeze" issue. The flash card is interfaced to the IDE bus via a card reader. I then tried another HDC project. I still have a problem. All other tasks are frozen for a split second every so often. If the computer is running a game, for example, the game will "hiccup" about every 5 seconds.

I'm turning off this project since I'm out of ideas on how to resolve this issue. I don't have any problem with other projects.
[Sep 1, 2006 2:36:51 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Alther
Former World Community Grid Tech
United States of America
Joined: Sep 30, 2004
Post Count: 414
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: CPU/Disk Load to 100% every 15 minutes

ALL: Please read my earlier post on 25 July. It appears no one has read it. The problem with the I/O hit should only be happening with the UD Agent, as I described in my earlier post. If the I/O is affecting you, I would recommend running the BOINC client (at least for the HDC project), which doesn't have the I/O overhead that UD does.
----------------------------------------
Rick Alther
Former World Community Grid Developer
[Sep 1, 2006 2:01:05 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 29   Pages: 3   [ Previous Page | 1 2 3 | Next Page ]
[ Jump to Last Post ]
Post new Thread