Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 20
Posts: 20   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 2553 times and has 19 replies Next Thread
Dirk Gently
Senior Cruncher
England
Joined: Mar 1, 2005
Post Count: 153
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Variation in BOINC Benchmark score

This is NOT a question about the fairness of BOINC points allocation!

I have just done a processor upgrade. Boinc saw this and ran benchmarks. The result was abysmal - worse than old procerssor! I naturally ran the benchmarks again. What I noticed was that the benchmark score- particularly for Integer - varied wildly with succesive runs. I think that the benchmark runs on the same very low priority that Boinc itself runs on. It is therefore easily interrupted by other running processes. I paused my "on access" virus scanning, and the benchmark score was consistently higher.

The questions -

1) Boinc stores the latest benchmark figure in the file "Client_State.XML". Not sure, but I think Boinc uses these figures together with the Processing time of a WU to calculate the number of points allocated. Is this true, and is it therefore worth ensuring that the benchmark that is in use is the best that can be obtained (by repeated runs of Benchmarks with virus scanning etc disabled)?

2) Does Boinc periodically re run the benchmarks automatically? So if these figures are used for points calc, is it possible to be penalising oneself with a poor score without realising it?
----------------------------------------
----------------------------------------
[Edit 1 times, last edit by Dirk Gently at Feb 18, 2007 2:56:19 PM]
[Feb 18, 2007 2:54:35 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

This variation has been noticed and the developers are working on it. They have made some improvements in the 5.8.11 and later clients. Allthough it still is not as good as I would like.

The benchmarks use CPU time when figuring the results so whatever else is running normally makes little difference (win 9x excluded).

The benchmarks rerun automatically every 5 days.
[Feb 18, 2007 4:08:36 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sekerob
Ace Cruncher
Joined: Jul 24, 2005
Post Count: 20043
Status: Offline
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

Hi Dirk,

It's not clear why the benchmark was made Susceptable to influences of other processes running at Normal Priority, which is what the BOINC.exe benchmark runs at (see Taskmanager). To me given that credit is computed on CPU time and not even includes the slowing due i.e. science related disk i/o, the 'computational power' captured in the benchmark should be standard at a optimal level and not at a level impacted by everything else going on.... that's my opinion.

Here's a link from the Help section explaining things: http://www.worldcommunitygrid.org/help/viewSearch.do?searchString=benchmark
It includes some link through revealing that the foundation is in part still based on SETI logic How benchmarks are calculated . What Keck said on the redo of the benchmark is new to me (see nothing in the 2007 CVS log to date)....can't say 5.8.11 or 5.8.12 done anything for me (5.8.14 test version is out). Was only aware of it being reworked so that linux would get a matching value to windows.... an old sore i think more or less resolved.

BOINCview actually tells u in one of the tabs for all the computers you've linked, when the next benchmark is due. Get up middle of the night one time, force the benchmark and it will always run that time of the night... until one day you did not have the machine running at night ;O
----------------------------------------
WCG Global & Research > Make Proposal Help: Start Here!
Please help to make the Forums an enjoyable experience for All!
----------------------------------------
[Edit 3 times, last edit by Sekerob at Feb 19, 2007 11:14:49 AM]
[Feb 18, 2007 4:42:25 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Diana G.
Master Cruncher
Joined: Apr 6, 2005
Post Count: 3003
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

Hi. I noticed that my benchmark's run at exactly the same time as when it first ran 6:15 AM when I upgraded to 5.8.8. Yup every 5 days exactly at 6:15 AM.

Diana G.
----------------------------------------

[Feb 19, 2007 5:20:48 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

The questions -

1) Boinc stores the latest benchmark figure in the file "Client_State.XML". Not sure, but I think Boinc uses these figures together with the Processing time of a WU to calculate the number of points allocated. Is this true, and is it therefore worth ensuring that the benchmark that is in use is the best that can be obtained (by repeated runs of Benchmarks with virus scanning etc disabled)?


That figure and the WU crunch time are used to calculate your credit claim which may turn out to the amount awarded or allocated. You are essentially correct though I wouldn't use the word allocated where you did, that's all.

You are also correct about ensuring the benchmark in use is the best that can be obtained, however it isn't necessary to run it repeatedly. More on that below.


2) Does Boinc periodically re run the benchmarks automatically? So if these figures are used for points calc, is it possible to be penalising oneself with a poor score without realising it?


The benchmarks do rerun automatically and you can penalise yourself if your benchmarks are not what they ought to be. That sword has 2 edges and cuts both ways. You get penalised if they're too high as well as if too low.

Since WCG insists on sticking with the broke-the-day-it-was-implemented-and-never-will-work purple dinosaur benchmark based credit system and the BOINC devs will never really fix the problem, you need to McGyver it. Here's how...

1. Set all projects to "no new WUs".

2. Drain your WU cache of all WUs except for 1 WU for each project you crunch. To do that, suspend 1 WU from each project and allow the rest to crunch, upload and report.

3. Shutdown BOINC and backup your client_state.xml file in the BOINC directory.

4. Open client_state.xml file in Notepad. Use Notepad or some other plain text editor, not Wordpad or Word because those editors try to put text formating commands into the file you are editing and you definitely don't want that in your client_state.xml.

5. About 10 lines from the top find lines that start with:

  • <p_fpops> - this is the floating point benchmark
  • <p_iops> - this is the integer benchmark
  • <p_calculated> - this tells BOINC when to re-calculate the benchmarks


6. Alter the numbers following <p_fpops> and <p_iops> to whatever they should be for your CPU. Sorry, I don't have a clue what the numbers should be for your CPU because I don't even know what CPU you use. You'll have to find that info on your own but it won't be too hard if you ask around in various BOINC related forums.

7. Alter the number following <p_calculated>. Initially, it will be something like 1171776064.343750 which tells BOINC to re-calculate the benchmarks every 5 days. Increase the 3rd digit by 1 to tell BOINC to re-calculate something like every 30 years. OK, it's not exactly 30 years but it's a very long time and your benchmarks will stay the way you set them until that time.

8. Now check your typing carefully. When you're satisfied, save the file and exit Notepad.

9. Now you must let the remaining WUs in your cache crunch before you allow more new WUs to download or else BOINC will download too many. Actually, if you blow this part it's not a big deal, you can just abort whatever you know you can't finish in time.

BEWARES and GOTCHAS:

  • I think maybe each number you edit must be followed by 6 decimal digits but i'm not sure, play it safe and make sure there are exactly 6 digits to the right of the decimal points in the 3 numbers you edit.
  • If you make the <p_iops> and <p_fpops> numbers too high then your credit claims will be classified as extreme outliers by WCG's validator and you will be penalised. Do take the time to find out what those numbers should be and don't cheat too much or you'll burn forever in Outlier Hell ;-)

----------------------------------------
[Edit 1 times, last edit by Former Member at Feb 19, 2007 11:26:50 AM]
[Feb 19, 2007 11:24:50 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

Ooops! There are a few ambiguities in my previous post.

7. Alter the number following <p_calculated>. Initially, it will be something like 1171776064.343750 which tells BOINC to re-calculate the benchmarks every 5 days. Increase the 3rd digit by 1 to tell BOINC to re-calculate something like every 30 years. OK, it's not exactly 30 years but it's a very long time and your benchmarks will stay the way you set them until that time.


Increase the 3rd digit from the left. The 1171776064.343750 given as example would become 1181776064.343750

BEWARES and GOTCHAS:

  • I think maybe each number you edit must be followed by 6 decimal digits but i'm not sure, play it safe and make sure there are exactly 6 digits to the right of the decimal points in the 3 numbers you edit.


For example, 235.49 may not be acceptable to BOINC because it wants the number to have 6 decimal digits for some strange reason. At least that's what some people have told me. 235.123456 for example, is acceptable.
[Feb 19, 2007 3:11:46 PM]   Link   Report threatening or abusive post: please login first  Go to top 
David_L6
Senior Cruncher
USA
Joined: Aug 24, 2006
Post Count: 296
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

Would it be OK to just change the <p_calculated> part so that it doesn't run benchmarks every 5 days?

I have 3 machines with Intel P4's with HyperThreading. You probably already know what kind of problems they cause.....
----------------------------------------

[Feb 19, 2007 6:39:57 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

Would it be OK to just change the <p_calculated> part so that it doesn't run benchmarks every 5 days?

I have 3 machines with Intel P4's with HyperThreading. You probably already know what kind of problems they cause.....

Adjusting only <p_calculated> won't cause any problems. I am not sure it will cure the problems you have in mind. I don't have any CPUs with HyperThreading so I don't know what problems you might be looking to fix.
[Feb 19, 2007 7:10:00 PM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

We strongly discourage you from modifying and adjusting the results of the BOINC benchmarks. Although the benchmarks are not perfect, they are significantly better than everyone choosing their own particular performance number and using that.

The long term improvement to the benchmarks /credit scoring system is going to becoming in the future - possibly the 5.12 BOINC client (the 5.10 client will be a relatively small release that largely only includes an auto-update feature). The general gist of the change is that instead of just running a general benchmark - everyone will actually run a short workunit using the actual science application. Thus your computer will be measured on how long it takes for you to run the actual science project - which is the most accurate way to measure your computers performance for running that particular science project.

Also - BOINC is open source. The above feature is on the schedule to be implemented - but if you want it done sooner - than dig in! Look at the code, figure out how it works, start working on the change. Check out: http://boinc.berkeley.edu/boinc_dev.php to learn more.
----------------------------------------
[Edit 1 times, last edit by knreed at Feb 20, 2007 2:41:08 AM]
[Feb 20, 2007 2:40:15 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Variation in BOINC Benchmark score

We strongly discourage you from modifying and adjusting the results of the BOINC benchmarks. Although the benchmarks are not perfect, they are significantly better than everyone choosing their own particular performance number and using that.


The projects, including WCG, have created the current situation by building and encouraging a system driven by worthless credits instead of a system driven by brotherly love and the desire to make the world a better place. Don't chastise people now for embracing the monster you've created.

The long term improvement to the benchmarks /credit scoring system is going to becoming in the future - possibly the 5.12 BOINC client (the 5.10 client will be a relatively small release that largely only includes an auto-update feature). The general gist of the change is that instead of just running a general benchmark - everyone will actually run a short workunit using the actual science application. Thus your computer will be measured on how long it takes for you to run the actual science project - which is the most accurate way to measure your computers performance for running that particular science project.


Within 10 days of the release of that "upgrade" somebody will break it. Store the test result on disk? They'll find it and mod it. Send the test result immediately and not store on disk? They'll intercept the transmission, mod the number and send their own result. Encrypt the number? They'll break it because they'll have lots of clues into the encryption. Encryption is strong only when you have no clues. The more clues you gather the easier the encryption breaks. So just think of how many clues a team of 20 can find with a farm of 60 C2Ds and 10 days of crunching. You don't stand a chance, lol.

Also - BOINC is open source. The above feature is on the schedule to be implemented - but if you want it done sooner - than dig in! Look at the code, figure out how it works, start working on the change. Check out: http://boinc.berkeley.edu/boinc_dev.php to learn more.


Why would anyone work 8 hours a day at their job and then come home and help build yet another flawed credit system that will be cheated? Just for the fun of seeing new flamewars and knowing they had a hand in starting the wars?

If the project can build a test WU and assure us that the test unit is representative of all the project's WUs (you said yourself that is the plan) then the project surely has suffucient knowledge of their WUs to just fix the credits on the server and never allow crunchers to evaluate themselves. Self-evaluation will always be cheated and there is a powerful motive to cheat... it brings enormous attention and respect from the community when you can outfox the dummies gurus who built the system.

And why would I want to waste my time developing the next broke system when WCG pees away the time of paid employees to write something as useless as the simple GUI? Ya should have been fixing the credits and writing better docs for the standard GUI instead of fixing something that ain't broke.
.
[Feb 20, 2007 4:33:58 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 20   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread