Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 45
Posts: 45   Pages: 5   [ Previous Page | 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 9078 times and has 44 replies Next Thread
cjslman
Master Cruncher
Mexico
Joined: Nov 23, 2004
Post Count: 2082
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

Why just "spare" computing time??? That's a complete fallacy.
Not sure what you mean by "fallacy" ? I would like to understand your comment before commenting myself. As for the "Anonymous zero results entries" I imagine those are persons that requested that their names/results not be published (under GDPR rules).

Thanks,
CJSL

Crunching on the thin ice of a new day...
----------------------------------------
I follow the Gimli philosophy: "Keep breathing. That's the key. Breathe."
Join The Cahuamos Team


[Feb 27, 2020 10:35:33 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

Finally, I understand the frustration in the community about throttling and whatnot. But give this stuff some time to work itself out. Even minor issues get bogged down in bureaucracy when you have a giant mutinational corp on one end, and an academic institution at the other. Both have several layers of systems for decisions to go through before they get approved. Gotta understand that it may take some time to get everything approved and implemented. Where funding is an issue, things get ten times more complicated, too. Things will also generally be slower right now due to health scares going through several countries. If we were at the 6 month or 1 year mark, I'd get it. But when a brand new project is barely two months in? Let's just say I'm not surprised biggrin

They had almost 2 years prior to launch to work this out. ~36000 work units per iteration at 60MB per work unit would be about 2.1TB per iteration. Then there is about 182.5 iterations(let's use 183) so, 183 time 2.1TB is ~395TB but then there is minimum quorum of 2 so that 395TB becomes 790TB. If I look around the room and don't see 790TB of storage there, I don't launch. I assume what happened though, is the techs said "Wait, we don't need all that because we will transmit data to the researchers at the end of each iteration.." That's when there should have been a discussion with the research team, yes, prior to launch saying we are going to be sending you 2.1 TB (or 4.2TB if quorum of 2) approximately every 3 weeks. Do you have a place for this data and oh, BTW this project is going to generate about 1 PB of data. Do you have the resources? If the answer is NO, don't launch. Do the researchers need to second copy of the results? If the work unit validated, aren't the results identical copies? If the researchers say they want both for comparison purposes, then fine but they have to have a place to put it. The fact that this project just started and is throttled just screams out "DON"T LAUNCH...". If it would have come up after a year of processing one could reasonably argue that more data was generated than what was anticipated and we need to adjust, so we adjust. So, in contrast to your statement, I am surprised and I have been since the first month of launch. This just seems like project management 101
[Feb 27, 2020 11:50:25 PM]   Link   Report threatening or abusive post: please login first  Go to top 
UBT - JohnR
Cruncher
Joined: Apr 30, 2006
Post Count: 35
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

Let's get real
Linus runs a youtube tech channel, one man with a few employees.
If he can do this - https://www.youtube.com/watch?v=EtZXMj_gUjU

Then WTF. I just spent $90 on a 6TB drive.
[Feb 28, 2020 11:29:29 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

"Do the researchers need to second copy of the results?"

Also know as the canonical copy. If there is bit by bit matching then that would be enough to retain and transmit.
[Feb 28, 2020 12:40:28 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Mike.Gibson
Ace Cruncher
England
Joined: Aug 23, 2007
Post Count: 12594
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

Slightly off topic, but my 8 threads, restricted to 4 arp and 4 mcm/mip because of crunching capacity problems, are being kept fully occupied at the current 4000 units per day. Plus i have a variable cache of between 1 and 4 (maximum to keep my machine 'reliable'.

I am quite happy with the current supply.

Mike
[Feb 28, 2020 1:20:03 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

"Do the researchers need to second copy of the results?"

Also know as the canonical copy. If there is bit by bit matching then that would be enough to retain and transmit.

But if one is in a storage/budget crunch, where do we trim? That seems like a good spot and can cut your requirements in half. Additionally, what does the extra copy really get you anyway? In the event that one needed to go back and re-compare results(which seems like the only reason to keep it), which is the correct copy? All you would know is that they are or aren't different. Also, if the BOINC system did a bit by bit compare to validate the WU, wouldn't they be identical? So why would you need to keep it? Also, in the rare case something seemed a little "off", it might be easier to just rerun the WU locally and be done with it rather than digging through a string of bits.
[Feb 28, 2020 2:45:45 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

If, as has happened in past where the validator screwed up, or rather the validation rules were messed up, the only science proper option is to rerun. Think we just had such an instance on a swath of batches of some other science. With bit by bit comparison, how messed could it ever get.

PS JIC I was not abundantly clear, keeping the canonical copy would be enough to transmit in the BtB case.
----------------------------------------
[Edit 1 times, last edit by Former Member at Feb 28, 2020 3:30:26 PM]
[Feb 28, 2020 3:28:30 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Dayle Diamond
Senior Cruncher
Joined: Jan 31, 2013
Post Count: 452
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

Just to add to what others are saying, a quick google search shows that in 2017, storing a petebyte on a server cost $10,000 a month. Let's make that two for $20,000.

That's a lot of electricity, but again, probably less then IBM spends on incidentals in a week.

You can save costs by unplugging backup drives once they fill, until or unless they are needed to remedy an error.

Just about everybody knows somebody who's gotten cancer, and due to the societal costs of treatment just about everybody would see themselves better off if chronic, debilitating, and expensive conditions like TB & HIV/AIDS were a thing of the past.

I don't judge people who pick other charitable causes, I absolutely take note the folks who give almost nothing. The ones who ask me how much BOINC will pay them and cannot comprehend a donation that does not somehow work its way back to themselves.

Millions of Africans aren't getting enough food. That's been true for what - most of our lifetimes? It's been so bad for so long that we're used to it. That caring 'too much' is seen as un-serious. "Let's do what we can to help, so long as it's minimal. Or comfortable. Or cost effective. And goodness, don't judge those radiant, positive people who are too busy enjoying our status quo to give much of a care to anybody else's. Their greed is innocent and pure, and precious and vulnerable and mustn't be confronted".
Edit: Clarity.
----------------------------------------
[Edit 2 times, last edit by Dayle Diamond at Feb 29, 2020 5:54:26 AM]
[Feb 29, 2020 5:46:03 AM]   Link   Report threatening or abusive post: please login first  Go to top 
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1684
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

Throughput Throttled Until a Storage Space Solution is Found

Dayle,
you put a statement without any reference resp. link.
I think that it is really disturbing, incl. the emoticon you add to the thread. I don't really know what is your objective.
Indeed an ARP is demanding project on both sides: WCG and scientists as well as on member side, in particular in terms of internet bandwidth.
We - WCG and members - do our best to support all projects, including ARP. Fighting against WCG/IBM is not very helpful.
Cheers,
Yves
----------------------------------------
[Feb 29, 2020 9:56:31 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Dayle Diamond
Senior Cruncher
Joined: Jan 31, 2013
Post Count: 452
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Throughput Throttled Until a Storage Space Solution is Found

I don't think you're personally disturbed by a lack of a reference link or an emoji in an internet post.

You may want to reexamine your feelings on that one on what's actually troubling you.

Internet bandwidth is not the stated reason for the throttling.
We're all free to ask IBM for more support.

At this point the Africa Rainfall Project is barely alive.
Ladies and Gentlemen, IBM can expand it.
They have the technology.
They have the capability to build a two petabyte storage solution.
Africa Rainfall Project will be that catalyst.
Better than it was before.
Better, stronger, faster.
[Mar 1, 2020 4:26:12 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 45   Pages: 5   [ Previous Page | 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread