| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
| World Community Grid Forums
|
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 8
|
|
| Author |
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
Don't know why there's the 250 per-fetch limit, but with the migration to the highly performing CC (ClusterCloud ©™ :O), would WCG consider upping that limit to 500-1000, more? Instead of hammering your server, presently 89 times sequentially during 1 minute 39, which is what my stopwatch proc prints, maybe half the time. Certainly the turnover per second, you mentioned 50,000 per minute the system can handle, serious plans, implies the possibility, whilst I'm still managing to miss items, 20 this morning, on average about 28, which necessitates a second pass or third pass... everything keeps shifting up and down whilst the scheduler continues transacting at high pace. Ideally, dream phase, a snapshot of the whole account RS pages is taken, but fear that causes performance issues.
Thanks for considering. |
||
|
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2346 Status: Offline Project Badges:
|
Thinking about the matter, the 250 per-fetch limit might perhaps be somewhat antique in computer years. Several of my computers — while subscribed to all WCG-projects — run with a 0.2 day buffer with an additional 2 day buffer and this quadcore has about 75 results available, while the octacore has about 325, which is more than 250, the predefined API-fetchsizelimit. With the number of cores, memory size and overall speed of computers increasing over time and the boundary of the maximum number of tasks in the buffer per core, formerly 35, that was doubled to 70 recently, it sounds like it would be justified to increase the 250 per-fetch limit. Agreeing with SekeRob here.
|
||
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7846 Status: Offline Project Badges:
|
You two guys are talking about two different kind of fetches. Sekerob is talking about the API fetch limit of results and Adriverhof is talking about work unit fetches for the machine to crunch. Correct me if I am wrong.
----------------------------------------Cheers
Sgt. Joe
*Minnesota Crunchers* |
||
|
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2346 Status: Offline Project Badges:
|
Sorry to say, Sgt. Joe, I was using wordings such as "250 per-fetch limit" (using the API) and "predefined API-fetchsizelimit", not in any way referring to fetching workunits, since that is a different chapter.
One can use the API to fetch 'results' that are still In Progress. For a device that has more than 250 tasks in progress, you'd need at least two fetches through the API, that's what I somehow was pointing at by the phrase "while the octacore has about 325, which is more than 250, the predefined API-fetchsizelimit". |
||
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7846 Status: Offline Project Badges:
|
Adriverhof:
----------------------------------------I was confused by your wording. Yes, I realize you can query the API for the "in progress, in addition or in combination with additional statuses (stati ?). Thank you for the clarification. Cheers
Sgt. Joe
*Minnesota Crunchers* |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
That change is a reasonable and easy to implement request. We will make it as part of the next website update (not sure when that will be - sometime in the next couple of weeks most likely).
|
||
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
Nice, looking forward, the noon sequential hit counted through to 296*250 of ServerState=5, but that was a cycle without the FileDeleteState filter (and a bulk freighter load of short FAHV)... it resolved 30 of the 35 missing over the last 4 half day statistic periods... analysis outstanding as to why they got missed with FDS=0 applied, but considering there's 43200 seconds in half a day and near 50K new valid for the 12 hours, the 7:32 minutes it took to complete the last fetch rounds had 500+ added at the top while getting later/older pages.
Thanks PS: for comparison, with FDS=0 filter needed 93x250 to come across. |
||
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
Kevin,
----------------------------------------Did the launch of MIP1 comprise a new website update? Only asking, eager to start the 'less is more' conversion. (tried with &limit=500 and got the old 250 max back) thx [Edit 1 times, last edit by SekeRob* at Aug 24, 2017 3:50:18 PM] |
||
|
|
|