Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 8
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 2952 times and has 7 replies Next Thread
SekeRob
Master Cruncher
Joined: Jan 7, 2013
Post Count: 2741
Status: Offline
Reply to this Post  Reply with Quote 
The API and the per-fetch limit of 250 records

Don't know why there's the 250 per-fetch limit, but with the migration to the highly performing CC (ClusterCloud ©™ :O), would WCG consider upping that limit to 500-1000, more? Instead of hammering your server, presently 89 times sequentially during 1 minute 39, which is what my stopwatch proc prints, maybe half the time. Certainly the turnover per second, you mentioned 50,000 per minute the system can handle, serious plans, implies the possibility, whilst I'm still managing to miss items, 20 this morning, on average about 28, which necessitates a second pass or third pass... everything keeps shifting up and down whilst the scheduler continues transacting at high pace. Ideally, dream phase, a snapshot of the whole account RS pages is taken, but fear that causes performance issues. biggrin

Thanks for considering.
[Jul 24, 2017 9:08:45 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: The API and the per-fetch limit of 250 records

Thinking about the matter, the 250 per-fetch limit might perhaps be somewhat antique in computer years. Several of my computers — while subscribed to all WCG-projects — run with a 0.2 day buffer with an additional 2 day buffer and this quadcore has about 75 results available, while the octacore has about 325, which is more than 250, the predefined API-fetchsizelimit. With the number of cores, memory size and overall speed of computers increasing over time and the boundary of the maximum number of tasks in the buffer per core, formerly 35, that was doubled to 70 recently, it sounds like it would be justified to increase the 250 per-fetch limit. Agreeing with SekeRob here.
[Jul 24, 2017 9:09:44 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7846
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: The API and the per-fetch limit of 250 records

You two guys are talking about two different kind of fetches. Sekerob is talking about the API fetch limit of results and Adriverhof is talking about work unit fetches for the machine to crunch. Correct me if I am wrong.
Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Jul 24, 2017 9:25:46 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: The API and the per-fetch limit of 250 records

Sorry to say, Sgt. Joe, I was using wordings such as "250 per-fetch limit" (using the API) and "predefined API-fetchsizelimit", not in any way referring to fetching workunits, since that is a different chapter.

One can use the API to fetch 'results' that are still In Progress. For a device that has more than 250 tasks in progress, you'd need at least two fetches through the API, that's what I somehow was pointing at by the phrase "while the octacore has about 325, which is more than 250, the predefined API-fetchsizelimit".
[Jul 24, 2017 11:35:50 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7846
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: The API and the per-fetch limit of 250 records

Adriverhof:
I was confused by your wording. Yes, I realize you can query the API for the "in progress, in addition or in combination with additional statuses (stati ?). Thank you for the clarification.
Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Jul 25, 2017 1:29:28 AM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: The API and the per-fetch limit of 250 records

That change is a reasonable and easy to implement request. We will make it as part of the next website update (not sure when that will be - sometime in the next couple of weeks most likely).
[Jul 26, 2017 1:53:05 PM]   Link   Report threatening or abusive post: please login first  Go to top 
SekeRob
Master Cruncher
Joined: Jan 7, 2013
Post Count: 2741
Status: Offline
Reply to this Post  Reply with Quote 
Re: The API and the per-fetch limit of 250 records

Nice, looking forward, the noon sequential hit counted through to 296*250 of ServerState=5, but that was a cycle without the FileDeleteState filter (and a bulk freighter load of short FAHV)... it resolved 30 of the 35 missing over the last 4 half day statistic periods... analysis outstanding as to why they got missed with FDS=0 applied, but considering there's 43200 seconds in half a day and near 50K new valid for the 12 hours, the 7:32 minutes it took to complete the last fetch rounds had 500+ added at the top while getting later/older pages.

Thanks

PS: for comparison, with FDS=0 filter needed 93x250 to come across.
[Jul 26, 2017 2:30:58 PM]   Link   Report threatening or abusive post: please login first  Go to top 
SekeRob
Master Cruncher
Joined: Jan 7, 2013
Post Count: 2741
Status: Offline
Reply to this Post  Reply with Quote 
Re: The API and the per-fetch limit of 250 records

Kevin,

Did the launch of MIP1 comprise a new website update? Only asking, eager to start the 'less is more' conversion.

(tried with &limit=500 and got the old 250 max back)

thx
----------------------------------------
[Edit 1 times, last edit by SekeRob* at Aug 24, 2017 3:50:18 PM]
[Aug 24, 2017 3:46:56 PM]   Link   Report threatening or abusive post: please login first  Go to top 
[ Jump to Last Post ]
Post new Thread