| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Support
Forum: Website Support
Thread: API Does Not Return All Results Status Since Feb 28 Maint |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 11
|
|
| Author |
|
|
foxfire
Advanced Cruncher United States Joined: Sep 1, 2007 Post Count: 121 Status: Offline Project Badges: 
|
I am using the API "https://secure.worldcommunitygrid.org/api/members/{member name}/results?code={verification code}" to pull Results Status. Following the Feb 28 maint update it is returning erratic data in that between 1 and 10 WUs that are in Results Stats are not included in the returned data.
----------------------------------------If I rerun the API the missing WUs are returned and usually new ones are missing. I have tried pulling all and then pulling modified since the time of the previous pull and still have missing WUs. I usually pull the data 3 times a day, and WUs that were not reported on in the previous pull are now there and new ones are missing. Before pulling the data I turn network communication off so that no new WUs are pulled or completed ones reported. Any possibility this could be looked at and if posible fixed?  |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Been running this off and on, and used the sort feature to ensure the newest are at top, then set it to pull 250 and several more, skipping to the next 250 and the next 250. Here's an example for 251-500:
https://secure.worldcommunitygrid.org/api/mem...0&SortBy=ReceivedTime Can't say I'm missing anything [Oh do wish I could replicate the issue]. Ran the query in Excel, predefined to not show In Progress and only wanted the Android completed tasks. Have 14 on the RS pages, have 14 in the Excel sheet. Somewhere there's a request for some API expansion, so that e.g. In Progress can be excluded. |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
N.B. Long links are compressed in posts, so you need to right click on the sample, then select copy link location, then edit for your name/account key and test if this improves the situation. Unsorted, strikes me as too random a data order when pulling.
|
||
|
|
foxfire
Advanced Cruncher United States Joined: Sep 1, 2007 Post Count: 121 Status: Offline Project Badges:
|
Sort in SentTime and loop thru 250 at a time until number available is reached is what I do as well. I have been running between 2500 - 4500 WUs in Results Status, so it may be related to the number of results to pull.
----------------------------------------My guess is that if the Limit were raised to permit pulling all in one request the problem might go away. If you use the ServerState=, Outcome= and/or ValidateState= options you should be able to excluded In Progress. |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Beg your pardon, if you hover on the sample link given there's already a ServerState=5 included i.e. only those processed are being returned. Must be some other API extension... memory not IIRC ;>)
Yes, 4500 takes a while to pull. My filtered 288 is taking 15 seconds to come across with a 'Refresh All' i.e. the 250 query blocks run serial without my intervention. Doubtlessly the 250 limit was done for some performance reason... the RS pages hit on the life database. The rate of validation is today around 1.9 million or 21 per second. That would also be the rate at which they are cleared off to the master database and having a state change of some kind. Lots going on while viewing and querying. |
||
|
|
foxfire
Advanced Cruncher United States Joined: Sep 1, 2007 Post Count: 121 Status: Offline Project Badges:
|
Rob, thanks for the responses but your suggestions have already been tried and I understand that WUs can age out of Results Status during the pull.
----------------------------------------My logic is: Pull all RS to a file. Once the pull is done I format the records and add new and update existing records in a WU history file. Once done I check the WU History file for any WU that have a status of In Progress, Pending Val or Pending Var against the file of Results Status I just pulled. In other words I only check WUs that are "open". If any of these open WUs were not pulled I count them as missing. I've been using the API since it was released and prior to the 28th only 3 WUs came up missing and these were because they had cycled thru multiple wingmen and were more than 10 days old by the time they were completed. Because I only pull 2-3 times a day they had aged out before I ran the pull. Since the 28th I have between 1 - 10 on every pull. I have checked the web RS and they are in the list, just not sent in the pull. This AM I had one that was sent 15 minutes before my pull that wasn't returned in the pull. Because I turn off network communication before doing the pull I will not have any WUs sent/added and none returned which means the number of WUs will not change during the pull, just their status. It is a problem with the API and it started with the maintenance done on the 28th. |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
OK, well it's then up to the techs... maybe a stale index, one not rekeyed maybe when switching over to the new DB [hope that is the cause and not some bug in the API].
|
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
It's a pity the pirogue app went belly up at the time of forcing the use of xml, breaking the scraping method. That was a superb solution keeping a history and it updating records anytime they had a status change.
|
||
|
|
foxfire
Advanced Cruncher United States Joined: Sep 1, 2007 Post Count: 121 Status: Offline Project Badges:
|
After writing all this I think I understand what is happening. The DB is now aging WUs out much faster than before. I've gone back and checked the 200 odd ones that have been missed and I think what is happening is that WU(s) pulled in an earlier group of 250 age out, so the number of WUs changes and the count I'm using is inaccurate causing me to "miss" WUs. If I had 500 WUs at start, pull the first 250 and during that pull one of those first 250 age out when I pull the second set I start at what was 251. That means I "miss" what was 250.
----------------------------------------Unless my logic is bad I think the only solution is to allow all to be returned in one request. |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Does sorting on Received time not put the latest changes on top rather than Sent time? [I'm hunting for latest returns first]. That would gain a little time.
Yes, at the indicated 1.9 million/day techs will be clearing out [assimilating] completed results sooner to maintain performance. |
||
|
|
|