alanb1951:First, let's recall which values for fileDeleteState are available and what they mean. From the Help pages:
- fileDeleteState: Return results based on their file delete state. 0 means not deleted. 1 means ready to delete. 2 means deleted.

My FileDeleteStates for 2024-01-04 and 2024-01-05 so far:
("04T00" means 2024-01-04 between 00:00 and 00:59)

@│04T00│04T01│04T03│04T04│04T05│04T06│04T07│04T08│04T09│04T10│04T11
─┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────
0│62087│62119│62192│62276│62286│62314│62428│62433│62468│62473│62500

@│04T12│04T13│04T14│04T15│04T16│04T17│04T20│04T21│04T22│04T23
─┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────
0│62518│62650│62485│62240│62069│61774│61101│60909│60739│60513
1│     │     │     │     │    1│    1│    1│    1│    1│     
2│   17│  183│  353│  602│  779│ 1078│ 1768│ 2042│ 2282│ 2515

@│05T00│05T01│05T03│05T04│05T05│05T06│05T07│05T09│05T10│05T12
─┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────
0│60262│60025│59786│59612│59356│59201│58991│58642│58518│58285
1│     │     │     │     │     │     │     │   16│   86│    1
2│ 2771│ 3014│ 3330│ 3628│ 3894│ 4134│ 4350│ 4693│ 4807│ 5156

^
0, 1 and 2 are the FileDeleteStates.

As you can see, there are times at which data are missing, this means that something went wrong in the transfer from WCG's website to my computer.

Adri

Adri,

Neat status display :-)

Regarding "something went wrong in the transfer from WCG's website to my computer", I have been hitting the same issue quite often in the last two or three days; I think it's in some way related to the 500/503 errors that one of my periodic scripts sees occasionally, and the way the old API fetches data[*1] (which maximizes the likelihood of seeing one of those errors!).

Cheers - Al.

P.S. I've just noticed a slight reduction in my completed tasks count on the web site results page so, finally, it is starting on the purge process :-) -- the decrease roughly matches the number of items that moved to delete state 2 before 17:00 yesterday (adjusting for new tasks returned!)

*1 As part of looking into why my (old API) bulk data fetch script seems to show different date/time values for Sent or Received time on different days, I wrote another, much simpler, script using the new API to fetch only validated items; that hasn't failed yet although I run it 5 minutes after the other script (on a different system). Old API does lots of chunks, new API gets all the data in one big chunk!

Jan/06/2024 1,396,638

Looking better.
Cheers

Al wrote:Thanks!


I'm using the old API to record all my results, getting the data in chunks of 250 results. Since I was downloading over 50,000 results at the time, it needed 50,000/250 = 200 files to fetch all results. That took about 12 minutes.
However, on 22 December I switched to getting the data in one big chunk (yes, with the old API) to deal with the long download time. That takes less than a minute usually and it went well for a while, until 24 December; I think the website was in trouble, because 15 hours later it worked again. It worked without one hiccup until 26 December: my hourly downloads started to fail at 08:00 and 18:00, on 27 December at 03:00, 05:00, 06:00, 15:00, 19:00, 23:00, while on 28 December attempts at 05:00 and 12:00 failed; it wasn't until I noticed four hourly failed attempts in a row at 14, 15, 16 and 17 o'clock on that same day when I decided to switch back to fetching smaller chunks of 250 results again, since that seems more stable. My results grew to more than 63,250 at 5 January and fetching them all in pieces of 250 took 18-22 minutes.

Yes, probably temporarily (at least I hope so).


So I repeated the process from where I left off last time.
(NB These times are all localtime and since I'm only 1 hour faster than UTC, it shouldn't matter much at this point.)

@│05T14│05T15│05T16│05T17│05T18│05T19│05T20│05T22│05T23│06T00│06T01│06T02
─┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────
0│57999│58140│58117│58168│58167│58166│58202│58288│58334│58264│58367│58444
2│ 5272│ 5177│ 5030│ 4714│ 4500│ 4267│ 4019│ 3601│ 3354│ 3180│ 2944│ 2700

@│06T03│06T05│06T06│06T07│06T08│06T09│06T10│06T11│06T12│06T13│06T14│06T15
─┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────
0│58627│58744│58759│58751│58743│58753│58782│58874│58871│58857│58941│58950
2│ 2454│ 1953│ 1726│ 1456│ 1275│  974│  761│  579│  457│  253│   90│    4

@│06T16│06T17│06T18│06T21│06T22│06T23│07T00│07T01│07T02│07T03│07T04│07T05
─┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────┼─────
0│58955│58976│59065│59082│59207│59212│59216│59223│59226│59233│59237│59240

@│07T06│07T07│07T08│07T09│07T10│07T11
─┼─────┼─────┼─────┼─────┼─────┼─────
0│59381│59384│59390│59395│59400│59405

^
0 and 2 in this column are the FileDeleteStates. Value 1 is missing currenly in my results.

Since the number of valids is only very slowly increasing in my account I think the actual file deletion process is now busy.

Adri

I have had a quick look at the results of my latest data fetch (02:00 UTC 2024-01-09) and there is still absolutely no evidence of anything having passed through the assimilator since Friday afternoon...

So no quick-and-easy fix :-(

Hopefully, tomorrow will show some progress (or at least an announcement that they're aware that it's still an issue...)

Cheers - Al.;