Mine are purged to 12 December.

Mike

I've seen some good progress on clearing out the backlog of valid tasks over the last couple of days; I haven't quite got as far as Mike has, but that'll probably be down to the mix of WU numbers and corresponding file deleters :-)

As at 03:40 UTC[*1] on 2024-03-12, my current MCM1 state is:

    Total results purged over 12 days:  9126
    Results awaiting file deletion:        6
    Results awaiting d/b purge:         1949
    Results awaiting assimilation:     11593

    Oldest results still awaiting removal are from 2023-12-10;
    oldest results waiting for assimilation are from 2023-12-23

The last two days saw nearly 1600 items purged (over 1000 of them going away on 2024-03-10) and a decrease of about 1300 in valid tasks awaiting assimilation; again, I'm seeing about 150 results being validated each day, so that's about right.

About 1000 of the results awaiting assimilation are from 2023, and all the results currently awaiting db_purge are from 2023; if it continues purging at such a rate that about 500 or 600 of mine disappear each day, Unixchick might be able to celebrate by the beginning of next week :-)

Cheers - Al.

P.S. I note that all my OPNG tasks have also been purged (at last!)

*1 -- the API I use to collect delete states has seen the return of an issue Adri and I documented a while back, so sometimes the 02:00 collection fails and I need to invoke a back-up procedure. Unfortunately that isn't automatic -- it's rare enough that I didn't want to invest a lot of effort into it, and I can't be bothered to do the mass recoding that would be needed to use the back-up procedure instead of what's currently in use! :-)

[Edit: added note on OPNG]

Mike,

I'm not 100% sure what sort of detail you are asking for, so don't blame me it this is "too much information"...

Firstly, I should point out that I'm usually interested in snapshots at a consistent time of day so that I can see what is changing on a daily basis -- all I need to know ()most of the time) is which day, not which hour :-)

Nowadays, I know when all results are returned (or timed out, or whatever) for any WU I've helped to process - I needed the new API to track wingmen (or else I would've been driven to screen scraping HTML pages -- no thank you!) That said, knowing when a result returns doesn't tell me when it validates, just that it might become a candidate -- I might be able to deduce vallidation time by other means, but the new API doesn't answer that question :-)

Another thing for which I can't give a more accurate time is when a WU is actually purged (and at the moment the "24 hours after file deletion" estimate is not a realistic guess...)

As for knowing when WUs pass through assimilation or the file deletion stage, that's down to data I can scavenge from daily bulk collections using the old API (my BOINC task performance and credit monitoring database has been in use for a long time!); that has the useful side-effect of letting me monitor "ModTime" and "delete states."

ModTime (when the WU database record was last changed, I believe) is the only date/time returned as a UNIX timestamp (integer value) and is the only date/time that seems to be 100% trustworthy when large batches of data are retrieved[*1] :-) It reflects the times when the assimilator and file-deleter marked their work completed (and might reflect validation time if one catches it in a gap between Pending Validation/Verification and assimilation.)

In order to narrow down when a task was validated or purged would require much more frequent sampling than my current daily bulk snapshots, which might be acceptable if there are only a couple of hundred results, but with 10,000+ tasks that's unacceptable as far as I'm concerned :-) so daily it is (API availability permitting!)

Cheers - Al.

*1 There's something a bit dodgy in the mechanism the old API uses to convert dates from the [UNIX timestamp] format kept in the database to the human-friendly format it returns for the times tasks are sent, due, and returned -- I've seen impossible combinations (future sent or returned dates, returned before sent; sent after due, and so on) and invalid dates (31st November seems to have been a favourite during the current problems!) Fortunately, it leaves ModTime alone! The new API doesn't seem to have the same troubles (and yes, I've done simultaneous data retrieval tests to verify that!)

I think it is time for a mini celebration. I don't have any 2023 MCM WUs in my results list.

Yup - I think we can all celebrate the improvement in WU removal! :-)

I underestimated when that might happen when I posted earlier in the week -- however, my remaining tasks are now disappearing at least twice as fast, which has to help :-)

I note that Adri made a more recent post about possible finish times in that Assimilators thread in the MCM1 forum. If it can keep up the current rate of purging, I think he may be close!

Cheers - Al.

P.S. The sudden improvement in how fast items are being purged is presumably down to some of the changes mentioned in TigerLily's post at the beginning of the week. If so, well done Tech Team! (And if not, it's a happy coincidence!)