Purged to 14 November.

Mike

Until now, I have observed a build-up of tasks marked for deletion but not being purged; all my results returned before 2023-12-05 have been assimilated but I still have tasks from 2023-11-15 onwards waiting to be purged...

On 2024-03-07 I saw over 600 WUs purged, quite an improvement over the usual (350 to 400 a day...) -- I suspect this is a [beneficial] side-effect of the lack of new work being generated, and the number of my results marked for deletion but not disappearing has stopped going up by over 100 a day, and suddenly dropped by 200 instead! (The number of assimilated results stuck waiting for files to be deleted also dropped...)

I don't know whether the system is set up to run db_purge at the same time as WU generators; if it is, then there will be some degradation of d/b efficiency in both those systems as they will be changing the same tables and indexes :-)

(The alternative scenario suspends generators whilst purge is running, which should improve d/b performance whilst restricting the amount of time available to each process.)

It would be interesting to know whether the current lack of work is deliberate or just a side-effect of efforts to get more workunits cleared out.

Cheers - Al.

P.S. As at 02:00 (UTC) on 2024-03-08 I have had about 5000 results assimilated and about 2750 have actually disappeared -- currently about 13600 results still await assimilation...

Still not getting WUs. I'm ok with that if good things are happening. My results list continues to shrink slowly.

TigerLily has given us good updates in a couple of threads. OPNG is on a pause until the end of the month. The MCM clean up operation is happening, but it is slow. There might be a db access issue. The techs are working on the issue. They know we aren't getting WUs and are trying to balance the clean up operation with continuing to send out WUs.

Fire up the backup projects, and we will see what Monday brings.

The number of Valid tasks (determined by the command wcgstats -wsV -aMCM1 -m0 -P1) in my result list has shrunk by two thousand in the past 8 hours, so purging is taking place.

The speed of distributing (MCM1-)workunits looks quite decent with the occasional gap (according to my latest downloaded general task, updated hourly).

The website is flaky, (parts of) pages are failing to load at times (or do take extra time to load).

Adri

In support of what Adri says about WU purging... since the day after the assimilators resumed work (which being the first day purging could take place) I've not had a single day where nothing disappeared!

As at 02:00 (UTC) on 2024-03-10, my current MCM1 state is:

    Total results purged over 10 days:  4282
    Results awaiting file deletion:      450
    Results awaiting d/b purge:         1375
    Results awaiting assimilation:     12885

    Oldest results still awaiting removal are from 2023-11-15;
    oldest results waiting for assimilation are from 2023-12-09

Note the significant difference between those two dates! That is in part due to the different rates of progress of the various MCM1 assimilators, and may also reflect different rates of progress on file deletion if there are multiple deleters running. And although the db_purge daemon is clearing a lot of results, it isn't necessarily keeping up!

Over the 10 days so far, my queue of valid results has dropped by 2603 -- given that I also return around 150 to 180 results a day that's about right :-)

The only worrying aspect of all this is that if the rate of purging I'm seeing is typical, I reckon it won't get back to normal for a couple of months :-( -- I hope I'm wrong about that :-)

Cheers - Al.

P.S. I haven't been looking at OPNG in such detail as MCM1, but with a much smaller volume of tasks to deal with it appeared that the assimilator(, file deleter and purge process were coping fairly well for OPNG until the MCM1 assimilators got turned on again :-) Now there's no more new work, OPNG WU removal seems to be catching up again...