Following up with explanations and examples..;.

As I've mentioned elsewhere when the topic of stalled assimilation has come up, I've been tryiog to watch assimilator behaviour using data from a daily results statistics collection script I run using the old API, which happens to provide information on the "File Delete State" flag on each result.

Over the months since I started looking at that information, I've refined the toolbox to the point where I can produce a summary of the data about my MCM1 tasks that are still shown as Valid for each day's data collection. Normally, the data is collected at about 02:00 UTC; as well as the summaries (examples below) it allows me to observe daily changes...

Once I realized that since 2024-01-11 only odd-number WUs were going away I adapted one of my scripts to include information about how many odd-numbered WUs were amongst those removed and those still held. Once it became apparent that equation for removal seemed to be

   Workunit ID modulo 4 = 1

so I modified it again to show four counts, one for each grouping.

I will show a few sample reports, ranging from normal behaviour before the backlog built up to what is now happening. As mentioned above, data is collected early in the day so it mostly reflects activity on the day before. (This post is getting long, so most of them will be in follow-ups...)

The mod-4 counts indicate how many WU IDs have each remainder modulo 4. so (for example) "mod-4 counts [1, 1, 0, 4] in the second report shown below indicates 1 WU in the "remainder 0" and "remainder 1" groups and 4 WUs in the "remainder 3" group.

The ModTime values quoted are from the result record in its "file delete state 2" phase; such results are normally purged by the next day...

(Note that during this period I typically returned between 140 and 180 results a day, so if it assimilates 200+ it is [just] ahead of the game. Of course, the validation rates are uneven at the best of times, for various reasons[*1])

Here is a sample from before the problems began -- note the low numbers of WUs and their fairly even distribution:

Examined data collected on 2023-10-01

Found 213 MCM1 items:  mod-4 counts [54, 61, 46, 52]

There were 129 items in delete state 2:  mod-4 counts [31, 36, 28, 34]
    Lowest ModTime was 2023-09-30 02:03:06 (1696039386):     WU Id was 387741771
    Highest ModTime was 2023-10-01 01:15:38 (1696122938):    WU Id was 388835091

And one from just before the assimilator(s) seemed to die completely -- note the low assimilated items cont and a small build-up in waiting tasks.

Examined data collected on 2023-10-24

Found 598 MCM1 items:  mod-4 counts [87, 94, 321, 96]

There were 6 items in delete state 2:  mod-4 counts [1, 1, 0, 4]
    Lowest ModTime was 2023-10-23 04:45:15 (1698036315):     WU Id was 404088191
    Highest ModTime was 2023-10-23 21:59:30 (1698098370):    WU Id was 404161780

More in another follow-up post -- I fear this may already be too long for some :-)

Cheers - Al.

*1 The need for retries seems to vary in intensity, so the number of results that need validation can climb quite steeply on occasion, then drop back down two or three days later (though the current "Waiting to be sent" problem can turn that into 8+ days later...) This is not the right thread for debating why that happens; hence the footnote :-)

A lovely piece of analysis, thank you.

Hi everyone,

I forwarded your analyses and insights to the tech team and they provided the following update:

Correct, as deduced by alanb1951 and adriverhoef the workunits assigned to an instance of the MCM1 assimilator are determined by applying the modulo operator to the workunit ID, so 3/4 assimilators for MCM1 are not currently running due to problems with the workunits in their assigned ranges triggering a non-zero exit condition for the assimilator. After resolving the initial issue, we encountered another issue that triggers a different non-zero exit condition in the MCM1 assimilator code. We have tested a modified version of the assimilator that is able to handle these errors and continue processing workunits that are not affected in the assigned range. The new assimilator will mark the workunits that flagged as problematic in assimilation with an appropriate error_mask, so that they can be handled separately as a group, and once we get through them all we should be in the clear. In the interim, we should be able to process the vast majority of workunits that pass the checks in the assimilator by catching these two error conditions, but allowing the assimlator to continue. The new version of the assimilator should be up and running Monday, and the expected outcome is that we will return to the "flurry of assimilations" from early January. If all goes well, we should be able to double the rate again, and then again, essentially by starting more assimilators with a higher modulo, after allowing 48h in between to monitor.

Adri - thanks for the confirmation report. Hope you liked the response!

Now we play "Watch this space"...;

Cheers - Al.

Thank you TigerLily for communicating the great information provided in this thread to the tech team, and for letting us know the response. It truly makes me happy to see the community working together so well.