You have more horsepower than I do, I only have 15000 in "valid not purged."

Cheers

Regarding "kicking the MCM1 assimilators":

Quite a while ago, the techs reported that the assimilators would stop working (unclear whether they hung or exited) and if restarted they'd soon stop again. There was a brief period in November where they stood up for over a day and that cleared out a fraction of what was waiting then.

More recently, someone posted that they seemed to recall that this happened to IBM/WCG at some point in the past. I went looking at Kevin Reed's posts for anything about work not being purged, and found one or two references in 2013 :-) -- apparently, the assimilator was deadlocking with one of the other processes because both were trying to access the same data (which he seemed to think shouldn't have been able to happen...) It wasn't clear [to me] what the solution to that specific problem was, as there were other issues around the same time, ending in a MySQL database upgrade and a few days offline to clear up the mess.

In the current situation, what I find a little disturbing is that we've not even heard a "We know what causes it but we don't yet know how to fix it!" statement. I'd hope they've reached that point by now, as there is quite a lot of diagnostic information that can be logged if required, although it is possible that the issue is something that doesn't trigger a logging event so it might be a case of zeroing in on a possible cause (which won't necessarily be easy for folks who've not had years working with the BOINC code![*1], especially if they have other duties too!)

If the issue is data-related, allowing more data to accumulate is unlikely to be helping;. Unfortunately, as the only CPU project with work at present is MCM1 and that appears to be the only project having problems, any resolution is likely to irritate a lot of users! I sometimes think some of the users here and on other projects think that all problems can be solved with a reboot [because it works for Windows?] :-)

Instinct tells me that the BOINC side of the system needs a few days with no user activity so that they don't have to try to run assimilators, validators, file deleters and the purge mechanism concurrently most of the time -- IBM used to shut off one or more of the subsystems at stress times (e.g. the statistics runs) anyway. That might well make doing diagnostics and trying solutions a lot easier!

If kicking things was going to work, it would have worked by now :-)

Cheers - Al.

*1 There's quite a steep learning curve :-) -- I did quite a lot of code-diving when Milkyway was having problems, as I reckoned anything I could find out might help (I hope it did); I've also spent a while trying to convince myself I understand the data flows involved in validating. assimilating and purging work units (I'm not 100% convinced!) -- I certainly wouldn't claim to be competent to run a BOINC service and I don't envy them the debugging task! I just wish I could help somehow...

Maybe they could reach out to Kevin Reed, Uplinger, or Al Seippel (?), if they could find them and pick their brains (if they are willing) or maybe try to retain their services as consultants. It might cost a few bucks but it would probably be money well spent in not having to reinvent the wheel.

Cheers

TigerLily, This problem is still with us. I now have over 100,000 MCM tasks in valid state. They are not purging. This means it can take minutes for the My contribution-> Results query to run. Many times it just hangs out. This must be slowing down the backend as well.

Can you tell us if the problem has been accepted by the team?

Hello Blount and bfmorse,

The tech team is aware of the issue and are working on resolving it. The issue is that some host IDs are reading 0 in the results table, causing MCM1 assimilators to quit at times. However, this has only affected about 10,000 results and new results are accumulating that are unaffected by this issue. They confirmed that we still have the time and space on the server to allow these results to accumulate while they work on finding a fix for this problem. Thanks for your patience as they work on resolving this issue.