Hi Al,

Looking at the apache logs I see just over 1000 such cases from the following 10 batches of which 9/10 were created with the new system, all parsed from 404 log lines since roughly nov 3rd around 16:40 UTC and there's probably more in the rotated/archived logs:
0241027 0241652 0241721 0241727 0241781 0241784 0241791 0241843 0241896 0241921

Best guess as to why this is happening is some requests are being redispatched to the wrong backend if the initial HTTP request that should be routed to the right server/bucket fails. Checking a few of the specific files, that bucket was not in the contiguous range of the node's "partition". As I wait for my new indexes to finish creating in the BOINC database so I can restart with an updated server config as well, I'll review the HAProxy config and see if it is as simple as removing the redispatch directive. If there is another way the request is falling through to round-robin and skipping my use-server directive, that might take a bit more research to make sure I don't mess up what is working.

Part of the reason reconciling the workunits that were uploaded right after we restarted was an off-by-one error in my routing policy that caused nearly every POST to miss the use-server directive that is supposed to direct it to the correct partition, and instead of going to the partition corresponding to the contiguous 1/6th of the 1024 fanout buckets I have been rambling about in updates it just went willy nilly everywhere, which I discovered later. For uploads, I did create all 1024 buckets to accept from any server in this case, but not for downloads although the result would still have been a 404 if my theory holds.

The 403s might be lock contention in the DB2 instance that holds the website and forum database, but I'm not so sure about that theory yet. I started looking into it while hosting was fixing the Ceph issue, but I didn't get far in the exploratory phase.

Aaah! Thanks for the update!

That's why I have exhausted all WCG WU's in my machines. Most of my machines still have quite a que of WU's that are finished and ready to upload but just sitting there.
Well... while this is going on, I think I'll just build another Boinc machine with recycled hardware. Got nothing better to do over the weekend. lol

Dylan,

Thanks for the detailed responses on progress. I dread to think how many items needed to be incorporated in each new index -- not really surprised to know it has been taking a while! It will be interesting to see what sort of long-term performance effects that might have on BOINC d/b access -- index maintenance and usage versus full table scans? :-)

Good luck with any fixes...

Cheers - Al.

P.S. I had to back out of the forums and go back in again to get this to post. SIde-effect of proxy fixes?

Thanks for the update. Hopefully I can put the machine to use shortly.

same for me