Ralf,

I really enjoy your analysis. However, I also think (like Al) there is some sort of infrastructure shortage. How do you explain the non-responsive home page? Especially the 3 pictures that load very slowly. I would think that has to be a bandwidth shortage, or a shortage of CPU power behind that bandwidth. I don't see how a slow web page load could be anything else. If it was a connection type of shortage, I would expect an all or nothing scenario. Of course, I could be wrong . I would be interested in your thoughts.

Mark

I did not think to mention that I routinely clear my web browser cache and history (I use a cleaner program). If I do not clean the cache, the home web page is quite fast after the first (slow) load. I would be curious if you see the same thing if your clear your cache.

I think you make a valid point, Mark. I had noticed the Stats overview page (masochist that I am I keep checking it!) and Forum pages loading very slowly just recently. (Not make a cup of tea slow but definite white screen for a frustrating number of seconds. Initially, I had assumed it was my end but checking other, more graphic heavy websites they loaded as fast as usual. It is very noticeable tonight which prompted this comment.
Cheers
Phillip
PS: I am finding this remote problem analysis by you, Ralf and others interesting (especially in the absence of detailed information from Kembrill). Appreciate the insights. Thanks!

That is something that should have happened BEFORE they made the announcement that they are moving in 2021!
But from September 2021 to February 2022, that would have been 5 months to get all the basic stuff figured out, necessary bandwidth, certificates, required rackspace incl. connectivity, etc.
That's all just "pencil pusher" work that could all be done while the system still was running at IBM.
Then after the shutdown in end of February 2022, creating snapshots of all servers (supposedly, these were all cloud instances!), backup of all the databases, moving the required stuff (which had been figured out in the 5 months before) to Toronto, restoring data and VMs.Then standing up all the server instances one by one as necessary. That included the web site and forum, which came up first (May?, can't find the announcement right now), That would have been 3 months. Then in June, they claimed to have the basic data servers back up to the point they could send out "test" WUs. When then immediately connectivity issues came up, we were told they had to wait for the networking guy to come back from summer vacation. Weeks gone by before there was any apparently progress. And now we are here, end of September, the leaves are starting to turn color, and we are still not one step further. Not to mention such silly snafu as forgetting to extend the certificates for the whole thing. And that even after people were harping about switching the data transfer from http to https, which would have required to install/access the certificates, at which point the latest it should have been noticed that they were set to expire soon.
I think that every transfer problem (after been able to catch one of those scarce connections) that shows up is a symptom rather the cause of the issues here. I am not sure if most people (no offense intended) actually understand what "bandwidth" entails.And what the actual series of steps in case of data transfer for a request of WUs from the BOINC client actually are.I don't know the details of the way this is now implemented at Krembil (including database clusters, load balancers, data aggregators), but I am participating long enough in WCG (and other DC projects before that) that I think I have a more than fair understanding what is involved, beside having myself designed and programmed functionally similar industrial (MRP) setups, so the flow of data was reversed (tons of rather small(ish) data packets push in very rapid succession onto a database server to be processed, instead of pulling sending out lots of small files to a lot of different clients).
Well, that is the problem with any discussion on our side. We can base any ideas on the results/symptoms we are seeing. And it is very easy to mistake symptoms for causes.As I mentioned, I have not seen any unusual slowdown of either web site (the Results page is the only thing I more or less frequently refresh), and the forum, but again, those should be hosted on different server instances and (if properly implemented in the first place) not be effected (or have and effect to) by the projects database/BOINC side of things. Pointing to the same general domain should be the only commonality, hence IMPE, things like a (temporary/false positive) DDoS detection could be one possible explaination. There might be others, but that again is something that can only be deduced by symptoms experienced in the whole picture.

Ralf

Exactly!