| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Retired Forums
Forum: Known Issues [read only]
Thread: Database Server trouble - Stats will run at unusual times |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 11
|
|
| Author |
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges: 
|
We just experienced a crash of the website database. It went into recovery properly and has come back to life. We are examining what happened. During this time users were unable to access the website or the forums.
----------------------------------------[Edit 1 times, last edit by knreed at May 8, 2007 12:24:08 AM] |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
It appears that the database server crash on Friday caused some problems with the table and index statistics. This lead to a significant degradation in performance of the system. We have rebuilt those statistics on those problematic tables and this appears to have resolved the issue but we will continue to watch it slowly.
This problem only impacts the website. Both the UD grid and the BOINC grid continue to operate correctly. |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
We are going to run the stats process right now. We are trying to force a problem to occur and occurs more frequently when the stats are running.
We apologize for the inconvenience . |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
The 0:00 UTC run that is normally going to start in one hour will be delayed while we investigate a few more things.
|
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
We are going to temporarily disable the stats update process until this problem is resolved. We have determined that the problem is being caused while the stats update is underway and it makes the site unstable after that point.
Please note that no-one is losing any credit or points while the update is disabled - you just won't be able to see them on the website until the next time we run the update. We expect to resolve this within the next 24 hours. |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
Ok - we just updated the site and resolved the major issue. We are going to run stats now including the team stats for yesterday. This should run for about 1.5 hours.
|
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
We have only partially addressed the issue. The good news is that the website no longer crashes due to a problem with the new release. However, the stats update is still running at 1/3 the speed it was running last week so we still have a problem there. We are looking at the database now.
In the meantime we interrupted the stats update as it would have run for around 4-5 hours. |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
We continue to have problems with the stats running very slow. In particular the steps where we do an 'import' are taking about 50 times or so longer then normal (steps that last week ran in 10-15 seconds now run in 10-15 minutes).
We are going to go ahead and let the stats run to completion this time. We will also let the end of day stats run so that everyone can see how they are doing. However, until we can resolve the issue they will be slow. As a result we will not run the 12:00 update until it is fixed. |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
It appears that the troubles with the import process running slow is isolated to a particular server (that just happens to be the server where we run the statistics). While we continue to diagnose and resolve those issues we are moving the stats update process to another server. I've run a few of the basic steps and performance is where it should be.
In particular, I am going to run a mid-day update in the next few minutes. This should take about 30 minutes. Then tonight the end of day stats will also be run from this server and should run in its normal 50-60 minutes duration. We do apologize for the inconvenience and confusion this has caused. |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
The stats update that just ran had the website unavailable for about 26 minutes which is what the normal time was. This workaround should suffice until we are able to resolve the problems on the other server.
The good news to you as the members is that this means that we should be back in normal operations now! |
||
|
|
|