| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
| World Community Grid Forums
|
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 8
|
|
| Author |
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Since the beginning of the WCG Initiative, it's taking progressively more time to calculate stats. At first it was around 15 minutes, then 18, 19, 21...now it's taking over 30 minutes to calculate the stats. Is this a function of the increase in user database or something else? If so, when the database approaches 280,000 users, will it take 60 minutes to calculate stats?
Thanks, dreplogle |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Dreplogle --
The length of the stats job has increased a bit, but it is also starting later and I think that has had the greater impact on the time that it completes. You may have noticed that I consolidate a few stats and post them on the "My Online Team" team thread each day and have been watching and waiting for the Stats job to complete on a daily basis for at least the past six months. When I first started doing it, the stats job appeared to kick off at about 00:01 GMT and would usually complete sometime between 00:13 and 00:15. Recently, it appears that the start time usually is around 00:08 (or even a bit later) with a completion around 00:22 to 00:23. I think it was two nights ago that the job did not appear to start until about 00:27 and did not complete until about 00:46! Obviously the job is not submitted via any automated scheduling package or we would see more consistant timings. I don't think the increase in members has had more than a five minute impact on the update. |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Thanks for the response Dave!
I'm really confused then. I see on other sites (for example, http://www.boincstats.com/ ) that have stats for WCG which seem to actually post several minutes before the hour! So you can see your stats updated before the WCG site even starts processing! That can't be possible.Any comment? Thanks, dreplogle |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Dreplogle --
I really haven't watched stats from other sites. Perhaps the boincstats site is reporting only those statistics from WCG users that are using the BOINC agent here. Anyone who would want to report current statistics from this site would have to be driven by the cycle of updates here (individual statistics are updated shortly after 06:00, 12:00 and 18:00 with individual, team regional, etc. statistics updated after 00:00 GMT) and then they would have to be aware that the jobs usually finish about a half hour after the stated times. I know that our team captain updates our team website at about 00:45 and has been noticing the slow creep towards later completion times and is considering delaying his update as well. |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
The stats process starts at 00:06, 06:06, 12:06, 18:06 UTC each day (it is scheduled via cron). The six minute delay is to make sure that any issues slight differences in time between the servers does not impact the stats. Six minutes is more then enough (all servers sync with a time server so it should be negigible but if the time server goes down or a new firewall rule prevents communications and we don't notice for a short period this makes sure that the stats are correct even if a minor time difference occurs.)
The first step is to actually extract the data from the BOINC and UD databases and do some pre-processing. The site is not marked as unavailable during this time. This has variation in how long it takes to execute and that is why the 'start time' from the members perspective varies. The second step is to actually load this data into the website and it is during this time that you get the unavailable message. We are planning to improve the stats part of the site in 1Q 2006. Part of this will be displaying new information and some of this will be improving the performance. There are a couple of places where the performance is not what it could be so we will be fixing that and this should improve the experience. |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
Thanks for the response Dave! I'm really confused then. I see on other sites (for example, http://www.boincstats.com/ ) that have stats for WCG which seem to actually post several minutes before the hour! So you can see your stats updated before the WCG site even starts processing! That can't be possible.Any comment? Thanks, dreplogle I am not certain what exactly those other stats are representing as the last update timestamp. Compare what you find here: http://www.worldcommunitygrid.org/boinc/stats/tables.xml with what you see on the website. We regenerate the xml files at 0:00, 6:00, 12:00, 18:00 UTC (and actually at 15:50 UTC becuase the owner of BOINC stats indicated that he grabs the stats from WCG around 16:15 UTC). Kevin |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Kevin --
Thanks for clarifying the process. I don't understand, however, what happened the night of Jan. 3/4 though. The second phase in which we cannot log in didn't start until at least 00:27 UTC. It didn't complete until around 00:47 which is much later than the norm which has been between about 00:22 and 00:30 each evening. Tonight's update was complete at just about 00:27. |
||
|
|
knreed
Former World Community Grid Tech Joined: Nov 8, 2004 Post Count: 4504 Status: Offline Project Badges:
|
Kevin -- Thanks for clarifying the process. I don't understand, however, what happened the night of Jan. 3/4 though. The second phase in which we cannot log in didn't start until at least 00:27 UTC. It didn't complete until around 00:47 which is much later than the norm which has been between about 00:22 and 00:30 each evening. Tonight's update was complete at just about 00:27. When the first step is running, the UD server and BOINC servers are still in full action. This means that they are actively updating the tables accessed by the stats update process. This causes variation in how long the export takes (both because or row/table locks and becuase of usage). Also - if a new batch is being loaded while the stats are running that slows down the export as well (it adds additional load). |
||
|
|
|