My Windows machine again stopped incrementing CPU time. Had to stop the service and re-start which set them all back to 0.5% again. Hopefully, I can get them to complete. They ran just short of 4 hours before locking up again. About the same time they ran on the first pass before CPU time stopped.

Got one of them to complete, here's the result log which will hopefully offer some insight. The breaks are where I stopped the service and restarted:

World Community Grid

Result Log

Result Name: BETA_ BETA_ 9999983_ 0968_ 2--



<core_client_version>7.0.64</core_client_version>
<![CDATA[
<stderr_txt>
Commandline = projects/www.worldcommunitygrid.org/wcgrid_beta17_7.21_windows_x86_64 -SettingsFile BETA_9999983_0968.txt -DatabaseFile dataset-GDS2771-v1.txt
Initializing
wcg_learn_limit = 100000
Running
[16:38:35]: Computing pass 0
[16:39:35]: Computing pass 1
[16:41:03]: Computing pass 2
[16:42:36]: Computing pass 3
[16:43:45]: Computing pass 4
[16:44:57]: Computing pass 5
[16:46:34]: Computing pass 6
[16:51:06]: Computing pass 7
[16:52:44]: Computing pass 8
[16:53:39]: Computing pass 9
[16:54:47]: Computing pass 10
[16:55:41]: Computing pass 11
[16:56:33]: Computing pass 12
[16:57:27]: Computing pass 13
[16:58:22]: Computing pass 14
[16:59:17]: Computing pass 15
[17:01:49]: Computing pass 16
[17:02:48]: Computing pass 17
[17:03:33]: Computing pass 18
[17:06:50]: Computing pass 19
Commandline = projects/www.worldcommunitygrid.org/wcgrid_beta17_7.21_windows_x86_64 -SettingsFile BETA_9999983_0968.txt -DatabaseFile dataset-GDS2771-v1.txt
Initializing
wcg_learn_limit = 100000
Running
[17:10:41]: Computing pass 0
[17:11:36]: Computing pass 9
[17:12:45]: Computing pass 10
[17:13:38]: Computing pass 11
[17:14:29]: Computing pass 12
[17:15:21]: Computing pass 13
[17:16:15]: Computing pass 14
[17:17:11]: Computing pass 15
[17:19:44]: Computing pass 16
[17:20:45]: Computing pass 17
[17:21:43]: Computing pass 18
[17:24:53]: Computing pass 19
[17:25:54]: Computing pass 20
[17:30:01]: Computing pass 21
[17:31:18]: Computing pass 22
[17:32:05]: Computing pass 23
[17:33:02]: Computing pass 24
[17:34:25]: Computing pass 25
[17:35:18]: Computing pass 26
[17:36:32]: Computing pass 27
[17:37:37]: Computing pass 28
[17:38:38]: Computing pass 29
[17:42:58]: Computing pass 30
[17:44:14]: Computing pass 31
[17:45:02]: Computing pass 32
[17:45:59]: Computing pass 33
[17:47:26]: Computing pass 34
[17:48:17]: Computing pass 35
[17:49:32]: Computing pass 36
[17:50:37]: Computing pass 37
[17:51:39]: Computing pass 38
[17:55:57]: Computing pass 39
[17:57:11]: Computing pass 40
[17:57:58]: Computing pass 41
[17:58:56]: Computing pass 42
[18:00:26]: Computing pass 43
[18:01:18]: Computing pass 44
[18:02:33]: Computing pass 45
[18:03:36]: Computing pass 46
[18:04:38]: Computing pass 47
[18:08:51]: Computing pass 48
[18:10:05]: Computing pass 49
[18:10:51]: Computing pass 50
[18:11:46]: Computing pass 51
[18:13:12]: Computing pass 52
[18:14:02]: Computing pass 53
[18:15:15]: Computing pass 54
[18:16:19]: Computing pass 55
[18:17:19]: Computing pass 56
[18:21:25]: Computing pass 57
[18:22:40]: Computing pass 58
[18:23:27]: Computing pass 59
[18:24:28]: Computing pass 60
[18:25:59]: Computing pass 61
[18:26:52]: Computing pass 62
[18:28:17]: Computing pass 63
[18:29:24]: Computing pass 64
[18:30:26]: Computing pass 65
[18:34:32]: Computing pass 66
[18:35:45]: Computing pass 67
[18:36:29]: Computing pass 68
[18:37:21]: Computing pass 69
[18:38:49]: Computing pass 70
[18:39:39]: Computing pass 71
[18:40:41]: Computing pass 72
[18:41:46]: Computing pass 73
[18:42:50]: Computing pass 74
[18:46:58]: Computing pass 75
[18:48:14]: Computing pass 76
[18:49:00]: Computing pass 77
[18:49:58]: Computing pass 78
[18:51:26]: Computing pass 79
[18:52:16]: Computing pass 80
[18:53:24]: Computing pass 81
[18:54:24]: Computing pass 82
[18:55:21]: Computing pass 83
[18:59:18]: Computing pass 84
[19:00:34]: Computing pass 85
[19:01:18]: Computing pass 86
[19:02:12]: Computing pass 87
[19:03:41]: Computing pass 88
[19:04:30]: Computing pass 89
[19:05:41]: Computing pass 90
[19:06:43]: Computing pass 91
[19:07:45]: Computing pass 92
[19:11:35]: Computing pass 93
[19:12:47]: Computing pass 94
[19:13:31]: Computing pass 95
[19:14:23]: Computing pass 96
[19:15:49]: Computing pass 97
[19:16:39]: Computing pass 98
[19:17:54]: Computing pass 99
[19:18:57]: Computing pass 100
[19:19:57]: Computing pass 101
[19:24:04]: Computing pass 102
[19:25:20]: Computing pass 103
[19:26:06]: Computing pass 104
[19:27:02]: Computing pass 105
[19:28:24]: Computing pass 106
[19:29:14]: Computing pass 107
[19:30:24]: Computing pass 108
[19:31:26]: Computing pass 109
[19:32:25]: Computing pass 110
[19:36:21]: Computing pass 111
[19:37:34]: Computing pass 112
[19:38:18]: Computing pass 113
[19:39:12]: Computing pass 114
[19:40:32]: Computing pass 115
[19:41:23]: Computing pass 116
[19:42:35]: Computing pass 117
[19:43:36]: Computing pass 118
[19:44:35]: Computing pass 119
[19:48:36]: Computing pass 120
[19:49:50]: Computing pass 121
[19:50:35]: Computing pass 122
[19:51:30]: Computing pass 123
[19:52:49]: Computing pass 124
[19:53:39]: Computing pass 125
[19:54:50]: Computing pass 126
[19:55:52]: Computing pass 127
[19:56:53]: Computing pass 128
[20:01:04]: Computing pass 129
[20:02:19]: Computing pass 130
[20:03:03]: Computing pass 131
[20:03:58]: Computing pass 132
[20:05:22]: Computing pass 133
[20:06:10]: Computing pass 134
[20:07:21]: Computing pass 135
[20:08:26]: Computing pass 136
[20:09:25]: Computing pass 137
[20:13:30]: Computing pass 138
[20:14:44]: Computing pass 139
[20:15:29]: Computing pass 140
[20:16:22]: Computing pass 141
[20:17:44]: Computing pass 142
[20:18:34]: Computing pass 143
[20:19:47]: Computing pass 144
[20:20:50]: Computing pass 145
[20:21:56]: Computing pass 146
[20:26:16]: Computing pass 147
[20:27:30]: Computing pass 148
[20:28:15]: Computing pass 149
[20:29:10]: Computing pass 150
[20:30:35]: Computing pass 151
[20:31:23]: Computing pass 152
Commandline = projects/www.worldcommunitygrid.org/wcgrid_beta17_7.21_windows_x86_64 -SettingsFile BETA_9999983_0968.txt -DatabaseFile dataset-GDS2771-v1.txt
Initializing
wcg_learn_limit = 100000
Running
[20:34:41]: Computing pass 0
[20:36:01]: Computing pass 124
[20:36:51]: Computing pass 125
[20:38:03]: Computing pass 126
[20:39:05]: Computing pass 127
[20:40:03]: Computing pass 128
[20:44:13]: Computing pass 129
[20:45:27]: Computing pass 130
[20:46:13]: Computing pass 131
[20:47:08]: Computing pass 132
[20:48:32]: Computing pass 133
[20:49:22]: Computing pass 134
[20:50:35]: Computing pass 135
[20:51:38]: Computing pass 136
[20:52:37]: Computing pass 137
[20:56:41]: Computing pass 138
[20:57:53]: Computing pass 139
[20:58:38]: Computing pass 140
[20:59:32]: Computing pass 141
[21:00:56]: Computing pass 142
[21:01:45]: Computing pass 143
[21:03:17]: Computing pass 144
[21:05:20]: Computing pass 145
[21:07:19]: Computing pass 146
[21:14:21]: Computing pass 147
[21:15:33]: Computing pass 148
[21:16:18]: Computing pass 149
[21:17:12]: Computing pass 150
[21:18:35]: Computing pass 151
[21:19:24]: Computing pass 152
[21:20:40]: Computing pass 153
[21:21:42]: Computing pass 154
Commandline = projects/www.worldcommunitygrid.org/wcgrid_beta17_7.21_windows_x86_64 -SettingsFile BETA_9999983_0968.txt -DatabaseFile dataset-GDS2771-v1.txt
Initializing
wcg_learn_limit = 100000
Running
[21:25:50]: Computing pass 0
[21:26:51]: Computing pass 146
[21:30:59]: Computing pass 147
[21:32:16]: Computing pass 148
[21:33:04]: Computing pass 149
[21:34:00]: Computing pass 150
[21:35:26]: Computing pass 151
[21:36:17]: Computing pass 152
[21:37:29]: Computing pass 153
[21:38:32]: Computing pass 154
[21:39:31]: Computing pass 155
[21:43:33]: Computing pass 156
[21:44:46]: Computing pass 157
[21:45:29]: Computing pass 158
[21:46:22]: Computing pass 159
[21:47:51]: Computing pass 160
[21:48:40]: Computing pass 161
[21:49:54]: Computing pass 162
[21:50:56]: Computing pass 163
[21:51:55]: Computing pass 164
[21:56:01]: Computing pass 165
[21:57:13]: Computing pass 166
[21:57:58]: Computing pass 167
[21:58:51]: Computing pass 168
[22:00:16]: Computing pass 169
[22:01:04]: Computing pass 170
[22:02:07]: Computing pass 171
[22:03:09]: Computing pass 172
[22:04:07]: Computing pass 173
[22:08:02]: Computing pass 174
[22:09:14]: Computing pass 175
[22:09:59]: Computing pass 176
[22:10:52]: Computing pass 177
[22:12:18]: Computing pass 178
[22:13:07]: Computing pass 179
[22:14:15]: Computing pass 180
[22:15:15]: Computing pass 181
[22:16:13]: Computing pass 182
[22:20:05]: Computing pass 183
[22:21:18]: Computing pass 184
[22:22:02]: Computing pass 185
[22:22:55]: Computing pass 186
[22:24:20]: Computing pass 187
[22:25:08]: Computing pass 188
[22:26:16]: Computing pass 189
[22:27:16]: Computing pass 190
Result.out = 205.000000
Run complete, CPU time: 16292.651380
22:28:14 (3348): called boinc_finish

</stderr_txt>
]]>

Here's a little snapshot of 3 betas, with values copied manually from BOINC Manager while they were running, and then at completion.
These WUs were started manually at about the same time.
This instance of BOINC Manager is not reporting checkpointing in its Event Log, for some reason unknown to me.

Name . . . . . . . . . . . . . . . . . . . . . . | Progress | Elapsed | Remaining | El. Time at completion
BETA_BETA_9999987_0764a_0 | 37.9% | 47:00 | 11:11 (increasing) | 1:58
BETA_BETA_9999981_0725_2 | 38.0% | 47:55 | 11:22 (increasing) | 1:58
BETA_BETA_9999987_0758a_0 | 22.4% | 49:32 | 08:40 (increasing) | 2:51 ("suspended" for approx 40 min at 49.6% / 1:46 elapsed)

i7-3770K, Win 7 x64, BOINC 7.0.64 x64

As can be seen, during run time the "Remaining" time estimate underestimates the value obtained by extrapolating the Elapsed time by the Completion %. The initial time estimates are too low.

During runtime, the task Properties function on this machine reported:
* that they are checkpointing about every 10 min.
* that when running 3 betas at a time, CPU time is quite close to Elapsed time, eg 1:15:39 vs 1:15:53, ie efficiency seems OK.

When I suspended BETA_BETA_9999987_0758a_0 (LAIM ON), BOINC Manager reported it as suspended and a new WU was started, but Windows Task Manager showed that the beta was still running, with 8 threads' CPU time then being shared among 9 WUs.
Checkpoint files in the suspended beta's slot directory were updated once, but then updating seemed to stop. I resumed the beta in BOINC Manager after about 40 min. Progress % remained frozen. BOINC Manager WU Properties resumed incrementing the Elapsed time but retained the ~40min lag, and CPU Time estimate remained frozen. Working Set and Private Set Memory in Task Mgr dropped to only 8MB and 7MB, both fluctuating slightly. No progress seemed to be happening, though.
I was going to let the WU run to about 3:30 Task Manager Time, and then restart BOINC to test restarting from a checkpoint, but next time I looked, the WU had finished without further intervention. The claimed CPU time is way below the real value by about the 40 min "suspended" time plus subsequent runtime to finish.
Validated OK.

Also noted: Each of these WUs runs as a single task under Windows, unlike CEP2 and VINA WUs, which each run as 2 tasks with a submanager process that administers the actual science programs as sub-tasks. This new project is not just CEP2 or VINA with a new grille and headlights.

You might still get re-sends, but for now I believe they are indeed gone. I am yet to deliver 7 WUs and I bet there are others with long queues too, so everything is possible still. Either way, it's pretty obvious (to me) a new Beta is needed here so stay tuned!

Cheers!

Edit: added the quote from X-Files 27

There might be an issue with the "Maximum elapsed time exceeded" check. I've just successfully completed BETA_ BETA_ 9999981_ 0555_ 3 as follows:

BETA_ BETA_ 9999981_ 0555_ 3-- 721 Pending Validation 05/11/13 03:53:39 05/11/13 07:38:23 3.70 94.7 / 0.0
BETA_ BETA_ 9999981_ 0555_ 2-- - In Progress 04/11/13 19:43:17 06/11/13 19:43:17 0.00 0.0 / 0.0
BETA_ BETA_ 9999981_ 0555_ 1-- 721 Error 04/11/13 17:56:48 05/11/13 03:53:35 0.82 28.7 / 0.0
BETA_ BETA_ 9999981_ 0555_ 0-- 721 Error 04/11/13 17:56:47 04/11/13 19:43:07 1.09 21.8 / 0.0

My elapsed time was 3.71 hours. Both _0 and _1 errored with Maximum elapsed time exceeded, but note the Sent Time and Return Time for _0 - between 17:56:47 and 19:43:07 (both 4 Nov) is only 1h 46m (1.77 hours). That is much less than the elapsed time that my _3 took without hitting that error.