Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 36
Posts: 36   Pages: 4   [ Previous Page | 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 5825 times and has 35 replies Next Thread
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2089
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

Ralf:
One thing that I have noticed that is (not) happening since the revival after the SSL certificate f/u, is that validated WUs don't seem to be purged from the result database

Ralf,
Would the implication of this be that the number of results is continually increasing, or to put it another way, not decreasing?
Yes, it has been increasing steadily since about Friday. For me, this has been depending on how many and what kind of WUs (OPNG usually comes in with a couple hundreds at a time sometimes), I used to have between 700 and 1200 result entries in the Result page. Right now, there are 3290, with actually a decrease in WUs in progress (for example, no OPNG since about Friday), and a modest increase in WUs in PVa/PVe jail (a bit over 300) roughly 450 in progress and 2434 valid WUs.

Well, I have been investigating what is going on for my results, just for fun. wink

Between 05:00 UTC and 06:00 UTC this morning my number of validated results that are still in the database increased by one.
But I'm returning results in a much faster pace.
So, you may ask, what is happening here?
When looking a little bit closer I'm seeing this:

Results that disappeared from the database between 05:00 UTC and 06:00 UTC with their ReceivedTime (specified by the server):
< 2022-09-07T13:56:14 OPN1_0111978_00918_0
< 2022-09-12T05:51:37 OPN1_0111954_00402_0
< 2022-09-12T05:13:42 OPNG_0156099_00379_1
< 2022-09-12T05:37:24 OPNG_0156560_00409_0
< 2022-09-12T05:34:07 OPNG_0154441_00437_0
< 2022-09-12T05:38:24 OPNG_0154441_00432_0
< 2022-09-12T05:55:42 OPNG_0154441_00443_0
< 2022-09-12T05:57:52 OPNG_0154441_00426_0
< 2022-09-12T05:25:36 OPNG_0154441_00450_0
< 2022-09-12T05:47:03 OPNG_0154441_00438_0
< 2022-09-12T05:51:25 OPNG_0154441_00410_0
< 2022-09-12T05:23:28 OPNG_0154441_00416_0
< 2022-09-12T05:49:11 OPNG_0154441_00441_0
< 2022-09-12T05:27:44 OPNG_0154441_00419_0
< 2022-09-12T05:31:58 OPNG_0154441_00436_0
< 2022-09-12T05:42:45 OPNG_0154441_00413_0
< 2022-09-12T05:40:32 OPNG_0154441_00411_0
< 2022-09-12T05:18:28 OPNG_0154542_00171_1

Results that showed up for the first time, passed validation between 05:00 UTC and 06:00 UTC, with their ReceivedTime (specified by the server):
> 2022-09-14T05:28:33 OPN1_0112462_02118_0
> 2022-09-07T17:22:29 OPN1_0111319_01388_0
> 2022-09-14T05:05:20 MCM1_0190459_0901_1
> 2022-09-14T05:11:50 MCM1_0190459_0834_0
> 2022-09-14T05:51:56 MCM1_0190459_0894_0
> 2022-09-14T05:47:42 MCM1_0190459_0883_0
> 2022-09-14T05:56:11 MCM1_0190459_0753_1
> 2022-09-14T05:37:07 OPN1_0112451_00601_0
> 2022-09-14T05:28:42 OPN1_0112451_00473_0
> 2022-09-14T05:35:00 OPN1_0112451_00573_0
> 2022-09-14T05:32:53 OPN1_0112451_00538_0
> 2022-09-14T05:30:49 OPN1_0112451_00652_0
> 2022-09-14T05:35:00 OPN1_0112451_00497_0
> 2022-09-14T05:32:53 OPN1_0112451_00574_0
> 2022-09-14T05:08:48 OPNG_0150703_00443_0
> 2022-09-14T05:39:06 OPNG_0150703_00456_0
> 2022-09-14T05:13:58 OPNG_0156551_00034_0
> 2022-09-14T03:49:18 OPNG_0156551_00051_0
> 2022-09-14T05:09:43 OPNG_0156551_00047_0

So I'm seeing results being purged from the database after 48 hours as usual.

Now one might indeed say that's nothing out of the ordinary, but what about the number of results that await validation?
I'm downloading the JSON data hourly (have been doing so for the past five years) and each result contains a line with '"GrantedCredit":', either with the value 0.0 (can be in progress, awaiting validation, error downloading, server aborted, etc.(*1)) or validated with credit.
[*1] I've counted 20 results this month (September 2022) that errored out with either 'download error' (7x) or got Server aborted (13x).
I'm also adding the data for August 2022 from at most 31 days ago.
Measured at 09:00 UTC each day, each line below contains the date of measurement; next to that there are two columns, left column is the number of tasks with zero credit (can be in progress, awaiting validation, error downloading, etc.), right column is the number of results that have claimed credit and are pending validation:

15-08-2022: 91 20
16-08-2022: 170 33
17-08-2022: 148 40
18-08-2022: 128 31
19-08-2022: 597 112
20-08-2022: 789 262
21-08-2022: 576 124
22-08-2022: 631 99
23-08-2022: 819 86
24-08-2022: 773 106
25-08-2022: 523 110
26-08-2022: 319 152
27-08-2022: 354 160
28-08-2022: 636 175
29-08-2022: 586 218
30-08-2022: 723 186
31-08-2022: 1088 202
01-09-2022: 1153 203
02-09-2022: 913 221
03-09-2022: 1249 225
04-09-2022: 920 170
05-09-2022: 904 178
06-09-2022: 707 188
07-09-2022: 1139 168
08-09-2022: 1160 194
09-09-2022: 973 252
10-09-2022: 814 220
11-09-2022: 949 353
12-09-2022: 867 229
13-09-2022: 912 270
14-09-2022: 895 297

So you see, if I would have left out the data from August, the outcome would have been distorted. There is an increase in results that await validation in the past month, but since the past few days I'd say it's more or less fluctuating.

I need to fix a stupid bug that I introduced trying some optimization in my own stats program (based on the Result's page history.csv), then I can tell exactly when which WUs have been purged at what time (and a lot of other things)...

Let us know, Ralf, I'm interested in the kind of glitch that you introduced. cool It's always fun to hear about mistakes, 'cause it's human. biggrin wink

Adri
EDIT: It appears I've been adding up results In Progress and Pending Validation etc., so I have added an extra column with "ValidateState" 0 (= pending validation) for results that have been returned to the server.
----------------------------------------
[Edit 2 times, last edit by adriverhoef at Sep 14, 2022 9:51:28 PM]
[Sep 14, 2022 9:22:15 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 1932
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

I need to fix a stupid bug that I introduced trying some optimization in my own stats program (based on the Result's page history.csv), then I can tell exactly when which WUs have been purged at what time (and a lot of other things)...

Let us know, Ralf, I'm interested in the kind of glitch that you introduced. cool It's always fun to hear about mistakes, 'cause it's human. biggrin wink
Well, making mistakes is indeed human... wink

Haven't had the time yet to look at the code again, but it is likely a combination of a "off by one" error when switching from a temporary fixed size array of records for the WUs to a dynamic array and a mixup of index variable into such array when refactoring reading of the CSV line for each WU record for a single file into a doing so for a whole batch of files (as I didn't think it would be several weeks to run this). That results in a glitch when deciding if a WU needs to be added to the database (dynamic array), an already existing WU needs to be updated due to change of status or if it is an existing WU with status unchanged.
Will likely have time again to look at that (beside my regular work) some time this afternoon, and I can put the code up on Github for those interested (it's written in FreePascal/Lazarus on Windows, but with some minor changes regarding the filenames/directory differences, could be adapted to run on Linux and macOS as well).

As for a quick check on the numbers, what I mentioned before, there are now 383 in PVa/PVe jail, 2624 valid and 516 in progress out of 3554 total WUs in the history.csv, with the remainder is various error states. (that's compared to 350, 2434, 450 and 3290 respectively from my previous post, didn't take note of the exact number of valid/PV WUs back then). So at this point, I can't judge if any increase in WUs with PVa/PVe status is due to something amiss or due to the overall increase in WUs (in progress also increased by about the same ratio).

hth,

Ralf
----------------------------------------

[Sep 14, 2022 3:57:15 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 1932
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

And this morning, despite getting quite a few new WUs over night, there are only 2116 WUs in my Results, which means that about 1500 WUs have been purged from that database over night...

Ralf
----------------------------------------

[Sep 15, 2022 4:24:56 PM]   Link   Report threatening or abusive post: please login first  Go to top 
bfmorse
Senior Cruncher
US
Joined: Jul 26, 2009
Post Count: 294
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

I thought there was about a three day rolling window of availability for finalized data presented in the RESULTS window.
The data does not get deleted it just becomes unavailable for use by the RESULTS search function.
[Sep 15, 2022 8:55:29 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 1932
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

Found my bug and fixed it last night.
Uploaded source (and an executable) to GitHub at

https://github.com/tpcbf4wcg/wcgstats

But be warned, I have still some fixed file path in the source code, which I will fix at a later point, so this serves at this point more to satisfy the curiosity of Adrian... wink

Ralf cool
----------------------------------------

[Sep 16, 2022 6:42:02 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2089
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

Hey Ralf,

So you have a script called 'wcgstats', too? 'Luckily' you used capitals (WCGS) whereas my scripts' names are all in lowercase.smileTo avoid confusion.laughing

More differences I noted, just for fun:

Your script is written in FreePascal, while mine is based on Bash.

Furthermore, you use 'history.csv' to process statistics, while I'm using wcgresults to download results and wcgstats to view results.
The difference here is that wcgresults needs the verificationcode for your account to download any data and wcgstats needs the login-credentials to reach the Results page.

There are many more differences, but I think this is enough for now. wink

Adri
[Sep 17, 2022 2:10:23 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 1932
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

Hey Ralf,

So you have a script called 'wcgstats', too? 'Luckily' you used capitals (WCGS) whereas my scripts' names are all in lowercase.smileTo avoid confusion.laughing
No, I do NOT have a script with that name. It's the name of the GitHub repository. And I don't know why I would be "lucky" here, beside a) the program is written for a non-case sensitive OS b) the actual executable (which is not a script but a native binary executable) is called WCG (or WCG, of WcG, wCg, or however you like to type it).
More differences I noted, just for fun:
You seem to have an odd definition of "fun"... wink
Your script is written in FreePascal, while mine is based on Bash.
And? Well, for all practical purposes, bash doesn't run on Windows...
And FreePascal/Lazarus runs on a multitude of OS and environments,and Windows 10 is what I have on all of my every day work computers. And it would take only minor adjustments for file/pathnames in the source to transfer and recompile the program on a GUI Linux or macOS.
Furthermore, you use 'history.csv' to process statistics, while I'm using wcgresults to download results and wcgstats to view results.
The difference here is that wcgresults needs the verificationcode for your account to download any data and wcgstats needs the login-credentials to reach the Results page.
Well, I didn't use that file name, that is the filename that the download procedure on the Results page offers. And that was the easiest to acquire source of the data that I need for my intents and purposes...
There are many more differences, but I think this is enough for now. wink
No surprise, as we are likely having different usages for our respective programs. I just posted it up on GitHub, as you seem to be interested in seeing the source code, unless that was a snide from your side implying that I wouldn't have any such program to begin with...

Ralf

PS: updates to the program are likely coming some time Sunday afternoon...
----------------------------------------

----------------------------------------
[Edit 3 times, last edit by TPCBF at Sep 17, 2022 8:19:09 PM]
[Sep 17, 2022 8:15:50 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2089
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

I just posted it up on GitHub, as you seem to be interested in seeing the source code,
Well, I was just interested in what language you used in the first place as I was busy tweaking and testing my own wcgstats program. wink

And you're right, the name of your program was not WCGStats, it was part of the name of the ZIP-file (and directory after unpacking) that I noticed and that stuck with me. biggrin

bash doesn't run on Windows...
I have heard of Cygwin.
YouTube-link: "Cygwin: It's Bash, but on Windows..."
----------------------------------------
[Edit 1 times, last edit by adriverhoef at Sep 17, 2022 11:50:09 PM]
[Sep 17, 2022 11:41:37 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 1932
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Bandwidth

bash doesn't run on Windows...
I have heard of Cygwin.
YouTube-link: "Cygwin: It's Bash, but on Windows..."
Yeah, I heard of that one too. But why would I install such a behemoth if I a) have a proper GUI RAD tool already installed and b) can run the resulting program just natively, without any additional dependency... wink

Btw, I do have a question for you (and possible other long time, multi-host crunchers): Have you ever had the case where one of your hosts turned out to be the wing man for another one of your's? So far, I process WUs by processing the "workunitId", only to just realize that there could be the case where a WU shows up twice (or more), with different status, because two different devices could get the same WU, with the same "workunitId" but with a different "resultId."

Ralf
----------------------------------------

[Sep 20, 2022 8:36:44 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 36   Pages: 4   [ Previous Page | 1 2 3 4 | Next Page ]
[ Jump to Last Post ]
Post new Thread