Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
World Community Grid Forums
Category: Support Forum: Website Support Thread: Bandwidth |
No member browsing this thread |
Thread Status: Active Total posts in this thread: 36
|
Author |
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2089 Status: Offline Project Badges: |
Ralf: Yes, it has been increasing steadily since about Friday. For me, this has been depending on how many and what kind of WUs (OPNG usually comes in with a couple hundreds at a time sometimes), I used to have between 700 and 1200 result entries in the Result page. Right now, there are 3290, with actually a decrease in WUs in progress (for example, no OPNG since about Friday), and a modest increase in WUs in PVa/PVe jail (a bit over 300) roughly 450 in progress and 2434 valid WUs.One thing that I have noticed that is (not) happening since the revival after the SSL certificate f/u, is that validated WUs don't seem to be purged from the result database Ralf, Would the implication of this be that the number of results is continually increasing, or to put it another way, not decreasing? Well, I have been investigating what is going on for my results, just for fun. Between 05:00 UTC and 06:00 UTC this morning my number of validated results that are still in the database increased by one. But I'm returning results in a much faster pace. So, you may ask, what is happening here? When looking a little bit closer I'm seeing this: Results that disappeared from the database between 05:00 UTC and 06:00 UTC with their ReceivedTime (specified by the server): < 2022-09-07T13:56:14 OPN1_0111978_00918_0 < 2022-09-12T05:51:37 OPN1_0111954_00402_0 < 2022-09-12T05:13:42 OPNG_0156099_00379_1 < 2022-09-12T05:37:24 OPNG_0156560_00409_0 < 2022-09-12T05:34:07 OPNG_0154441_00437_0 < 2022-09-12T05:38:24 OPNG_0154441_00432_0 < 2022-09-12T05:55:42 OPNG_0154441_00443_0 < 2022-09-12T05:57:52 OPNG_0154441_00426_0 < 2022-09-12T05:25:36 OPNG_0154441_00450_0 < 2022-09-12T05:47:03 OPNG_0154441_00438_0 < 2022-09-12T05:51:25 OPNG_0154441_00410_0 < 2022-09-12T05:23:28 OPNG_0154441_00416_0 < 2022-09-12T05:49:11 OPNG_0154441_00441_0 < 2022-09-12T05:27:44 OPNG_0154441_00419_0 < 2022-09-12T05:31:58 OPNG_0154441_00436_0 < 2022-09-12T05:42:45 OPNG_0154441_00413_0 < 2022-09-12T05:40:32 OPNG_0154441_00411_0 < 2022-09-12T05:18:28 OPNG_0154542_00171_1 Results that showed up for the first time, passed validation between 05:00 UTC and 06:00 UTC, with their ReceivedTime (specified by the server): > 2022-09-14T05:28:33 OPN1_0112462_02118_0 > 2022-09-07T17:22:29 OPN1_0111319_01388_0 > 2022-09-14T05:05:20 MCM1_0190459_0901_1 > 2022-09-14T05:11:50 MCM1_0190459_0834_0 > 2022-09-14T05:51:56 MCM1_0190459_0894_0 > 2022-09-14T05:47:42 MCM1_0190459_0883_0 > 2022-09-14T05:56:11 MCM1_0190459_0753_1 > 2022-09-14T05:37:07 OPN1_0112451_00601_0 > 2022-09-14T05:28:42 OPN1_0112451_00473_0 > 2022-09-14T05:35:00 OPN1_0112451_00573_0 > 2022-09-14T05:32:53 OPN1_0112451_00538_0 > 2022-09-14T05:30:49 OPN1_0112451_00652_0 > 2022-09-14T05:35:00 OPN1_0112451_00497_0 > 2022-09-14T05:32:53 OPN1_0112451_00574_0 > 2022-09-14T05:08:48 OPNG_0150703_00443_0 > 2022-09-14T05:39:06 OPNG_0150703_00456_0 > 2022-09-14T05:13:58 OPNG_0156551_00034_0 > 2022-09-14T03:49:18 OPNG_0156551_00051_0 > 2022-09-14T05:09:43 OPNG_0156551_00047_0 So I'm seeing results being purged from the database after 48 hours as usual. Now one might indeed say that's nothing out of the ordinary, but what about the number of results that await validation? I'm downloading the JSON data hourly (have been doing so for the past five years) and each result contains a line with '"GrantedCredit":', either with the value 0.0 (can be in progress, awaiting validation, error downloading, server aborted, etc.(*1)) or validated with credit. [*1] I've counted 20 results this month (September 2022) that errored out with either 'download error' (7x) or got Server aborted (13x). I'm also adding the data for August 2022 from at most 31 days ago. Measured at 09:00 UTC each day, each line below contains the date of measurement; next to that there are two columns, left column is the number of tasks with zero credit (can be in progress, awaiting validation, error downloading, etc.), right column is the number of results that have claimed credit and are pending validation:
So you see, if I would have left out the data from August, the outcome would have been distorted. There is an increase in results that await validation in the past month, but since the past few days I'd say it's more or less fluctuating. I need to fix a stupid bug that I introduced trying some optimization in my own stats program (based on the Result's page history.csv), then I can tell exactly when which WUs have been purged at what time (and a lot of other things)... Let us know, Ralf, I'm interested in the kind of glitch that you introduced. It's always fun to hear about mistakes, 'cause it's human. Adri EDIT: It appears I've been adding up results In Progress and Pending Validation etc., so I have added an extra column with "ValidateState" 0 (= pending validation) for results that have been returned to the server. [Edit 2 times, last edit by adriverhoef at Sep 14, 2022 9:51:28 PM] |
||
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1932 Status: Offline Project Badges: |
I need to fix a stupid bug that I introduced trying some optimization in my own stats program (based on the Result's page history.csv), then I can tell exactly when which WUs have been purged at what time (and a lot of other things)... Let us know, Ralf, I'm interested in the kind of glitch that you introduced. It's always fun to hear about mistakes, 'cause it's human. Haven't had the time yet to look at the code again, but it is likely a combination of a "off by one" error when switching from a temporary fixed size array of records for the WUs to a dynamic array and a mixup of index variable into such array when refactoring reading of the CSV line for each WU record for a single file into a doing so for a whole batch of files (as I didn't think it would be several weeks to run this). That results in a glitch when deciding if a WU needs to be added to the database (dynamic array), an already existing WU needs to be updated due to change of status or if it is an existing WU with status unchanged. Will likely have time again to look at that (beside my regular work) some time this afternoon, and I can put the code up on Github for those interested (it's written in FreePascal/Lazarus on Windows, but with some minor changes regarding the filenames/directory differences, could be adapted to run on Linux and macOS as well). As for a quick check on the numbers, what I mentioned before, there are now 383 in PVa/PVe jail, 2624 valid and 516 in progress out of 3554 total WUs in the history.csv, with the remainder is various error states. (that's compared to 350, 2434, 450 and 3290 respectively from my previous post, didn't take note of the exact number of valid/PV WUs back then). So at this point, I can't judge if any increase in WUs with PVa/PVe status is due to something amiss or due to the overall increase in WUs (in progress also increased by about the same ratio). hth, Ralf |
||
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1932 Status: Offline Project Badges: |
And this morning, despite getting quite a few new WUs over night, there are only 2116 WUs in my Results, which means that about 1500 WUs have been purged from that database over night...
----------------------------------------Ralf |
||
|
bfmorse
Senior Cruncher US Joined: Jul 26, 2009 Post Count: 294 Status: Offline Project Badges: |
I thought there was about a three day rolling window of availability for finalized data presented in the RESULTS window.
The data does not get deleted it just becomes unavailable for use by the RESULTS search function. |
||
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1932 Status: Offline Project Badges: |
Found my bug and fixed it last night.
----------------------------------------Uploaded source (and an executable) to GitHub at https://github.com/tpcbf4wcg/wcgstats But be warned, I have still some fixed file path in the source code, which I will fix at a later point, so this serves at this point more to satisfy the curiosity of Adrian... Ralf |
||
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2089 Status: Offline Project Badges: |
Hey Ralf,
So you have a script called 'wcgstats', too? 'Luckily' you used capitals (WCGS) whereas my scripts' names are all in lowercase.To avoid confusion. More differences I noted, just for fun: Your script is written in FreePascal, while mine is based on Bash. Furthermore, you use 'history.csv' to process statistics, while I'm using wcgresults to download results and wcgstats to view results. The difference here is that wcgresults needs the verificationcode for your account to download any data and wcgstats needs the login-credentials to reach the Results page. There are many more differences, but I think this is enough for now. Adri |
||
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1932 Status: Offline Project Badges: |
Hey Ralf, No, I do NOT have a script with that name. It's the name of the GitHub repository. And I don't know why I would be "lucky" here, beside a) the program is written for a non-case sensitive OS b) the actual executable (which is not a script but a native binary executable) is called WCG (or WCG, of WcG, wCg, or however you like to type it).So you have a script called 'wcgstats', too? 'Luckily' you used capitals (WCGS) whereas my scripts' names are all in lowercase.To avoid confusion. More differences I noted, just for fun: You seem to have an odd definition of "fun"... Your script is written in FreePascal, while mine is based on Bash. And? Well, for all practical purposes, bash doesn't run on Windows...And FreePascal/Lazarus runs on a multitude of OS and environments,and Windows 10 is what I have on all of my every day work computers. And it would take only minor adjustments for file/pathnames in the source to transfer and recompile the program on a GUI Linux or macOS. Furthermore, you use 'history.csv' to process statistics, while I'm using wcgresults to download results and wcgstats to view results. Well, I didn't use that file name, that is the filename that the download procedure on the Results page offers. And that was the easiest to acquire source of the data that I need for my intents and purposes...The difference here is that wcgresults needs the verificationcode for your account to download any data and wcgstats needs the login-credentials to reach the Results page. There are many more differences, but I think this is enough for now. No surprise, as we are likely having different usages for our respective programs. I just posted it up on GitHub, as you seem to be interested in seeing the source code, unless that was a snide from your side implying that I wouldn't have any such program to begin with...Ralf PS: updates to the program are likely coming some time Sunday afternoon... [Edit 3 times, last edit by TPCBF at Sep 17, 2022 8:19:09 PM] |
||
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2089 Status: Offline Project Badges: |
I just posted it up on GitHub, as you seem to be interested in seeing the source code, Well, I was just interested in what language you used in the first place as I was busy tweaking and testing my own wcgstats program. And you're right, the name of your program was not WCGStats, it was part of the name of the ZIP-file (and directory after unpacking) that I noticed and that stuck with me. bash doesn't run on Windows... I have heard of Cygwin.YouTube-link: "Cygwin: It's Bash, but on Windows..." [Edit 1 times, last edit by adriverhoef at Sep 17, 2022 11:50:09 PM] |
||
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1932 Status: Offline Project Badges: |
bash doesn't run on Windows... I have heard of Cygwin.YouTube-link: "Cygwin: It's Bash, but on Windows..." Btw, I do have a question for you (and possible other long time, multi-host crunchers): Have you ever had the case where one of your hosts turned out to be the wing man for another one of your's? So far, I process WUs by processing the "workunitId", only to just realize that there could be the case where a WU shows up twice (or more), with different status, because two different devices could get the same WU, with the same "workunitId" but with a different "resultId." Ralf |
||
|
|