Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 115
Posts: 115   Pages: 12   [ Previous Page | 2 3 4 5 6 7 8 9 10 11 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 482246 times and has 114 replies Next Thread
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 286
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

When the error occurred initially, it created 10 new databases for me with some of the words from the error messages :-) After cleaning this mess up, and from the words in those corrupt databases had a good idea of what was happening. Parsed the output looking for 'denied' and re-directed the output to a log - a sample was provided above... The database update then uses the output of the previous good return - marking time :-)

However, on checking the logs just now have not had a failure over the last 24 hours... problem was transient and gone?

If you want me to run a modified script - am happy to help where I can...
----------------------------------------
[Jun 2, 2018 4:24:40 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Jim1348
Veteran Cruncher
USA
Joined: Jul 13, 2009
Post Count: 1066
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

If you are accessing data about other users, then GDPR puts an obligation on us to ensure that anyone who access this data (those people are called 3rd party processors) complies with certain obligations including if a user requests that their account is deleted on our site, then that 3rd party processor must also delete the data about the user. GDPR requires that we put it into a legally binding contract and thus anyone who wants to access to data about other users will have to sign the agreement in order to get the authentication tokens necessary to access that data.

There should be (but probably isn't) a way to allow users to waive that requirement, just as they can allow export of their own information to a 3rd party website. If there were such a way, the IBM/WCG lawyers would have probably found it by now. But maybe you can get the EU to modify the regulation?

I really don't see much need for privacy protection for WCG data, or a lot of other publc-interest data. It places quite a restriction on the free interchange of information.
----------------------------------------
[Edit 1 times, last edit by Jim1348 at Jun 2, 2018 12:50:49 PM]
[Jun 2, 2018 12:46:21 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

@TonyEllis
Sorry if this mess is what has happened to you, Tony. I understand part of what has happened: instead of writing the message "Errors detected: etc." to stderr my script has been writing the error messages to stdout. Also, if login fails then the script should stop there. This is corrected now and I've uploaded an updated version of my script. You can check if login works by running 'wcgresults -dw'. wink
[Jun 2, 2018 1:26:04 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 286
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

adriverhoef - thanks for that...
This is corrected now and I've uploaded an updated version of my script. You can check if login works by running 'wcgresults -dw'. wink

Thanks for the update - sorry to worry you if you already read the previous contents of this append - have edited this append as have have found the problem. The order of the credentials was altered and missed the 'MEMBER" option which had been moved down lower - all is well...
----------------------------------------
----------------------------------------
[Edit 1 times, last edit by TonyEllis at Jun 2, 2018 3:28:09 PM]
[Jun 2, 2018 3:08:56 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

Tony,
sorry to worry you if you already read the previous contents of this append
Luckily I missed that. wink
all is well...
Wonderful! Just wonderful. blushing
----------------------------------------
[Edit 1 times, last edit by adriverhoef at Jun 2, 2018 4:28:37 PM]
[Jun 2, 2018 4:27:46 PM]   Link   Report threatening or abusive post: please login first  Go to top 
keithhenry
Ace Cruncher
Senile old farts of the world ....uh.....uh..... nevermind
Joined: Nov 18, 2004
Post Count: 18667
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

Well, I've been pulled into the GDPR/API issues now. Yesterday, a retired member of my team "disappeared" from the HTML pages. No, I haven't been using the APIs. I have a "legacy" screen-scraping process.

I think I understand the intent behind GDPR. Basically, the EU is trying to impose a rather dramatic paradigm shift. Data about a user collected by others is the "property" of that user and its use fully controlled by that user. That's pretty much the exact opposite of how data collection/management/etc. has become. Thank social media for that. Imagine being able to tell FB that they can collect data on you solely to enable you to use their site but they cannot sell it in any personally identifiable form to anyone. I think the spirit of GDPR is to in fact make such a scenario a real possibility. For businesses who exist to sell data about you, that's a veritable sword of Damocles. It's rather appealing to me personally.

The fact the IBM/WCG has been working on the GDPR issue for quite a while is no surprise at all. IBM's sizable legal staff exists for issues like this. It BAU for them. Other projects are usually run by at best a handful of research assistants/ graduate assistants for some professor in some academic setting. They will eventually have to address the GDPR issue and they will do so in the simplest/cheapest/least work way possible. It will be in a reactive mode though. IBM's sizable legal staff is there to ensure issues like this get addressed proactively - before some given deadline or point in time where not having addressed the issue could adversely affect IBM. Also, some professor running a BOINC project to do research has a MUCH lower profile that IBM. IBM is a VERY big target and quite visible. For its legal staff, it's not just about complying with legal requirements, it's about risk management (ie. eliminating it whenever possible and minimizing it the rest of the time). This is why there is no effective means of us volunteers communicating with each other. All the other BOINC projects have PM available as it is a part of the BOINC software. WCG disabled that. Someone in legal way back when told them they couldn't allow it. IBM could not risk some nutcase using the PM function in BOINC to stalk a victim or some other similar scenario where that victim could attempt to go after IBM for providing a means to the nutcase.

By defaulting new volunteers to being excluded from any data exports, we have been shot with another silver bullet. Yes, this complies with the GDPR requirements I'm sure but I doubt GDPR specifically requires such exclusion by default. You just have to give the user to option to be excluded. They are being excluded to address risk issues, not legal issues. New volunteers are NOT going the understand the whole area of data export, not typically anyway. How many will choose to opt in when registering? Go look at the forum user list (hmm, how long will that still be around?) and see how many of the miniscule number of volunteers that actually use the forums have provided an email address. So, as time progresses, we are going to see an ever declining number of users showing up in the data exports. Sites like FreeDC, BOINCStats and such - well, they are not dead yet but their fate is sealed. I fear that pretty much the same fate awaits TEAMS. If not opting in to the data export prevents any data about a user from going outside of WCG, I think it is only a matter of time before that also means no data about that user will be included in the team-related HTML pages. If it's in the HTML pages available to persons other than the specific individual, well, you can't prevent screen scrapers from getting that data. So, as data for new volunteers gets excluded from the HTML pages (it's the exact same risk that exists if they are in the API data export), it will become increasingly pointless to be a part of or even have teams.

WCG is like any non-profit organization - without a committed and dedicated core of volunteers that support its existence, mission, goals and each other, it withers away and dies. It won't happen tomorrow or next year. It is matter of time though. The really scary part is that as it becomes ever increasingly difficult to welcome new volunteers, encourage them and build a sense of community and comraderie with them, their time here will be shorter or less productive at best. That will translate into less work getting done that would have otherwise and less work means longer timeframes to research and results being discovered which in turn means more people with die or suffer from diseases because the cures will take longer to come.

God I hope I am wrong.
----------------------------------------
Join/Website/IMODB



[Jun 5, 2018 1:04:55 AM]   Link   Report threatening or abusive post: please login first  Go to top 
PecosRiverM
Veteran Cruncher
The Great State of Texas
Joined: Apr 27, 2007
Post Count: 1054
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

+1

coffee
cowboy
----------------------------------------

[Jun 5, 2018 1:30:52 AM]   Link   Report threatening or abusive post: please login first  Go to top 
wujj123456
Cruncher
Joined: Jun 9, 2010
Post Count: 38
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

I read a bit more about GDPR since WCG broke my stats script, and so far it seems that pseudonymization would be sufficient for WCG so long as email and other personal information are not exported along with stats. Maybe the risk of pseudonymization being linked back to identity is too high when viewed by the eyes of lawyers?

I would second on sharing data and stats are part of spirit of volunteer computing. It would be sad if all these info has to be hidden from the public by default now. (Heck this is not even just by default here. I have no way of setting my privacy to public so that I don't have to login each time to get my own stats.)
[Jun 5, 2018 3:15:25 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Unable to Use API

By defaulting new volunteers to being excluded from any data exports, we have been shot with another silver bullet. Yes, this complies with the GDPR requirements I'm sure but I doubt GDPR specifically requires such exclusion by default. You just have to give the user to option to be excluded. They are being excluded to address risk issues, not legal issues. New volunteers are NOT going the understand the whole area of data export, not typically anyway. How many will choose to opt in when registering?


As I read the GDPR, the problem is that a web site can no longer allow sharing personal data by default. That means that a check box asking the user if they agree to data sharing CANNOT be pre-ticked. There is no middle ground, so the default must be to exclude data sharing by default.

BUT. It's also important to explain why data is collected and what it is used for. There is nothing wrong with strong words of encouragement to allow data sharing (just as it would be a good idea to strongly dissuade people from registering with their real names) but I think this is even more important when joining a team. The global WCG community is so large that I completely agree with WCG's actions to protect people, but I think team membership is another matter. Teams are all about encouraging chatter and gentle competition amongst members (as well as competition between teams) and for that data sharing is required. But that, in turn, puts GDPR requirements on whoever is doing that processing, and WCG is right to have an agreement between itself and those processing team data -- it is a requirement of the GDPR that they do so. But once that is in place, I don't personally see anything wrong with making data sharing a requirement of team membership. Joining a team is optional, so provided that members understand that joining a team implies allowing data sharing amongst the team (and, perhaps optionally, more widely), job done. (Of course, that might mess up our current ability to send friends a sign-up link which includes automatic team membership at sign up, but it's only code after all.)

As for BOINCStats and the others, then they need to comply with the GDPR too. It is quite right for IBM not to export data to them until they comply. Personally I hope that they do comply and that they continue to operate. If they don't comply, then it might be better if they close. But I also think the comment about risk is a good one. That is also what GDPR is about. And here we do have a clash between data which is (in my opinion) low risk if "lost", and a global company that is risk averse. But shouldn't we all err on the side of best practice?

The more we discuss this, the more I see that two years wasn't very long. And the thought of getting Brexit right in the same time-frame makes me quake!
[Jun 5, 2018 9:22:26 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Unable to Use API

"It is quite right for IBM not to export data to them until they comply."

The -not- exported summary data of WCG seems to infer a material beating right after May 25 (a Friday).

https://www.worldcommunitygrid.org/stat/viewGlobalHistory.do
06/05/2018		2:034:05:50:23		        3,437,676		5,736
06/04/2018 495:159:15:47:20 807,210,160 1,341,412
06/03/2018 461:295:21:24:44 744,716,941 1,221,267
06/02/2018 470:345:06:46:44 764,694,282 1,279,842
06/01/2018 489:152:06:43:11 781,738,456 1,298,467
05/31/2018 486:354:03:34:31 790,787,248 1,317,689
05/30/2018 494:104:14:47:27 801,929,052 1,321,931
05/29/2018 491:347:20:42:51 799,535,379 1,315,285
05/28/2018 487:330:03:10:33 791,004,240 1,314,496
05/27/2018 477:206:08:59:59 765,896,058 1,278,676
05/26/2018 476:276:07:56:45 763,640,320 1,287,644
05/25/2018 507:300:08:11:21 824,739,802 1,389,088
05/24/2018 523:046:14:36:38 849,953,941 1,440,706
05/23/2018 518:103:23:35:21 846,663,988 1,405,091

----------------------------------------
[Edit 1 times, last edit by Former Member at Jun 5, 2018 9:55:42 AM]
[Jun 5, 2018 9:53:56 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 115   Pages: 12   [ Previous Page | 2 3 4 5 6 7 8 9 10 11 | Next Page ]
[ Jump to Last Post ]
Post new Thread