Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 115
Posts: 115   Pages: 12   [ Previous Page | 1 2 3 4 5 6 7 8 9 10 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 482248 times and has 114 replies Next Thread
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1684
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

Hi everybody,
even if I am a strong supporter of GDPR and advocating for it, I think that what we experience during the last days is a complete pity and mess.
GDPR has been released in April 2016, with an effective date from 2018-05-25, i.e. today.
It gave over 2 years of time for elaborating/designing GDPR compliant solutions, to implement and to test them, to write some documentation / instruction, to release the new functionality, keeping the old approach available, informing users and to invite them to switch to the new implementation before 2018-05-25.
It's simply called: Good Project Management Practice and Good Software Engineering Practice !!!
Instead of such well planned approach, I received during the last 3 days over 200 "business" e-mails requiring some kind of confirmation, and WCG broke the statistic report. angry angry angry
I usually support the WCG tech team and its hard work, but not today evening, after an exhausting business week and just returning back home.
How could you so short time react without any kind of announcement, without any technical explanation, and with buggy help page?

It is a real pity !!!
Yves
----------------------------------------
----------------------------------------
[Edit 1 times, last edit by KerSamson at May 25, 2018 8:33:43 PM]
[May 25, 2018 8:08:49 PM]   Link   Report threatening or abusive post: please login first  Go to top 
alanb1951
Veteran Cruncher
Joined: Jan 20, 2006
Post Count: 1316
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

The changes to enable the following are now live and available for use. This will allow you to access your own data via these api's if you are logged into the website.

I can confirm that (after reprogramming to handle cookies) my Python scripts work again.
Thank you very much for providing a nice simple mechanism for "own data only" users in the post-GDPR world.
[May 25, 2018 9:40:39 PM]   Link   Report threatening or abusive post: please login first  Go to top 
wujj123456
Cruncher
Joined: Jun 9, 2010
Post Count: 38
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

The changes to enable the following are now live and available for use. This will allow you to access your own data via these api's if you are logged into the website.

Those API's that access the data for one user only, we are going to open up again if you have logged into the website.

This would look like the following if you are using wget to access the verification API.

# Login
wget --save-cookies cookies.txt \
--keep-session-cookies \
--post-data 'j_username=XYZ&j_password=ABC' \
--delete-after \
https://www.worldcommunitygrid.org/j_security_check


# Go to API
wget --load-cookies cookies.txt \
https://www.worldcommunitygrid.org/api/members/{member name}/results?code={verification code}


The following API's will be supported with this:


API's that return data for more than just your user or using the API's to get data for other users will require going through the process to get a token.

I really don't feel it's a good practice to put password anywhere in a script. It should really be a token of limited purpose even if it's accidentally exposed.

I've already asked this, but I will try one last time. What's the point of "verification code" we had to pass in even before this change? Isn't that enough to make sure only I can view my own data given only I can see the verification code after I log in?

Honestly I am more confused by this than I am upset by the sudden breaking change to the stats API.
[May 26, 2018 12:33:58 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 286
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

adriverhoef
I have also updated my script 'wcgresults' to fetch your own personal Results from the WCG website via the API:
$ wcgresults -daf WCG.log
You can download the latest version of 'wcgresults' here.


Thanks - downloaded a copy and working again, and thanks again for making the script available Decided to tidy up my hurried 'work-around' and run that on another system - so now have redundancy blushing
----------------------------------------
[May 26, 2018 1:33:46 AM]   Link   Report threatening or abusive post: please login first  Go to top 
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1684
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

After a couple of hours looking back back at WCG and GDPR, I think what is currently happening is a complete overreaction and misinterpretation of GDPR objectives.
  • What member privacy relevant data does WCG manage?
  • What member data are shared within the scope of member statistics export?

In case of additional technical controls are really required, how must the additional controls be designed and implemented in order:
  • to limit the implementation effort (KISS principle)?
  • to limit the negative impact on web sites consolidating member statistics, e.g. BoincStats, FreeDC, ..., as well as services providing signatures?

Regarding items #a and #b, the additional controls must not have any negative impacts on the operation of external services; ideally it should not have any impact.

Back to items #1 and #2:
#1 - What member privacy relevant data does WCG manage?
- User name
- WCG registration date
- E-mail address
- Team e-mail address
- Country
- Account number
- Account key
- Weak account key
- Cross-project ID
- Team ID
- external authorization code
- Device name
- Device ID
- Contributed projects (over the result report)
- Time when the work unit (result) has been downloaded, computed, and sent
- User connection data on WCG web site
- Forum activities (read) and contributions

#2 - What member data are shared within the scope of member statistics export?
- Name (user name)
- Account number (member ID)
- Number of devices
- Cumulative contribution statistics
- Cumulative project contribution statistics (only for statistics available for the member)

I do not pretend that I know every single data collected and managed by WCG, but the above lists should be very close to the reality.
How should GDPR be understood and applied?
  • WCG duties related to the members
    • Transparency: WCG should communicate clearly to its members what member related data are managed and maintained by WCG.
    • Ability to correct data: a WCG member should be able to modify its registration/profile data (where it is meaningful)
    • Ability to erase data: a WCG member should be able to delete its account
    • Ability to decide: a WCG member should be able to decide what data s/he is willing to share with third party (in particular within the scope of contribution/statistics export

  • WCG duties related to data storage and management
    • Necessity to elaborate and to publish a "GDPR compliant" privacy policy
    • Necessity to fetch formal member's consent: what data the member accept to share and not to share with third party.
    • Protection by design and default: members have to "opt-in" (opt-out by default)
    • Nomination of a Data Protection Officer (DPO)
    • Necessity to maintain records of data processing activities
    • Obligation to report data breaches

In case of a member would not accept to share its contribution statistics with third party, pseudonymisation should be used in order for WCG to be able to report the "complete picture" without compromising member data privacy.
Today (and since many years), WCG asks if the member does accept to show its hosts. Likewise, the detailed project contribution is not available for third party.
The only required functionality is to enforce / to enable member's opt-in for sharing its contribution statistics with external web sites; otherwise its contribution will pseudonymised reported.
---
Instead to create a huge chaos increasing the export complexity and breaking operational processes by other sites, it would have been adequate to think, to propose, and to share the needed changes with the members and with the third party web sites prospectively.
There was 25 months available for preparing the transition, not only one night !!!
I am not a lawyer, but since I follow GDPR since a lot of time (and GDPR is part of the e-compliance trainings I give to the pharmaceutical industry), the above advices should not be so far away from the expected reality. Regarding the GDPR implementation disaster at WCG during the last days, I assume that some "poorly knowledgeable" lawyers misinterpret/overinterpret the GDPR requirements and cause "last minute patch".
We are here a lot of members with various knowledge in different areas. We are available not only for contributing to the projects, but for contributing to discussions as well. We are able to help WCG reviewing ideas, improvements, innovations, and compliance.
Please restore the export process to third party as soonest as possible, without creating a lot of useless work on third party's side!
Cheers,
Yves
---
PS: Comments, improvements, and remarks are welcome.
----------------------------------------
[May 27, 2018 8:52:14 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Unable to Use API

Yves,

I, too, am not a lawyer, but also have an interest and have been following things, as well as having read a large part of the regulation. I think you did a pretty good job of stating things. The most important thing that you picked up on, as far as I can see, is:

In case of a member would not accept to share its contribution statistics with third party, pseudonymisation should be used in order for WCG to be able to report the "complete picture" without compromising member data privacy.


GDPR specifically allows for this in article 89 and it would seem to be a fairly simple solution.
[May 27, 2018 10:09:14 AM]   Link   Report threatening or abusive post: please login first  Go to top 
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1684
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

Hi Apis Tintinnambulator,
dealing with regulatory compliance requirements and issues since about 25 years, I am really convinced that in the most cases, the approach and its implementation should be as simple (not simpler) as possible (Einstein's principle).
GDPR compliance does not represent any exception at this level.
It is one of the reasons why I am really frustrated (and mad) to read and to listen a lot of people complaining instead of trying to read and to understand the requirements.
Frankly, I did not expect that WCG would have failed so strongly (and badly) about this topic.
Cheers,
Yves
----------------------------------------
[May 27, 2018 11:56:46 AM]   Link   Report threatening or abusive post: please login first  Go to top 
ChristianVirtual
Advanced Cruncher
Japan
Joined: Jan 11, 2014
Post Count: 55
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

Thanks KerSamson for the summary;

When I see my own data exposed in the "classical" XML stats exports for user I get the following

<user>
<id>887739</id>
<name>ChristianVirtual</name>
<country>JAPAN</country>
<create_time>1389411877</create_time>
<total_credit>10175511.812051</total_credit>
<expavg_credit>6737.305417</expavg_credit>
<expavg_time>1527077745.878393</expavg_time>
<cpid>f2f3f3a1a583c7554a6f8862ac944238</cpid>
<teamid>1411</teamid>
</user>

If a user would like to opt-out of XML export and public-domain of the data I would suggest the following

<user>
<id>887739</id>

<name>hidden</name>
<country>hidden</country>
<create_time>hidden</create_time>

<total_credit>10175511.812051</total_credit>
<expavg_credit>6737.305417</expavg_credit>
<expavg_time>1527077745.878393</expavg_time>
<cpid>hidden</cpid>
<teamid>1411</teamid>
</user>

The bare minimum for Stats sites are the ID, total_credit, expavg_time and teamed;

name, country, creation_time and in a worst case also CPID are not needed. CPID though removes the users ability to get himself a comprehensive overview.

This way I really don't see a negative impact for privacy but keep a bare minimum on usable data available to do challenges and statistics ....
----------------------------------------
Active with WCG, GPUGrid, F@H
----------------------------------------
[Edit 2 times, last edit by ChristianVirtual at May 27, 2018 12:40:44 PM]
[May 27, 2018 12:39:32 PM]   Link   Report threatening or abusive post: please login first  Go to top 
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1684
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

In order to preserve the value of the statistics and its consolidation, I would prefer to pseudonymise the user ID (account number) but I would keep country and team.
As soon as the link between the "pseudonymised" user ID and the account number is not accessible for third party, the pseudonymisation is effective.
Only WCG system administrators would be able to see the link between the pseudo user ID and the real account number.
If a member, because of the limited team size, wish to hide additionally its team membership, the member contribution will not be assigned anymore to the team outside of WCG.
Likewise, if a member, because of the limited country size, wish to hide additionally its country, the member contribution will not be assigned anymore to a country outside of WCG.
Already today, there are still many members who did select neither a country nor a team.
Yves
----------------------------------------
[May 27, 2018 12:58:26 PM]   Link   Report threatening or abusive post: please login first  Go to top 
KLiK
Master Cruncher
Croatia
Joined: Nov 13, 2006
Post Count: 3108
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Unable to Use API

The Byteball distribution is broken cause of this issue. I hope you will give them quickly an access.

Gridcoin team, one of top15 teams, are also having same problems. Unfortunately!
----------------------------------------
oldies:UDgrid.org & PS3 Life@home


non-profit org. Play4Life in Zagreb, Croatia
[May 28, 2018 5:22:26 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 115   Pages: 12   [ Previous Page | 1 2 3 4 5 6 7 8 9 10 | Next Page ]
[ Jump to Last Post ]
Post new Thread