Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go ยป
No member browsing this thread
Thread Status: Active
Total posts in this thread: 178
Posts: 178   Pages: 18   [ Previous Page | 9 10 11 12 13 14 15 16 17 18 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 600622 times and has 177 replies Next Thread
Bok7575
Advanced Cruncher
Joined: Jul 8, 2008
Post Count: 51
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

The other good news is that I have also regained access to the badges and subproject data for users signed up so that is working again and I'm running the updates now for Free-DC.
[Jul 13, 2018 8:44:30 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

Finally, a reminder that while access to the data exports is now unrestricted, the export files only include data of volunteers who have opted to make their information public.


I assume then, the stats sites will show a downward trend in team and project historical stats even though it probably doesn't represent reality or will the team and project level data contain the restricted data. If the restricted data is removed from the aggregate level data, then it will be interesting to see if new users allowing data export offset or exceed users restricting data export. With all the "ifs" and "known absences" what reliable conclusions can be drawn from the stats? One team may show a drastic drop just due to the percentage of team members that chose to restrict versus another team with no members that chose to restrict. How does this effect challenge stats? Crunchathons? Pentathalons?
[Jul 13, 2018 9:02:05 PM]   Link   Report threatening or abusive post: please login first  Go to top 
jhindo
Former World Community Grid Admin
Joined: Aug 25, 2009
Post Count: 250
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

Bok7575 - thanks for bringing this to our attention. Try now, the data should be refreshed.
[Jul 13, 2018 9:08:37 PM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

Finally, a reminder that while access to the data exports is now unrestricted, the export files only include data of volunteers who have opted to make their information public.


I assume then, the stats sites will show a downward trend in team and project historical stats even though it probably doesn't represent reality or will the team and project level data contain the restricted data. If the restricted data is removed from the aggregate level data, then it will be interesting to see if new users allowing data export offset or exceed users restricting data export. With all the "ifs" and "known absences" what reliable conclusions can be drawn from the stats? One team may show a drastic drop just due to the percentage of team members that chose to restrict versus another team with no members that chose to restrict. How does this effect challenge stats? Crunchathons? Pentathalons?


Aggregate stats are unaffected since they do not have personal info.

This means that if you look at the stats for a team at the team level (i.e. total stats and team history) those stats include the contribution of both public and private members. However, if you look at the members on a team, those team members who have choosen to be private are simply not included in those lists.

Thus any sort of challenge stats that look at aggregate totals should not be affected.
----------------------------------------
[Edit 1 times, last edit by knreed at Jul 13, 2018 9:12:43 PM]
[Jul 13, 2018 9:12:12 PM]   Link   Report threatening or abusive post: please login first  Go to top 
jhindo
Former World Community Grid Admin
Joined: Aug 25, 2009
Post Count: 250
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

If the restricted data is removed from the aggregate level data, then it will be interesting to see if new users allowing data export offset or exceed users restricting data export. With all the "ifs" and "known absences" what reliable conclusions can be drawn from the stats? One team may show a drastic drop just due to the percentage of team members that chose to restrict versus another team with no members that chose to restrict. How does this effect challenge stats? Crunchathons? Pentathalons?


Aggregate data isn't impacted by GDPR and includes all volunteers' data. Only statistics pages and data exports which include individual-level data are restricted to only volunteers who have agreed to make their data public.
[Jul 13, 2018 9:12:52 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Bok7575
Advanced Cruncher
Joined: Jul 8, 2008
Post Count: 51
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

Confirmed, my stats scripts just downloaded the data. Will watch the update

As info for others here are the before and after numbers from the tables.xml


Before
<update_time>1527100490</update_time>
<nusers_total>541976</nusers_total>
<nteams_total>26209</nteams_total>
<nhosts_total>3608752</nhosts_total>

After
<update_time>1531515841</update_time>
<nusers_total>542051</nusers_total>
<nteams_total>26265</nteams_total>
<nhosts_total>3731828</nhosts_total> <nusers_deleted_total>2538</nusers_deleted_total>
<nhosts_deleted_total>3516</nhosts_deleted_total>

----------------------------------------
[Edit 1 times, last edit by Bok7575 at Jul 13, 2018 9:24:10 PM]
[Jul 13, 2018 9:22:58 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

Finally, a reminder that while access to the data exports is now unrestricted, the export files only include data of volunteers who have opted to make their information public.


I assume then, the stats sites will show a downward trend in team and project historical stats even though it probably doesn't represent reality or will the team and project level data contain the restricted data. If the restricted data is removed from the aggregate level data, then it will be interesting to see if new users allowing data export offset or exceed users restricting data export. With all the "ifs" and "known absences" what reliable conclusions can be drawn from the stats? One team may show a drastic drop just due to the percentage of team members that chose to restrict versus another team with no members that chose to restrict. How does this effect challenge stats? Crunchathons? Pentathalons?


Aggregate stats are unaffected since they do not have personal info.

This means that if you look at the stats for a team at the team level (i.e. total stats and team history) those stats include the contribution of both public and private members. However, if you look at the members on a team, those team members who have choosen to be private are simply not included in those lists.

Thus any sort of challenge stats that look at aggregate totals should not be affected.

So if I add up the total for public team members and subtract from team stats, that should give me the total of private members of the team. I assume that the number of private members is a small percentage versus total members. As a result, it seems reasonable to assume that most teams will only have one or two members that are private. Seems like a backdoor way to identify private members. Whereas, if their data wasn't included in the aggregate it would be a lot harder to identify private members. Brings up another question, If a user starts out as private and joins a team, will anyone know they joined the team? I guess one could argue that if you want to be private, you wouldn't join a team.
----------------------------------------
[Edit 1 times, last edit by Doneske at Jul 13, 2018 10:22:17 PM]
[Jul 13, 2018 10:19:04 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

Thanks jhindo and the rest of the IBM/WCG team! Great to have stats again.
biggrin biggrin biggrin
And thanks to all the patient volunteers who stuck with WCG during "The Great Stats Outage of '18"

Cheers!
[Jul 13, 2018 10:53:16 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

I really admire people who stand up to company lawyers, but to actually WIN -- wow -- well done!
applause
[Jul 13, 2018 11:50:46 PM]   Link   Report threatening or abusive post: please login first  Go to top 
ChristianVirtual
Advanced Cruncher
Japan
Joined: Jan 11, 2014
Post Count: 55
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Data exports & status of Data Processing Agreement

also a big thanks from my side; the solution choose by WCG, BOINC and other participating projects with the additional export files of users/hosts opted out is a good one.
Hope those files are not growing too much after the initial interest of trying that functionality.
----------------------------------------
Active with WCG, GPUGrid, F@H
[Jul 14, 2018 3:17:18 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 178   Pages: 18   [ Previous Page | 9 10 11 12 13 14 15 16 17 18 | Next Page ]
[ Jump to Last Post ]
Post new Thread