World Community Grid - View Thread - API for devices list

World Community Grid Forums

Category: Support

Forum: Website Support

Thread: API for devices list

Quick Go »

No member browsing this thread

Thread Status: Active
Total posts in this thread: 15

[ ]

Author

This topic has been viewed 4962 times and has 14 replies

AntHill
Cruncher
Joined: Nov 3, 2017
Post Count: 1
Status: Offline
Project Badges:

200 year badge for Mapping Cancer Markers

200 year badge for Outsmart Ebola Together

200 year badge for FightAIDS@Home - Phase 2

100 year badge for Smash Childhood Cancer

200 year badge for Microbiome Immunity Project

14 day badge for Africa Rainfall Project

200 year badge for OpenPandemics - COVID-19


API for devices list

How to use APIs to get a list of devices with statistics, like on https://www.worldcommunitygrid.org/ms/device/viewStatisticsByDevice.do
=== IN RUSSIAN ===
Как средствами API получить список устройств со статистикой, как на странице https://www.worldcommunitygrid.org/ms/device/viewStatisticsByDevice.do

[Nov 20, 2018 12:20:30 PM]

Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline


Re: API for devices list

I don't think it's possible to get the "Device History Statistics" from the API, unfortunately. But as a last resort, you could scrape the screen and collect the data. I have a small very rudimentary script that will do that. You can find it here: https://github.com/msellan/wcg_dss/ It's a bash version 5 (unix shell) script.

Currently, it just processes the data on the page but doesn't support logging in. To use it, you have to login to the WCG site and navigate to the Device History Statistics page and then using your browser choose File/Save As and just save the html page. Point the script at that page and it'll strip out the data and save it into a delimited file that could easily be imported into an Excel spreadsheet or to a database.

At some point, I'll get around to automating the login so the whole operation can automatic and scheduled. Or better yet, perhaps someone at the WCG will just decide to expose this data via the API and I can discard this script. wink

Best,

-mark

[May 12, 2019 5:56:28 PM]

Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline


Re: API for devices list

Discussed in past

https://www.worldcommunitygrid.org/forums/wcg...ad,40823_offset,20#582183

2 ways if using IE

ieBrwsr.navigate WCGPath & "j_security_check?j_username=" & MemName & "&j_password=" & MemPw

Or into the little overlay that appears when clicking Log In top left

.j_username.Value = MemName
.j_password.Value = MemPw
.submit

Build in time out retry loops and delays... too fast succession and you get rejected.

[May 13, 2019 9:44:02 AM]

Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline


Re: API for devices list

Thanks lavaflow - I think that'll help me finish the script to automate the download of my Device History Statistics! I built a function to login that looks much like that but had the POST wrong. So hopefully you've saved me a good chunk of time.

Best,

-mark

[May 13, 2019 5:33:24 PM]

Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline


Re: API for devices list

At some point, I'll get around to automating the login so the whole operation can automatic and scheduled.

Ok, thanks again @lavaflow for the link to the past forum thread for automating login. applause

I've automated the login so you can download your 'Device History Statistics' on a schedule. It logs in using your WCG credentials and then retrieves and parses your 'Device History Statistics' page pulling the relevant data down and formatting it into a delimited file. It'll work on Mac/Linux and uses Bash v5 (it might also work on Windows with Linux Subsystem for Windows installed but I haven't tested it). Link to Github repo is above in an earlier post.

Best,

-mark

----------------------------------------
[Edit 2 times, last edit by Former Member at May 20, 2019 12:21:00 AM]

[May 18, 2019 6:37:43 PM]

adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:

5 year badge for Human Proteome Folding - Phase 2

90 day badge for Nutritious Rice for the World

2 year badge for Help Fight Childhood Cancer

2 year badge for Help Cure Muscular Dystrophy - Phase 2

14 day badge for Discovering Dengue Drugs - Together - Phase 2

180 day badge for The Clean Energy Project - Phase 2

1 year badge for Computing for Clean Water

1 year badge for Drug Search for Leishmaniasis

1 year badge for GO Fight Against Malaria

45 day badge for Computing for Sustainable Water

100 year badge for Mapping Cancer Markers

1 year badge for Uncovering Genome Mysteries

20 year badge for Outsmart Ebola Together

2 year badge for FightAIDS@Home - Phase 2

20 year badge for Smash Childhood Cancer

5 year badge for Microbiome Immunity Project

10 year badge for Africa Rainfall Project

50 year badge for OpenPandemics - COVID-19


Re: API for devices list

Hi mark,
I've tested your bash script and got some remarks. nerd

In the initial phase where you initialize your variables, I think you should add:
COOKIE_JAR=…

In the function get_device_history, the string ~/Downloads/devicestats.html should be replaced by:
"$INPUT_FILE"

That being said, I've programmed some more on my own version of a script that scrapes WCG's website to look for several statistics: My Contribution and Results Status pages.

I'm using my own script to log the daily My Contribution stuff, e.g.:

$ wcgstats | cut -f 1-7  ## COMMENT: my contribution
2019-05-19 12:06:03	71:228:15:54:34	2,849	135,105,223	1,869	234,955	2,113
$ wcgstats | cut -f 1,8-13  ## COMMENT: my team's contribution
2019-05-19 12:06:03	255:267:21:59:15	464	631,062,246	254	2,316,187	157
$ wcgstats -xx | cut -f 19  ## COMMENT: extended statistics for the BETA science
beta=0:161:12:28:35=944,283=1,599
$ wcgstats -xx -c | cut -f 37  ## COMMENT: extended statistics for SCC1 w/o comma
scc1=0:292:20:38:23=2203615=6749

... or to have a look at my Results page, e.g.:
$ wcgstats -w -sP  ## COMMENT: tasks that are Pending Validation and their wingmen
1--
MCM1_0151191_5397_1--   Linux Fedora   pc2   743   Pending Validation     5/18/19 10:39:39    5/20/19 02:44:15    3.53       126.7 / 0.0
MCM1_0151191_5397_0--   Linux CentOS         -     In Progress            5/18/19 10:39:13    5/25/19 10:39:13    0.00       0.0 / 0.0

$ wcgstats -w -sV -a MCM1 -t2  ## COMMENT: Valid MCM1 tasks, sorted by 'returnedTime'
0--
MCM1_0151154_2521_1--   Linux                   743   Valid                  5/16/19 20:05:08    5/18/19 07:26:09    3.32       127.5 / 127.7
MCM1_0151154_2521_0--   Linux Fedora   oldpc4   743   Valid                  5/16/19 20:05:03    5/20/19 08:23:28    4.91       127.9 / 127.7

1--
MCM1_0151195_6831_0--   Linux                743   Valid                  5/18/19 12:09:10    5/19/19 03:39:36    4.33       126.2 / 125.8
MCM1_0151195_6831_1--   Linux Fedora   pc2   743   Valid                  5/18/19 12:09:06    5/20/19 08:23:09    3.48       125.5 / 125.8

$ wcgstats -w -sV -p7 -m1  ## COMMENT: Valid tasks on page 7 for device 2
1--
MCM1_0151161_5013_0--   Linux                743   Valid                  5/16/19 00:13:43    5/16/19 12:12:35    4.10       128.0 / 128.7
MCM1_0151161_5013_1--   Linux Fedora   pc2   743   Valid                  5/16/19 00:13:22    5/20/19 07:34:25    3.49       129.3 / 128.7

2--
ZIKA_000429450_x5iq6_DENV3_NS5pol_s1_0799_2--   Linux Fedora   pc2   720   Valid          5/12/19 04:57:39   5/19/19 09:58:26   1.13   56.8 / 58.4
ZIKA_000429450_x5iq6_DENV3_NS5pol_s1_0799_1--   Linux Ubuntu         720   Valid          5/12/19 04:57:05   5/13/19 15:14:06   0.96   59.9 / 58.4
ZIKA_000429450_x5iq6_DENV3_NS5pol_s1_0799_0--   Linux Ubuntu         720   User Aborted   5/12/19 03:23:28   5/12/19 04:56:03   0.00   0.0 / 0.0

$ wcgstats -M  ## COMMENT: all of my devices (my own definitions)
1	pc1
2	newpc1
3	laptop2
4	oldpc4
5	android_de910c3b
6	android_fdbbab78

* Download my script 'wcgstats' here. *

[May 20, 2019 12:01:50 PM]

Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline


Re: API for devices list

Hi mark,
I've tested your bash script and got some remarks. nerd

In the initial phase where you initialize your variables, I think you should add:
COOKIE_JAR=…

In the function get_device_history, the string ~/Downloads/devicestats.html should be replaced by:
"$INPUT_FILE"

Hi adriverhoef, thanks for the feedback! You're absolutely right about the hard-coded path - that's a mistake on my part just being lazy. whistling

The variable initialization is a good suggestion too but wasn't actually bad here as it's getting initialized in the dss_env.sh script that I'm sourcing which has my userid and password in it (and so is not loaded to Github and therefore invisible to anyone reading the code!)

Design-wise, though, there's no reason to have it in the ENV script so I moved it into the main script. It's cleaner and easier to understand that way so I appreciate the nudge. wink

I'm a big fan of using

set -euo pipefail

in my scripts as that catches most mistakes such as uninitialized variables amongst other things. Usually it's the first clue if I've forgotten to initialize something as the script immediately fails because of the set -euo pipefail command.

I looked through your script and it's impressive how much you're handling with it! I wrote this one mostly because I haven't written a screen scraper in a while and thought it would be fun working out the regexs.

I still prefer using the API when possible but this historic data isn't available yet through the API so the screen scraper will have to do. I'll probably create another table in my WCG database and start writing the Device Statistics History there just for good measure.

Have you explored any further putting your data in a SQL database? Right now I'm just using a single table to store all the workunit data that comes from the API and I've started thinking about breaking it up into a normalized data structure and storing all the data about workunits across time in addition to historic (screen scraped) data correlated with devices, etc.

I'm not sure it's worth all the work or not. It'll be a good exercise as I don't write a lot of SQL applications. But maybe seeing the underlying patterns of how the workunits are processed could be interesting? @twilyh got me thinking about that and I'm curious if others have gone that route.

Best,

-mark

----------------------------------------
[Edit 1 times, last edit by Former Member at May 22, 2019 2:08:34 AM]

[May 22, 2019 2:07:26 AM]

adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:


Re: API for devices list

Hi Mark,
I feel a bit hurried today, but will try to express some thoughts.

I looked through your script and it's impressive how much you're handling with it!

It's something that grows (hopefully not out of hand biggrin

) as time progresses.

Have you explored any further putting your data in a SQL database?

Only Workunit Results and that works very well as far as I know. I should mention that I'm also still using the 'old' way of storing Workunit Results (tab separated) in a plain text file, since it doesn't grow that fast (something like 50 MB per year) and I'm 'tailing' the logfile (tail -f) daily to see if there are any anomalies.

I'm not sure it's worth all the work or not.

Well, if this isn't good for your programming skills I don't know what is ... wink

In any case I found out that the 'wrong date' bug in the API is still present, I noticed a few weeks ago:

$ grep -C1 ZIKA_000420712_x5k6k_ZIKV_NS1_MD_model_5_s2_2601_0 `wcglog -/` | wcglog -c iwmtrN
ResultId   WorkunitId ModTime    SentTime            ReceivedTime        Name
 926334787 1086513290 1556065300 2019-04-21T13:44:55 2019-04-24T00:21:35 ZIKA_000420712_x5k6k_ZIKV_NS1_MD_model_5_s2_2596_0
 926335175 1086513352 1556065662 2019-04-21T13:44:55 2019-04-19T05:16:26 ZIKA_000420712_x5k6k_ZIKV_NS1_MD_model_5_s2_2601_0
 926336444 1086513641 1556065299 2019-04-21T13:44:55 2019-04-24T00:21:35 ZIKA_000420723_x5k6k_ZIKV_NS1_MD_model_5_s2_0012_0

Clarification: The server received the middle result two days before it was sent to me. Go figure.

[May 22, 2019 12:12:28 PM]

Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline


Re: API for devices list

Well, if this isn't good for your programming skills I don't know what is ... wink In any case I found out that the 'wrong date' bug in the API is still present, I noticed a few weeks ago:

Haha, yes for sure! I really could use more practice with writing SQL, lol.

And thanks for pointing out the 'wrong date' bug. I haven't been on the forums long and am still discovering all the hidden gems like that from the past. I guess it also means I need to start combing through my data more. Hence the need for more SQL scripting skills! wink

Best,

-mark

[May 22, 2019 3:24:36 PM]

adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:


Re: API for devices list

Hi Mark,
Maybe it's funny to hear/read how my wcgstats script grew over time. biggrin

First all I wanted was to scrape the data from My Contribution in two ways:
* Left column and right column: My Contribution + My Team (that's one way cool

) - Total Run Time, Points Generated and Results Returned + all rankings
* All the data from Statistics By Project (that's the other way tongue

)

By saving those data to a logfile you can see the progress you're making over time (and with respect to My Contribution and My Team you can also see if you're climbing or falling in the rankings), e.g.:

2017-04-08 00:06:02	72:024:06:06:43	1985	176395553	824	641907	343
2017-04-09 00:06:02	72:041:05:01:55	1985	176976433	820	646370	341
2017-04-10 00:06:02	72:053:10:31:29	1985	177301206	817	649303	339
2017-04-11 00:06:02	72:072:19:32:57	1984	178002012	815	656219	336
2017-04-12 00:06:02	72:088:04:08:46	1982	178268317	815	657847	334
Date       Time    	Total Run Time	Rank	Points   	Rank	Results	Rank

Clearly the idea for this came up when I entered the top 2000 in runtime. laughing

After that I wanted to scrape the Results Status with all its options (Filter by Device Name, Result Status, Project Name, pagenumber, sort by dueTime, returnedTime, runTime).

Two years later, I decided that more was needed. confused

So recently I added an option to show the program version, furthermore I added a way for other users to update their version of the program without having to copy their credentials with each update. (Maybe you noticed that I don't source a file that's containing one's credentials.)

Recently I also found out that I'd better show the actual devicename, instead of showing the OS version. Then it was your idea that made me implement scraping the Device Statistics History (which you can also filter by devicename and by pagenumber). At last I also added the options -N and -n, e.g.:

$ wcgstats -n beta

	project: 
	completionDate: null,
	id: 1,
	name: "Beta Testing",
	overviewUrl: "/about_us/viewNewsArticle.do?articleId=375",
	releaseDate: "2004-11-15T00:00:00",
	shortName: "beta",
	statisticsUrl: "/stat/viewProject.do?projectShortName=beta",
	status: "proj_stats.beta"

Since my program doesn't have a way to scrape the My Contribution History yet, maybe that could be the next program addition; anyways I don't have an option for that … yet. sleep

[May 23, 2019 11:23:15 AM]

[ ]