Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 89
Posts: 89   Pages: 9   [ Previous Page | 1 2 3 4 5 6 7 8 9 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 464871 times and has 88 replies Next Thread
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

Thank you for your insightful comment, but numero uno: Changing the stats engine is -NOT- an option. Sure enough I can with piles of effort code around missing fields, but not going to spend piles of effort, when the ugly scrape still [again] works flawlessly. Simply reached my limit of schema's, maps, vba and web-querying to pull the charts together.
[Dec 10, 2013 3:10:55 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

Limits are always bound to be moved outward, so wasted another few hours 'learning' what I really did not need, but do I dislike a half a$$ed job. All reverse engineering ripped out, all data flowing now from xml pulls directly into the DB via the conversion process. =int(Mod (n;x)) proofed very slick to convert seconds to hours/days/hours/minutes and determining the residuals. For those on Office, here's a good and easy to understand instruction on xml schemas and maps, drag and drop ease rounding this out, and a little notepad massaging to get the schema's right, ready for any [known] condition: http://office.microsoft.com/en-us/excel-help/...elements-HP010206397.aspx

The dash is probably back in another day or two... found a few iffy things in the code [never look back at what you concocted in past, or you find you'll be writing yet another 'improved' version ;P]

P.S. One scrape remains, used as a x-verifier [of the null hypothesis ;]]]]]]]]]
[Dec 16, 2013 6:27:32 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

Anyone knows what's wrong with this XML pull off the global stats page:

http://www.worldcommunitygrid.org/stat/viewGlobal.do?&xml=true

In response I'm getting 3 instead of 1 data set, each coming with a different url, but for the remainder, identical in content.

http://www.worldcommunitygrid.org/stat/viewGlobalHistory.do?format=xml
http://www.worldcommunitygrid.org/stat/viewGlobal.do?format=json
http://www.worldcommunitygrid.org/stat/viewGlobal.do?format=csv

Implies there's a javascript and a csv format as well [in case there are interestees]. Speculate, these are for those snooping with e.g. Android devices.

Not that it stops the show, it must go on, just picking and choosing, but it is coming across. Must be stated, this was the very first xml draft I set up, so could be somewhere I picked one too many boxes when defining the query connection, but I'm buffed. So close to completion dread to go back a scrub, do it again [DIRFT policy not applied here :( ].
[Dec 17, 2013 10:41:44 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Tullus
Cruncher
Joined: Nov 14, 2008
Post Count: 29
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

What do you mean? When looking at the csv file in a text editor I get this:

"LastUpdated", "2013-12-17T12:06:02"

"Total Members", "Total Devices", "Total Run Time", "Total Points", "Total Results"
640037,2510603,26024932288230,851914816610,1694014557

"Run Time per Day", "Run Time per Result", "Points Per Hour Run Time", "Points Per Day", "Points Per Result", "Results Per Day"
7708806957,15363,117.84443109479999,2.5234443619964457E8,502.896986976718,501781.5630924171


"Total Members", "Total Devices", "Total Run Time", "Total Points", "Total Results"
"2013-12-16",21285081064,955818654,2032813

"Global Statistics History in JSON", "http://www.worldcommunitygrid.org/stat/viewGlobalHistory.do?format=json"
"Global Statistics in XML", "http://www.worldcommunitygrid.org/stat/viewGlobal.do?format=xml"
"Global Statistics in CSV", "http://www.worldcommunitygrid.org/stat/viewGlobal.do?format=csv"


And the same information in both the xml and json format. I don't see anything amiss here confused
[Dec 17, 2013 1:00:12 PM]   Link   Report threatening or abusive post: please login first  Go to top 
SNURK
Veteran Cruncher
The Netherlands
Joined: Nov 26, 2007
Post Count: 1217
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

If all stats pages, also at member, team and country level are going to be available in CSV too, I may just be tempted to revert to this old school format. thinking
It's got me interested because it will give me the smallest downloads. Requires a major rewrite though.

Just for the heck of it tested this url:
http://www.worldcommunitygrid.org/stat/viewMe...Name=SNURK&format=csv
..but no, it's not there (yet?).
----------------------------------------
[Dec 17, 2013 8:06:55 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

Try

http://www.worldcommunitygrid.org/stat/viewMe...at=csv&userName=SNURK

Don't ask me why with ?&xml-true it has to be put in front for the project xml pulls, but then it works for me e.g.

http://www.worldcommunitygrid.org/stat/viewPr...amp;projectShortName=mcm1

I'm just a trained monkey that's expected to fly [Project-X]
[Dec 17, 2013 9:12:10 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

For now a hacked WCGPI1 is back. The projects that are in an intermittent state and have no work have:

1) Removed the days left.
2) Considered 100% done (for the work we were sent by the scientists/project owners)

Of course a few will not enjoy, but that's the norm in life.

Some info lines were added that came off the System Requirement matrix which was discontinued since longer [not maintained] **. Maybe the science app versions get added. Very low maintenance items that can not be automatically updated.

** [Visit WUProp, participate in this NCI [Non-CPU-Intense] project, taking 0.001% on my octos, which reports all sorts of WU technical measurements to WUProp for detail CPU/Science/OS statistics with great CPU to CPU comparative graphing].
[Dec 18, 2013 11:53:56 AM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss


That would get my thumbs up too!! smile


These API's are now available.

As is the Result Status as documented here: https://secure.worldcommunitygrid.org/help/viewTopic.do?shortName=api#665
[Feb 18, 2014 2:32:03 AM]   Link   Report threatening or abusive post: please login first  Go to top 
foxfire
Advanced Cruncher
United States
Joined: Sep 1, 2007
Post Count: 121
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
applause Re: Screen Scrapers - Please Discuss


That would get my thumbs up too!! smile


These API's are now available.

As is the Result Status as documented here: https://secure.worldcommunitygrid.org/help/viewTopic.do?shortName=api#665

My thanks for this applause applause Putting the code together now
----------------------------------------

[Feb 18, 2014 3:04:21 PM]   Link   Report threatening or abusive post: please login first  Go to top 
foxfire
Advanced Cruncher
United States
Joined: Sep 1, 2007
Post Count: 121
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

WU Results Status XML looks great! Thanks to all that contributed hugs

Question, How to differentiate between User Abort, Server Abort, Invalid, and Error status?

I see that:
"User Abort" has:
ServerState=5 (been reported back to the server)
Outcome=3 (error)
ValidateState=2 (invalid)
ExitStatus=203

"Server Abort" has:
ServerState=5 (been reported back to the server)
Outcome=3 (error)
ValidateState=2 (invalid)
ExitStatus=202

I believe that:
"Invalid" will have:
ServerState=5 (been reported back to the server)
Outcome=6 (validation error)
ValidateState=2 (invalid)
ExitStatus=???

"Error: will have:
ServerState=5 (been reported back to the server)
Outcome=3 (error)
ValidateState=2 (invalid)
ExitStatus=???

I don't have any "Invalid" or "Error" in my Results Status but it appears that "Error" would be the same as "Server Abort" with the possible exception of ExitStatus.

Is that correct and if so, how do I distinguish them?
----------------------------------------

[Feb 19, 2014 8:26:48 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 89   Pages: 9   [ Previous Page | 1 2 3 4 5 6 7 8 9 | Next Page ]
[ Jump to Last Post ]
Post new Thread