Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 89
Posts: 89   Pages: 9   [ Previous Page | 1 2 3 4 5 6 7 8 9 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 463775 times and has 88 replies Next Thread
jonnieb-uk
Ace Cruncher
England
Joined: Nov 30, 2011
Post Count: 6105
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

It's possible that something has changed on the website in the last 12-15 hours confused

BladeD reports here that the commas have disappeared from the Accumlated Points displayed on the top ribbon.

Additionally my old web queries based on screen scrapes are inoperative. Not a problem as my data capture is now all ?&xml=true based.

[edit] sp.
----------------------------------------

To Join follow this link: Join the UK Team All Welcome! UK Team thread
----------------------------------------
[Edit 1 times, last edit by jonnieb-uk at Dec 5, 2013 8:48:22 AM]
[Dec 5, 2013 8:47:33 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

My web-query engine, see post with screenshot up in this thread with Office 2013, works up to the point where it goes to import, then a reply comes there's no data to import [Why?]. That'd be the end of the Performance Charts and Dashboard I'm afraid, where the ?&xml=true suffix gives such dramatic form change how data is pulled it means a very major rewrite... not in one afternoon.

Did notice BTW as of previous web-update, something had changed on the Result Status pages. Before when selecting a result to see the quorum detail, and returning to that page, the WU name was still selected [dotted square around the name]. That disappeared.
[Dec 5, 2013 10:14:16 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

Sadly, webqueries continue to fail. Up to point of getting the wanted pages to show in the preview screen [screenshot posted before], normally suppressed, things are fine, but then nothing comes across on import:



BTW, since that unannounced change, the top line on main web page looks misaligned both in Chrome [latest] and even more in Firefox. The language selector appears positively borged, half off the page edge and the Log Out is offset downwards.

Edit: The thread title really requires an extension in the form... "Please Discuss, amongst yourselves... we'll do anyway what we want to do, change without notice". Took a crash course xml schema designing and VBA coding. The front project/global pages are disastrous and inconsistent... a day where there is no results does not generate current day field data [why not simply populate with zero?]. Also the data somehow lands in reverse order which has more coding consequences. The history pulls were in the end, after the learned and heavily bruised kneecaps from falling and getting up, a piece of cake [courtesy of the superb Office 2013 mapping to table import automation]. From the imports, reversed engineered the data back to it's old form, so the main charting engine did not need recoding. Only main thing to do is find what the names of all these maps are so they can be put in place of the web-queries and then update the scheduled script.

Knreed, where do I send the bill ;?.. (and please hands off now or you find me blowing the main fuse :!)
----------------------------------------
[Edit 1 times, last edit by Former Member at Dec 6, 2013 5:14:31 PM]
[Dec 6, 2013 9:34:37 AM]   Link   Report threatening or abusive post: please login first  Go to top 
jonnieb-uk
Ace Cruncher
England
Joined: Nov 30, 2011
Post Count: 6105
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

@ SekeRob

The commas are back wink and following a quick test it appears funtionality is restored to web queries!
----------------------------------------

To Join follow this link: Join the UK Team All Welcome! UK Team thread
[Dec 6, 2013 5:30:40 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

As per the edited-in para above... Too Late, could not wait for something to happen or I would have lost noon data.

(Will the French protest there's commas and not spaces ;?)

Edit: Yes, the old web-queries worked again smile , which was a good test to compare the old-way processed data with the xml pulls [which work, granted, quite a bit faster]. Pressure off kettle, can slow down now, but as I'm at it, will now convert the whole shebang, and plan for removing the intermediate reverse engineering step.
----------------------------------------
[Edit 1 times, last edit by Former Member at Dec 6, 2013 5:48:07 PM]
[Dec 6, 2013 5:38:42 PM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

Glad you have moved to the XML interfaces vs screen scraping. Those are much more stable. Sorry that it had to be under pressure and due to a inadvertent bug introduced in a minor update.
[Dec 6, 2013 10:56:53 PM]   Link   Report threatening or abusive post: please login first  Go to top 
JmBoullier
Former Community Advisor
Normandy - France
Joined: Jan 26, 2007
Post Count: 3716
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

(Will the French protest there's commas and not spaces ;?)
No problem here. biggrin
But, as explained earlier in this thread, the correct language parameter is coded in all stats queries involved in my daily stats routine.
And in case of emergency I know how to quickly recover the correct setting if necessary.

See
https://secure.worldcommunitygrid.org/forums/...ead,35782_offset,0#437685 and https://secure.worldcommunitygrid.org/forums/...ead,35782_offset,0#438237 for details.
----------------------------------------
Team--> Decrypthon -->Statistics/Join -->Thread
[Dec 7, 2013 2:03:31 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

Whilst the Dashboard is still not it's old self yet, did add 3 lines:

Regular quorum
Standard Deadlines & Repairs Deadline percent
Where From: (To underline the 'International' nature of WCG). 6 Countries that have a science project on WCG.

@knreed:

The xml for the stats overview pages represent a problem. There are in effect 6 XML layouts. Beta with / without day activity. Project with / without day activity and an extra image link / Global with / without day activity, though that latter will be extremely rare. Would much prefer if the empty fields are populated with a 0 value, so the maps are consistent [see my previous post on that]. Had to revert for Beta to scraping to avoid the update crash.
[Dec 10, 2013 12:28:03 PM]   Link   Report threatening or abusive post: please login first  Go to top 
jonnieb-uk
Ace Cruncher
England
Joined: Nov 30, 2011
Post Count: 6105
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

The xml for the stats overview pages represent a problem. There are in effect 6 XML layouts.

@ knreed
A similar problem also exists with XML data for individual members. Layout varies where:

  • Member has yet to uploaded any results
  • Member has yet to earn any badge, Fields are ProjectName; Url; Description.

The first instance is unimportant but the second can cause issues.

As SekeRob says, it would be helpful if empty fields could be populated with a zero.
----------------------------------------

To Join follow this link: Join the UK Team All Welcome! UK Team thread
[Dec 10, 2013 12:41:25 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Tullus
Cruncher
Joined: Nov 14, 2008
Post Count: 29
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Screen Scrapers - Please Discuss

No offense, but are you guys missing the point of xml? If a simple csv (comma separated values) was used then missing values should be 0 or blank. But missing entries in xml is totally legal and normal. I would start looking at another xml parser if missing values causes problems.

(I would of course recommend python, but that is just my tool of choice for everything).
[Dec 10, 2013 2:29:04 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 89   Pages: 9   [ Previous Page | 1 2 3 4 5 6 7 8 9 | Next Page ]
[ Jump to Last Post ]
Post new Thread