| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Community
Forum: Chat Room
Thread: Hunting Tool 2.0 |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 116
|
|
| Author |
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
Almost all new "under-the-hood version" 2.0 , brewed under the influence of large quantities of robusta beans ground coffee:
A small digest, not everything covered: 1a/b) Now supports 20 active devices, autoexpanding this screen wide top section. 2) Sorts devices according to the 'last returned result order' 3) 12 hourly history shows the current period cumulatives in bolded blue, until the period is closed. 4) The expanded summary section shows the last result returned with timestamp. 5a/b) Trickle CPU time was integrated showing how many hours are in uncompleted tasks. 6) The action buttons show how many IP and Completed results [Valid/Error/Invalid] are on the Result Status pages. 7) Consolidation was big time simplified... create a new project tab from template and it gets automatically included in the consolidation overview. 8) Pending result count column includes those with trickle. 9) Sample of the consolidation formula... move a project left of 'All' and it no longer gets included. 10) The ResultXML action buttons for active results querying were merged... there's now a test how many active results there are on the RS pages, with cpu time... always running the quickest possible. The pulls now cover up the 67 full RS pages [excluding in progress] i.e. 1,000, a medium size 'farm' is tendered for. A larger farm will work too, ~25 nodes, whence the new API field FileDeleteState is integrated [in test]. Results archived and marked for deletion = 2, are no longer of interest, but found 2 cases where results were marked as such in the current statistics period [led to incomplete result detail at 12:06 and 00:06 as how things are now coded]. I.e. a result can get validated and marked for deletion in same statistics period. 11) All devices that have turned inactive [nothing on the RS pages], are highlighted in red. (Plan to make this nothing in past 24 hours, or something other value which can be entered for the total micro manager... last 12-6 hours ;p). 12) The computed anticipated badge/contribution level date is now shown... projects with now new target show as 'Done', those inactive with 'None' 13) The elevator has no button for this floor, crossed out. Anything below was moved up :))) 14) The 'In Progress' tasks can now be up-to 500. Relatively easily expandable, but for now good to cover 20 active devices with 25 buffered per host [Octo's with 25 FAHB on board will have trouble to complete all in time] Feel free to rip ideas from this, but as this is now in the public domain, you can't patent any part of this anymore... you'll have to proof original art ** ;O) ** Only those with Klikonium filled crania would pay for this  |
||
|
|
deltavee
Ace Cruncher Texas Hill Country Joined: Nov 17, 2004 Post Count: 4894 Status: Offline Project Badges: 
|
SekeRob*,
Very impressive. It puts my little screen scaping spreadsheet to shame. |
||
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
v2.0b is out, the biggest and remaining hurdle taken, and cosmo stuff [showing device last day and cumulative runtime, and last result showing automatically at local regular or DST time, automatically, poking into the OS kernel ;P].
The tool was scaled up to handle up to 99 active devices, and now can deal with up to 1049576 minus 1 results [for the field headers] a 100 fold from the 1000 active results it could before. There's no documentation anywhere on google how to, just trying and trying different code that would not fail on 'cannot expand table, because', the end result neater than neat, in 3 steps The ugly [for 1000]: =IFERROR(IF(C8>0;COUNTIFS(ResultXml1[DeviceId];C8;ResultXml1[Status];"Pend*";ResultXml1[AppName];AppF;ResultXml1[CpuTime];">"&0;ResultXml1[ModTime];"<"&PeriodEnd)+COUNTIFS(ResultXml2[DeviceId];C8;ResultXml2[Status];"Pend*";ResultXml2[AppName];AppF;ResultXml2[CpuTime];">"&0;ResultXml2[ModTime];"<"&PeriodEnd)+COUNTIFS(ResultXml3[DeviceId];C8;ResultXml3[Status];"Pend*";ResultXml3[AppName];AppF;ResultXml3[CpuTime];">"&0;ResultXml3[ModTime];"<"&PeriodEnd)+COUNTIFS(ResultXml4[DeviceId];C8;ResultXml4[Status];"Pend*";ResultXml4[AppName];AppF;ResultXml4[CpuTime];">"&0;ResultXml4[ModTime];"<"&PeriodEnd)+COUNTIFS(InProgressXml1[DeviceId];C8;InProgressXml1[AppName];AppF;InProgressXml1[CpuT ime];">"&0)+COUNTIFS(InProgressXml2[DeviceId];C8;InProgressXml2[AppName];AppF;InProgressXml2[CpuTime];">"&0););0) The not so bad [for 1 million] =IFERROR(IF(C7>0;COUNTIFS(OFFSET(ResultXml1[[#Headers];[DeviceId]];1;0;MAT;1);C7;OFFSET(ResultXml1[[#Headers];[S tatus]];1;0;MAT;1);"Pend*";OFFSET(ResultXml1[[#Headers];[AppName]];1;0;MAT;1);AppF;OFFSET(ResultXml1[[#Headers];[CpuTime] ];1;0;MAT;1);">"&0;OFFSET(ResultXml1[[#Headers];[ModTime]];1;0;MAT;1);"<"&PeriodEnd)+COUNTIFS(OFFSET(InProgressXml1[[#Headers];[DeviceId]];1;0;MAT;1);C7;OFFSET(InProgress Xml1[[#Headers];[AppName]];1;0;MAT;1);AppF;OFFSET(InProgressXml1[[#Headers];[CpuTime]];1;0;MAT;1);">"&0));0) The glorious [for 1048575] =IFERROR(IF(C6>0;COUNTIFS(ResDeviceId;C6;ResStatus;"Pend*";ResAppName;AppF;ResCpuTime;">"&0;ResModTime;"<"&PeriodEnd)+COUNTIFS(IpDeviceId;C6;IpAppName;AppF;IpCpuTime;">"&0));0) The MAT value is any value entered as variant. Got it set for 2000 as that parses much quicker on older Office versions, but on 2013 and 2016, it flies with 8GB even with 1 million dummy records [memory the constraint] (Think BOINCTasks falls over if you have that large a result history ;) |
||
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
On a roll again [it's raining on the other side of the window], the highlighted central section over the action buttons block [see previous post screenshot link] now orders the sciences based on their last result returned, latest at top. Inactive / complete projects drop to the bottom. Briefly contemplated, to fold this table, to gain screen real estate. Same for the action buttons, considering to move them to a quick access action ribbon, but shied away, for now. Already have a ribbon with few buttons from the old My 'Grid' days, primo the API availability. It makes distribution though harder, loading a custom ribbon, unloading and restoring the users private settings when exiting. The My Stuff ribbon I have is so deeply entwined with the system global macro system, it would be another major task to make it work for all and everyone... it's not supposed to be work, but fun to develop this... in whatever direction the wind blows.
The project Qry tabs now are turned green, auto-magically, when they have active results. Nothing on the RS pages, they turn orange again [RGB 255, 192,0], a Pop Pop, a chance hit enhancement, researching something else, a bug that caused new valid results to overwrite previous archive records because a manual filter was still in effect. Took care of that which caused another bug, which was taken smashed with If ActiveSheet.AutoFilterMode Or ActiveSheet.FilterMode Then ActiveSheet.ShowAllData The 'Or' for all eventualities Crunch On. |
||
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
From the really really big crunchers, the top 40 or so on page one of this member community report https://secure.worldcommunitygrid.org/stat/viewStatsByMemberY.do?sort=results , really those who have more than 166 Result Status pages, like to know how many *pages* they have
In Total: In Progress: Pending: Currently the tool can handle per 12 hour period, 2500 active, non IP records, and 750 IP, but for all I guess, it could well be that one runs only zero redundant projects, and only shorts [OET], which are quite quickly migrated off after validation, or long and wingman requiring, short caches or long caches. The 750 IP limit will definitely require scale up, more maps, but that's not a problem, as the next little diddy of code on the control sheet establishes how many records have to be handled upon the very first 250 result fetch which reveals 'ResultsAvailable'. =ROUNDUP(MATCH(VLOOKUP(REPT("z";50);OFFSET(ResultXml1[[#Headers];[Change]];0;0;1048576;1);1);OFFSET(ResultXml1[[#Headers];[Cha nge]];0;0;1048576;);1)/250;)*250 Yes, the1048576 records which is the office worksheet limit. The number derived from N / 250 determines how many fetches have to be run, each loop incrementing the offset with 250. Soon invitations will go out to test. Now on 2.0c which has lots of 'user input verification such as decimal comma checking when the locale is set to decimal point and ranges of valid values checking [users can only enter in yellow fields]. The tricky issue is the Data > Connection editing [your member name and verification code], and the few pages where there's no xml available, scraping html, where if things work as they are supposed to, asks one time for member name and password [the Devices Stats and Daily History]. While typing, think to have gotten an idea of how to automatically populate all of of the xml map connectoids, just limiting user input to one time member name and the 32 character verification code entry in the control sheet. All previous coding attempts have failed, [Excel VBA coding help is a dog unfortunately, still is]. Forward On (The Section) P.S. Other approaches took too long, it has to complete in under 1-2 minutes, as else the data fetch goes stale [Remember, we're fetching from a live system, where by the time of reaching page 166, the completed results may already have been migrated off or had their status changed from 'ready to delete' to 'deleted', giving a non-reconciling picture. |
||
|
|
deltavee
Ace Cruncher Texas Hill Country Joined: Nov 17, 2004 Post Count: 4894 Status: Offline Project Badges:
|
SekeRob*,
I am currently not in the top 40 because I am crunching FAHB and CEP2 which have long runtimes, so returning the requested information probably won't help. However, when I was crunching OET1 at times with short workunits, I would max my non-android crunchers at 35 workunits per thread. At 299 threads that was 10,465 workunits IP. And that was instantaneous, not in a 12 hour period. The most OET1 I have record of returning in a 24 hour period is 15,280. It may not be worth it to scale your Hunting Tool for a few outliers. I would like to be able to use such a tool, but wouldn't complain if it wouldn't fit. |
||
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
deltavee, how many valids were those per 12 hour period? The fetch filter can be optimized to only get valid from a modtime on or after, i.e. only get latest results, since last stats. I'd think at those scales, valids and errors are of main interest, those valids and TMEs [Too Many Errors for the android front] of previous period in the archive already, if not yet migrated off by WCG, no need to fetch them again.
Have half working code now to even create the maps and connects in automated fashion. Probably with a little for-next loop could create them all on the fly... its much faster than I thought it would be **, lots of testing too as it is sort of finding the holy grail at the end of the grotto, with all these booby traps and pitfalls to circumvent. So far sent 12 pions and none made it to the room where the old knight is guarding the goblet... but can see the geezer now ;P. ** 10K valids is 40 xml maps, if 1-35 already exists from a previous run, only need generating 36-40 and just set a new modtime start point. The data fetching has to be quick quick, the post processing can take longer, tallying, sorting, filtering, summarizing, hearing the ding ding ding when done, while drinking a mug of arabica. 299 threads... is that < 99 devices? That part does fit. Rediscovered Scrolling Forms today, maintain the key screen on display while looking at the detail records ***, which too can be set to filter for any particular field(s). [MS removed the forms access from the ribbon interface, so added a custom group to 'Data', and exported the GUI [Importable for anyone with Office 2010 - 2016 who wants it], *** With forms can even filter on results with for instance a poor computed efficiency or shoddy credit/hr [All there and more, just in case these items regain my interest  ]. |
||
|
|
deltavee
Ace Cruncher Texas Hill Country Joined: Nov 17, 2004 Post Count: 4894 Status: Offline Project Badges:
|
All of the 15,280 returned workunits should have been valid. I wasn't getting any errors at the time and the Results Returned shown in My Contribution does not count PV. The daily total was derived by subtracting Results Returned shown in My Contribution from the number of results the previous day. I only scrape once a day, so on average over a 12 hour period it would have been 7,640. The 299 threads were on way less than 99 machines. I salute your quest for the Holy Grail.
|
||
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
New magic with =GetURL (custom function) and scraping in Devices Statistics as full html formatting query. What you get is a hyperlinked column with the device names AND the hyperlink addresses that contain the unique device IDs. Then do a little extracting and voila
https://secure.worldcommunitygrid.org/ms/devi...=1234567&deviceType=B which through =IFERROR(MID(celladdress;FIND("=";celladdress)+1;FIND("&";celladdress)-2-FIND("=";celladdress)+1)*1;"") allows search for one-time/special characters and stripping out the device id: 1234567 (The *1 in the formula part makes sure the value is stored as a number and not as text [which occasionally is trouble-causing].) There you have it, the unambiguous link from the Device Statistics page totals of runtime/results/points per device to the API pulled data which lists the 1234567 with each results done by the same unique device regardless if you have duplicate names (I have half a dozen all named ubuntu resulting from trial-error test installations). Works for the androids inclusive, even if you have a whole flotilla of them. From this point on a little lookup and you can name your devices in the spreadsheet anything you want it to be and you can relate to [an Alias], including illegal characters that WCG does not like ;P. So now my tablet is not named android but listed as Nexus-7 #1. Of course only works for hosts that have at least 1 returned result and AFTER the 12 hourly stats have run, as else they're not listed on the Device Statistics page. Till then, Hasta la pasta. |
||
|
|
SekeRob
Master Cruncher Joined: Jan 7, 2013 Post Count: 2741 Status: Offline |
And so from the previous evolved in 15 minutes the link to pulling the device profiles connected to hosts. Now the 'Hunting' tool shows newest to oldest activity sort order, device name [whatever you want it to be, including androids], OS, location, unique device ID, detail activity of completed, pending and cache, accumulated totals, period crunching, AND also summary highlight for devices which have tasks that are getting hotly on towards the deadline [user definable, presently set at < 30 hours]... All in a single view and much much much more.
 Things users need to learn: How to modify an account connect, how to push 5 different custom buttons [fail saves build in], optionally and one-time, enter OS for each device, core count and efficiency, latter for optimal buffer time calculation and really really optional the real runtime estimate, not the grub that the BOINC manager is serving. For each active device you have, that's 15 seconds. Today, Hasta la Cous Cous |
||
|
|
|