Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 11
Posts: 11   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 4066 times and has 10 replies Next Thread
SekeRob
Master Cruncher
Joined: Jan 7, 2013
Post Count: 2741
Status: Offline
Reply to this Post  Reply with Quote 
[RESOLVED] API data fetch gives issues if the "Received" date returns null

It's not an issue when running the In Progress - ServerState=4 fetch, as I'm not mapping the 'Received' field for that query, but on SS=5 it is when No Reply state is reached on for instance lost device installs. Is there a way around this, even putting a nul in instead of leaving it null? It results in the system trying to squeeze a text value into a date field and hanging the script, with office having no escape clause other than .DisplayAlerts = False, with the obvious consequence... GIGO. Am appreciating that modifying the DB to not churn out null would be an act of the supranatural power, but guess the API that passes the data could be educated to give zero instead.

thx.

P.S: Any of the other API users who grapples with this as a problem?

Edit: Marked as [RESOLVED]. Functionality was added to omit the ReceivedTime field for any record where there's no date for the field.
----------------------------------------
[Edit 1 times, last edit by SekeRob* at Jul 14, 2017 11:22:01 AM]
[Jul 8, 2017 2:28:55 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

It's not an issue when running the In Progress - ServerState=4 fetch, as I'm not mapping the 'Received' field for that query, but on SS=5 it is when No Reply state is reached on for instance lost device installs. Is there a way around this, even putting a nul in instead of leaving it null? It results in the system trying to squeeze a text value into a date field and hanging the script, with office having no escape clause other than .DisplayAlerts = False, with the obvious consequence... GIGO. Am appreciating that modifying the DB to not churn out null would be an act of the supranatural power, but guess the API that passes the data could be educated to give zero instead.

thx.

P.S: Any of the other API users who grapples with this as a problem?
Thanks for pointing out this problem, Rob. As for me it can't be a problem with this (Perl) code snippet, right before all field values are written to the logfile:
$ReceivedTime = "" if $ENV{opt_w} > 0;
This snippet (from my script 'wcgresults') says that if the workunit is still in progress, than the ReceivedTime is set to the empty string (""). So in my script it doesn't matter if the text field ReceivedTime is empty or not.

You made me aware again of a possible problem in my script though: if a field is absent or if a text field is empty, what does the script do? My script would behave erratically: it would take the previous value because of the faulty pattern ': "([^"]+)"', meaning that it wouldn't match an empty string. I have repaired that last night thanks to your posting.
Although a repair was put in place, I can't say I have found an Outcome with No Reply (4) in my logfile.

In my script, as from now on, at this moment, not posted yet, an empty text field is almost handled in the same way as an absent text field. An absent text field will assign an empty text value (zero length string) to the variable, whereas an empty text field will yield the string "" (literally). That will go into the logfile and subsequent programs can deal with that if they need to.

I was thinking maybe we need to look at it this way:
What should a script do if a text field is absent? Wouldn't it be quite the same when that text field is empty? Not quite, you say, because an absent field is different. How different? If it were empty, you could do different things in your script than when it were absent, but in essence you have one and the same case if that text field is empty or absent when there is nothing to report. So, in conclusion, your script should handle the case of an empty or absent text field. I think putting something in a text field that doesn't belong there (no one expects ...!) is not the solution, unless that behaviour is defined.
[Jul 9, 2017 12:41:10 AM]   Link   Report threatening or abusive post: please login first  Go to top 
SekeRob
Master Cruncher
Joined: Jan 7, 2013
Post Count: 2741
Status: Offline
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

Thx for the input.

Office has no 'do this or that' conditional controls at record or field level while parsing the data in, there being an IMEX=1 (ImportMixedTypes) setting against which is warned for then 'you may get unexpected results'. Unbeware, the driver second-guesses from the first 8 received records what the data type of the fields is and then assumes an answer for the remainder, but as it is, have no control over the order [would love to have a sort on ModTime], or a Not i.e. exclude a condition like Outcome = 4. There's an option to disable date value interpretation, which means the value just comes in as a value, but if first 8 are numeric, the assumption is, the rest will be too (which explains now why not getting warnings on 'In Progress' / ServerState=4... not the absent mapping but the pass assuming all will be a text value for that field :( ). If there were the ability to be more selective of what comes in, I'd subdivide all the requests to main types.... give me only PV, give me only Valid, give me only Error and then merge the individual data sets in a power query.

FTM, have set the .DisplayAlert = False > Do XmlMap query > .DisplayAlert = True, so these No Reply come in without alarm bells going off, then process the data with a function and set null = 19700101 0000 as the condition to evaluate in case of a null value i.e. then it can handle the situation and puts the 'No Reply' status label on. Amazingly, the function, though the documentation claims "You can not" can change the value of a field... it is slow though when parsing through 274K records live at this time, when the objective was to get to 10 minutes per 250x250, had it sub 12 minutes, back to 15+ minutes with that code injection. sad

There's ActiveX to inject an IMEX=1 into the driver controls i.e. when anticipating to getting close to the last query cycle (1097 last night) as the No Reply are at the end when using sorting=ReceivedTime, but it's an iffy solution, since at start there's indication of for instance 1095 pages, but then in that hour the script is running 30-40 and more pages can disappear or add, since data is pulled from a live system... final results are migrated off after 24 hours having sat in valid state **, and no more copies are in circulation (not a fast and stead rule). Then instead the issue shows up much earlier.

(** Just added up the validated results and the sum of active results approximates 2.5 days of valid on the RS pages, which is puzzling)

For 1-10000 results/day it's no issue, against my own account all is done in under 30 seconds no matter what I seem to bolt on as fluff with sometime 4000 active results on the RS pages cool
[Jul 10, 2017 8:04:22 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

Just a small note, Rob. You wrote:
final results are migrated off after 24 hours having sat in valid state **, and no more copies are in circulation (not a fast and stead rule). Then instead the issue shows up much earlier.

(** Just added up the validated results and the sum of active results approximates 2.5 days of valid on the RS pages, which is puzzling)
Wondering about that, too, I saw the current oldest results were handed over - with Minimum Quotum = Replication = 1 - two days (48 hours) ago.
[Jul 10, 2017 10:41:23 AM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

Greetings,
I see the issue that you are describing (for others here it is):

<Result>
<AppName>oet1</AppName>
<ClaimedCredit>0.0</ClaimedCredit>
<CpuTime>0.0</CpuTime>
<ElapsedTime>0.0</ElapsedTime>
<ExitStatus>0</ExitStatus>
<GrantedCredit>0.0</GrantedCredit>
<DeviceId>3639918</DeviceId>
<DeviceName>oc7811586721.ibm.com</DeviceName>
<ModTime>1499677676</ModTime>
<WorkunitId>131373917</WorkunitId>
<ResultId>1598751658</ResultId>
<Name>OET1_0004827_x3MWP_rig_77968_0</Name>
<Outcome>0</Outcome>
<ReceivedTime>null</ReceivedTime>
<ReportDeadline>2017-07-20T09:07:56</ReportDeadline>
<SentTime>2017-07-10T09:07:56</SentTime>
<ServerState>4</ServerState>
<ValidateState>0</ValidateState>
<FileDeleteState>0</FileDeleteState>
</Result>


In particular, the <ReceivedTime>null</ReceivedTime> is ugly and not what should be there.

The options is that if the value doesn't exist, we could return:
<ReceivedTime/>
or we could simply skip the element if it is null.

Let me know which of those two options will work better in your script processing.
[Jul 12, 2017 7:44:01 PM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

Just a small note, Rob. You wrote:
final results are migrated off after 24 hours having sat in valid state **, and no more copies are in circulation (not a fast and stead rule). Then instead the issue shows up much earlier.

(** Just added up the validated results and the sum of active results approximates 2.5 days of valid on the RS pages, which is puzzling)
Wondering about that, too, I saw the current oldest results were handed over - with Minimum Quotum = Replication = 1 - two days (48 hours) ago.


After the result is validated, there is the 24 hour delete delay on the file (this is for all files not just ones waiting for a wingman). Following that there is a 24 hour delay before the workunit is deleted. Given delays this is a little more than 2 days of valid reuslts that hang around.

I wouldn't count on that remaining that way since based on database load and file system size we periodically adjust those values. However that is why you are seeing what you are seeing.
[Jul 12, 2017 8:10:06 PM]   Link   Report threatening or abusive post: please login first  Go to top 
SekeRob
Master Cruncher
Joined: Jan 7, 2013
Post Count: 2741
Status: Offline
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

Xml Mapping is not my fav passtime and Office being pre-loaded with many surprises, it'd be a 50/50 card. I'd bet on <ReceivedTime/> but am sure interested the comments by the connoisseurs.

(So very near to a John McClane victory release)
[Jul 12, 2017 8:28:06 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

Since I'm using JSON, I'd prefer just leaving out the field altogether instead of using "ReceivedTime": "". Would I be using XML, I'd adjust my script to the outcome of your query, although I have a slight preference for leaving out the <ReceivedTime/> element in case the event should happen.

(Didn't know that it really shows 'null' in XML, I guessed it was a zero length string, and in JSON the field is already being left out when a WU doesn't have a ReceivedTime (yet), if I'm right. Or does it change to 'null' when the Outcome changes to No Reply?)
----------------------------------------
[Edit 4 times, last edit by adriverhoef at Jul 12, 2017 9:08:26 PM]
[Jul 12, 2017 8:56:56 PM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

I will remove the field if the value is not present. Should be released in the next week and I will update here. If that doesn't work well for you guys, then lets discuss further.
[Jul 13, 2017 6:50:58 PM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: API data fetch gives issues if the "Received" date returns null

This change has been deployed. In the XML response if the value of ReceivedTime is null then the field will not be included in the response.
[Jul 14, 2017 1:20:29 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 11   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread