Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 
World Community Grid Forums
  Category: Support
  Forum: Website Support
  Thread: Something weird with the XML API

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 65
Posts: 65   Pages: 7   [ Previous Page | 1 2 3 4 5 6 7 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 13416 times and has 64 replies Next Thread
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Something weird with the XML API, [Resolved]
It is documented: https://www.worldcommunitygrid.org/help/viewSearch.do?searchString=verification .In old school, find your own way, me looking at available codes on the My Profile page and then quizzing the systems, be it WCG help, BOINC wiki or Google to find out what each of the codes stand for and do. Would [wishful thinking] the My Profile page have help links back to the documentation, the inquisitive, which mouse users are said to be more than keyboard users, latter more productive, the dissemination of that knowledge would be much broader and swifter. Anyway, my tool has maxed out on what can be done with all of the scraped together knowledge... data collection now at a level and duration that a myriad of plots can be developed, just never been asked for, so not allotting any time to such development.

(from the department of sigh ;o)
[Jan 7, 2017 5:38:41 PM]   Link   Report threatening or abusive post: please login first  Go to top 
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1684
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Something weird with the XML API, [Resolved]
Hi SekeRob,
even if I tend to experiment a lot on my own, I am also one of the crazy guy who tries to read the documentation from time to time.
I am maybe not smart enough but it is not documented by the API description (see https://www.worldcommunitygrid.org/help/viewTopic.do?shortName=api). Furthermore, the API description is not really easy to find if you don't know that it is available.
I am not willing to complain, I only hope that this discussion and the earned advices could help other members to find the information they need.
Again, many thanks for your accurate support.
Cheers,
Yves
----------------------------------------

Décrypthon team progress - KerSamson's contribution

----------------------------------------
[Edit 1 times, last edit by KerSamson at Jan 8, 2017 11:12:19 AM]
[Jan 8, 2017 11:07:01 AM]   Link   Report threatening or abusive post: please login first  Go to top 
knreed
Former World Community Grid Tech
Joined: Nov 8, 2004
Post Count: 4504
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Something weird with the XML API, [Resolved]
I tweaked the FAQ description so that the other FAQ is also returned when you search for API
[Jan 9, 2017 10:35:43 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Something weird with the XML API, [Resolved]
Posting a new version of my script called "wcgresults". By running it at regular intervals, say once per hour scheduled through crontab, you can collect a flat database of your results.

With this new version you can also:
* spot results that got 0 points (option -a);
* let the program explain about the meaning of each ExitStatus >= 192 (option -E);
* choose an alternative directory, instead of /var/tmp, to download files (option -D);
* record WUs that are still in progress (probably to some other file using option -f);
* use the option -h to see what options are available. :-)

The 19 fields (including WorkunitId and ResultId) in the flat database are now:
AppName, ClaimedCredit, CpuTime, ElapsedTime, ExitStatus, GrantedCredit, DeviceId, DeviceName, ModTime, WorkunitId, ResultId, Name, Outcome, ReceivedTime, ReportDeadline, SentTime, ServerState, ValidateState, FileDeleteState.

Running the program with the option -f will add records to a logfile that you specify.
Running it without the options -f and -d needs an earlier downloaded file, e.g.: wcgresults -d; : now take note of the downloaded filename, say "RESULT"; wcgresults RESULT;\

In your crontab you can put (where N is the value of the minute within the hour) e.g.:
N * * * * $HOME/bin/wcgresults -df $HOME/tmp/wcg.log

---8< --- SNIP SNIP SNIP --- CUT HERE ---8<
#!/bin/sh
#
# wcgresults - use WCG's API to get a list of recent results
# https://www.worldcommunitygrid.org/help/viewTopic.do?shortName=api
#
#	VERSION: 1.14
#	Author: Adri Verhoef

Prog=${0##*/}

#############
# CONFIGURE #
#############
TMPDIR=/var/tmp					# Directory for storing temporary files
URLBASE=https://www.worldcommunitygrid.org	# Don't change this, unless you know what you're doing
MEMBER="youraccountname"			# The name of your account with which you're crunching
VERIFY="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" 	# Your verification code on 'My Profile'
MEMBER1="anotheraccount"
VERIFY1="11111111111111111111111111111111"
########################
# END OF CONFIGURATION #
########################

usage () {
	printf "%s\n" \
		"Usage: $Prog [options] [-d | file ...]" \
		" Options:" \
		"  -a           [see -f] record also those results that got 0 points all in all" \
		"  -b sortby    sort results by the value of ’sortby’" \
		"               (0 = DeviceId, 1 = SentTime, 2 = ReportDeadline, 3 = ReceivedTime (default), 4 = CpuTime)" \
		"  -d           download a new status file before calculating averages" \
		"  -D dir       download the new status file(s) to directory “dir”" \
		"  -E           only show the textvalue of each defined ExitStatus >= 192" \
		"  -f logfile   record results that went through the validator to file ’logfile’, sorted by CpuTime" \
		"  -l limit     define the number of results to download (default: 250)" \
		"  -o offset    define the number of results to skip when downloading (default: 0)" \
		"  -s state     only select results with ”ValidateState” value ’state’" \
		"               (0 = pending validation, 1 = valid, 2 = invalid," \
		"                4 = pending verification, 5 = results failed to validate within given deadline)" \
		"  -u user      the name of your account on World Community Grid (default: you; ’user’ = 1 ... for other account(s))" \
		"  -w           only select results that are still 'in progress'" \
		;
}

# Options
OPTS=hab:D:dEf:l:o:s:u:w
export opt_a=0
export opt_b=3		# latest received
export opt_D=$TMPDIR
export opt_d=0
export opt_E=0
export opt_f=""
export opt_l=250
export opt_o=0
export opt_s=""
export opt_u=""
export opt_w=0
while getopts $OPTS opt; do
	case $opt in
		h)	usage; exit 0;;
		[adEw])	eval let opt_$opt++;;
		[a-zA-Z0-9])	eval opt_$opt='$OPTARG';;
		?)	usage; exit 1;;
	esac
done
shift $(($OPTIND - 1))

case $opt_b in [01234]) case $opt_b in 0) opt_b=DeviceId;; 1) opt_b=SentTime;; 2) opt_b=ReportDeadline;; 3) opt_b=ReceivedTime;; 4) opt_b=CpuTime;; esac; SortBy="&SortBy=$opt_b";; *) SortBy="";; esac
case $opt_o in 0) Offset="";; *) Offset="&Offset=$opt_o";; esac
case $opt_s in [01245]) ValidateState="&ValidateState=$opt_s";; *) ValidateState="";; esac
case $opt_u in "") ;; *) eval MEMBER=\$MEMBER$opt_u; eval VERIFY=\$VERIFY$opt_u;; esac
case $opt_w in 0) ServerState=5;; *) ServerState=4;; esac

TMPFILE=$opt_D/$Prog.`date +%Y-%m-%dT%H:%M:%S`

PRINT_LONG_FORMAT='
	printf "%-20s %6d", $str_ValidateState[$ValidateState], $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState};
	printf " %6.1f", $cc=($combi_DeviceName_AppName_ValidateState_ClaimedCredit{$DeviceName}{$AppName}{$ValidateState} / $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState});
	printf "/%-6.1f", $g=($combi_DeviceName_AppName_ValidateState_GrantedCredit{$DeviceName}{$AppName}{$ValidateState} / $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState});
	printf " %7.2f", $ct=($combi_DeviceName_AppName_ValidateState_CpuTime{$DeviceName}{$AppName}{$ValidateState} / $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState});
	printf "/%-7.2f", $combi_DeviceName_AppName_ValidateState_ElapsedTime{$DeviceName}{$AppName}{$ValidateState} / $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState};
	printf "\n";
'
ADDITIONAL_HEADER='(ValidationState)           (Claim/Grant)    (Cpu/Elapsed)'
PRINT_LONG_HEADER='
	printf "%-16s %-6s %4d\t'"$ADDITIONAL_HEADER"'\n", $DeviceName, $AppName, $combi_DeviceName_AppName{$DeviceName}{$AppName};
'

calculate_or_record () {
perl -wne '
	BEGIN {
		if ($ENV{opt_f} ne "") {
			if (open(LOGFILE, $ENV{opt_f})) {
				while (<LOGFILE>) {
					chomp;
					@t = split(/\t/);
					$NAME = 11;
					$VALIDATESTATE = 17;
					$workunit{$t[$NAME]} = $t[$VALIDATESTATE];
				}
				close(LOGFILE);
			}
			open(LOGFILE, ">>$ENV{opt_f}") || die "$ENV{Prog}: cannot open logfile ’$ENV{opt_f}’ for writing.\n";
		}
		$outcome_varies = 0;
		$str_Outcome[0] = "(waiting)"; # in progress
		$str_Outcome[1] = "success";
		$str_Outcome[3] = "error";
		$str_Outcome[4] = "no reply";
		$str_Outcome[6] = "validation error";
		$str_Outcome[7] = "abandoned";
		$str_ValidateState[0] = "Pending validation";
		$str_ValidateState[1] = "Valid";
		$str_ValidateState[2] = "Invalid";
		$str_ValidateState[3] = "In progress"; # originally undefined state, so it is now a user defined value
		$str_ValidateState[4] = "Pending verification";
		$str_ValidateState[5] = "Too late";
		#str_ExitStatus[192] = "EXIT_STATEFILE_WRITE";
		$txt_ExitStatus[192] = "The client will exit if it cannot write to the client_state.xml file. Make sure you have permission to write to the BOINC Data directory.";
		#str_ExitStatus[193] = "EXIT_SIGNAL";
		$txt_ExitStatus[193] = "The client, Manager and/or application will exit when getting the exit signal.";
		#str_ExitStatus[194] = "EXIT_ABORTED_BY_CLIENT";
		$txt_ExitStatus[194] = "Task was aborted by the BOINC client.";
		#str_ExitStatus[195] = "EXIT_CHILD_FAILED";
		$txt_ExitStatus[195] = "The science application running the task failed for unknown reasons.";
		#str_ExitStatus[196] = "EXIT_DISK_LIMIT_EXCEEDED";
		$txt_ExitStatus[196] = "The amount of disk space that the task could use was more than the limit set to the task.";
		#str_ExitStatus[197] = "EXIT_TIME_LIMIT_EXCEEDED";
		$txt_ExitStatus[197] = "The amount of time that the task can run was estimated to be far greater than the time limit set to the task.";
		#str_ExitStatus[198] = "EXIT_MEM_LIMIT_EXCEEDED";
		$txt_ExitStatus[198] = "The amount of memory that the task could use was greater than the limit set to the task.";
		#str_ExitStatus[199] = "EXIT_CLIENT_EXITING";
		$txt_ExitStatus[199] = "The task was ended when the client was exited.";
		#str_ExitStatus[200] = "EXIT_UNSTARTED_LATE";
		$txt_ExitStatus[200] = "Task was aborted due to it having not started and already past the deadline.";
		#str_ExitStatus[201] = "EXIT_MISSING_COPROC";
		$txt_ExitStatus[201] = "A coprocessor that was detected earlier went missing. Work for it is being aborted.";
		#str_ExitStatus[202] = "EXIT_ABORTED_BY_PROJECT";
		$txt_ExitStatus[202] = "Tasks on the computer were aborted by the project.";
		#str_ExitStatus[203] = "EXIT_ABORTED_VIA_GUI";
		$txt_ExitStatus[203] = "Applications exit with this code after you told them to abort.";
		#str_ExitStatus[204] = "EXIT_UNKNOWN";
		$txt_ExitStatus[204] = "Unknown exit.";
		if ($ENV{opt_E} == 1) {
			printf "Exit\tMessage\n";
			for ($i = 192; $i <= 204; $i++) {
				printf "%4d\t%s\n", $i, $txt_ExitStatus[$i];
			}
			exit(0);
		}
		$results_available = 0;
	}
	chomp;
	if (/"ResultsReturned": "([^"]+)",/) { print "Results returned: $1\n" unless $ENV{opt_f} ne ""; }
	elsif (/"AppName": "([^"]+)",/) { $AppName = $1; }
	elsif (/"ClaimedCredit": ([^,]+)/) { $ClaimedCredit = $1; }
	elsif (/"CpuTime": ([^,]+)/) { $CpuTime = $1; }
	elsif (/"ElapsedTime": ([^,]+)/) { $ElapsedTime = $1; }
	elsif (/"ExitStatus": ([^,]+)/) { $ExitStatus = $1; }
	elsif (/"GrantedCredit": ([^,]+)/) { $GrantedCredit = $1; }
	elsif (/"DeviceId": ([^,]+)/) { $DeviceId = $1; }			# only used for LOGFILE
	elsif (/"DeviceName": "([^"]+)",/) { $DeviceName = $1; }
	elsif (/"ModTime": ([^,]+)/) { $ModTime = $1; }				# only used for LOGFILE
	elsif (/"WorkunitId": ([^,]+)/) { $WorkunitId = $1; }			# only used for LOGFILE
	elsif (/"ResultId": ([^,]+)/) { $ResultId = $1; }			# only used for LOGFILE
	elsif (/"Name": "([^"]+)",/) { $Name = $1; }
	elsif (/"Outcome": ([^,]+)/) { $Outcome = $1; }
	elsif (/"ReceivedTime": "([^"]+)",/) { $ReceivedTime = $1; }		# only used for LOGFILE
	elsif (/"ReportDeadline": "([^"]+)",/) { $ReportDeadline = $1; }	# only used for LOGFILE
	elsif (/"SentTime": "([^"]+)",/) { $SentTime = $1; }			# only used for LOGFILE
	elsif (/"ServerState": ([^,]+)/) { $ServerState = $1; }
	elsif (/"ValidateState": ([^,]+)/) { $ValidateState = $1; }
	elsif (/"FileDeleteState": ([^,]+)/) { $FileDeleteState = $1; }		# only used for LOGFILE
	if (/"([^"]+)": ("([^"]+)"|[0-9]+(\.[0-9]+)?)$/) { # last item in record, line not ending in comma
		if ($ENV{opt_f} ne "") {
			# ServerState == 5: results that have already been reported back to the server
			# ValidateState == 0: pending validation
			# ValidateState == 4: pending verification
			if ($ServerState == 5 && $ValidateState != 0 && $ValidateState != 4 || $ENV{opt_w} == 1) {
				if (exists $workunit{$Name}) {
					if ($workunit{$Name} != $ValidateState) {
						print STDERR "$Name has its ValidateState changed from $workunit{$Name} to $ValidateState\n"; # Hopefully changed from ”Too Late” into something better. (Or probably opt_w=1)
					} else {
						next; # already recorded in LOGFILE; the names of workunits vary per device, so no need to use $DeviceId or $WorkunitId to distinguish between them
					}
				}
				do {
					next if $ClaimedCredit + $CpuTime + $ElapsedTime + $GrantedCredit == 0.0; # probably: 192 <= $ExitStatus <= 204
				} if $ENV{opt_a} + $ENV{opt_w} == 0; # opt_a=1: record all WUs, also those that got 0 points.
				$ReceivedTime = "" if $ENV{opt_w} == 1;
				printf LOGFILE "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", $AppName, $ClaimedCredit, $CpuTime, $ElapsedTime, $ExitStatus, $GrantedCredit, $DeviceId, $DeviceName, $ModTime, $WorkunitId, $ResultId, $Name, $Outcome, $ReceivedTime, $ReportDeadline, $SentTime, $ServerState, $ValidateState, $FileDeleteState;
			}
		} else {
			if ($Outcome != 1) {
				if ($Outcome == 0) {
					$ValidateState = 3; # In progress
					$ExitStatus = "-";
				} else {
					$results_available = 1;
				}
				$fmt0 = "%-16s %-4s %-9s %-20s %-6s";
				$fmt1 = "%s    %s";
				$fmt2 = "%6.1f/%-6.1f %7.2f/%-7.2f ";
				$fmt3 = "   %s\n";
				printf "$fmt0 $fmt1$fmt3", "(DeviceName)", "App", "(Outcome)", "(ValidationState)", "(Exit)", "(Claim/Grant)", "(Cpu/Elapsed)", "(Name)" if ++$outcome_varies == 1;
				if ($ExitStatus ne "-") {
					if ($ExitStatus >= 192 && $ExitStatus <= 204) {
						if (not exists $DidExitStatus[$ExitStatus]) {
							$DidExitStatus[$ExitStatus] = 1;
							printf "\tExitStatus $ExitStatus: $txt_ExitStatus[$ExitStatus]\n";
						}
					}
				}
				printf "$fmt0 $fmt2$fmt3", $DeviceName, $AppName, $str_Outcome[$Outcome], $str_ValidateState[$ValidateState], $ExitStatus, $ClaimedCredit, $GrantedCredit, $CpuTime, $ElapsedTime, $Name;
			}
			if ($ServerState == 5) {
				$results_available = 1;
				# results that have already been reported back to the server
				$combi_DeviceName_AppName{$DeviceName}{$AppName}++;
				$combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState}++; # number of validated results
				$combi_DeviceName_AppName_ValidateState_ClaimedCredit{$DeviceName}{$AppName}{$ValidateState} += $ClaimedCredit;
				$combi_DeviceName_AppName_ValidateState_CpuTime{$DeviceName}{$AppName}{$ValidateState} += $CpuTime;
				$combi_DeviceName_AppName_ValidateState_ElapsedTime{$DeviceName}{$AppName}{$ValidateState} += $ElapsedTime;
				$combi_DeviceName_AppName_ValidateState_GrantedCredit{$DeviceName}{$AppName}{$ValidateState} += $GrantedCredit;
			}
		}
	}
	END {
		if ($results_available > 0) {
			printf "%s %s %-6s %s %s\n", "=" x 16, " " x 4, " (Number)", " " x 20, "(Number)" if $outcome_varies > 0;
			foreach $DeviceName (keys %combi_DeviceName_AppName) {
				#print "$DeviceName:\n";
				foreach $AppName (keys %{$combi_DeviceName_AppName{$DeviceName}}) {
					if ($ENV{opt_s} eq "") {
						'"$PRINT_LONG_HEADER"'
						foreach $ValidateState (keys %{$combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}}) {
							printf "\t\t\t\t";
							'"$PRINT_LONG_FORMAT"'
						}
					} else {
						foreach $ValidateState (keys %{$combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}}) {
							if ($ENV{opt_s} eq $ValidateState) {
								printf "%-16s %-6s %4d\t", $DeviceName, $AppName, $combi_DeviceName_AppName{$DeviceName}{$AppName};
								'"$PRINT_LONG_FORMAT"'
							}
						}
					}
				}
			}
		}
	}
' "$@"
}

wcgget() {
	[ -n "$opt_f" ] && Offset="&Offset=$skipm"
	wget $QUIET -O $TMPFILE "$URLBASE/api/members/$MEMBER/results?code=$VERIFY&format=json&ServerState=$ServerState$ValidateState$SortBy$Offset&Limit=$opt_l" && [ -s $TMPFILE ] &&
	calculate_or_record $TMPFILE
	if [ -n "$opt_f" ]; then
		mv $TMPFILE $TMPFILE.$skipm
		[ -s $TMPFILE.$skipm ] || exit 1
	fi
}

wcgget_cumulative() {
	SortBy="&SortBy=CpuTime" # CpuTime is rather unique for each WU, in order to prevent fetching duplicate WU-entries from the database (resulting from overlap in two or more fetches) using other key selections such as SentTime.
				 # (https://www.worldcommunitygrid.org/help/viewTopic.do?shortName=api says "Default is SentTime.")
	wcgget
	let avail=`perl -wne 'print if s/\s+"ResultsAvailable": "([^"]+)",\s+/$1/' $TMPFILE.$skipm` #     "ResultsAvailable": "395",
	let toget=$avail-$opt_l
	while [ $toget -gt 0 ]; do let skipm=$skipm+$opt_l; wcgget; let toget=$toget-$opt_l; done
}

if [ $opt_E = 1 ]; then
	opt_d=0
fi
if [ $opt_d = 0 ]; then
	calculate_or_record "$@"
else
	let skipm=0 # offset
	if [ -n "$opt_f" ]
	then
		QUIET=-q
		wcgget_cumulative
	else
		QUIET=""
		wcgget
	fi
fi
[Jan 12, 2017 2:52:37 AM]   Link   Report threatening or abusive post: please login first  Go to top 
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1684
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Something weird with the XML API, [Resolved]
Many thanks Adri, I will try it.
Cheers,
Yves
----------------------------------------

Décrypthon team progress - KerSamson's contribution
[Jan 14, 2017 9:20:42 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Something weird with the XML API
This case is really something weird with the XML API:

On Thu 19 Jan 2017 at 23:59:11 (UTC) I downloaded a JSON status file of validated results and here's one of the WUs:
        {
           "AppName": "fahv",
           "ClaimedCredit": 3.422033411926014,
           "CpuTime": 0.03998752683401108,
           "ElapsedTime": 0.048217497766017914,
           "ExitStatus": 0,
           "GrantedCredit": 3.422033411926014,
           "DeviceId": *******,
           "DeviceName": "*****",
           "ModTime": 1484868028,
           "WorkunitId": 1924524908,
           "ResultId": 1208051036,
           "Name": "FAHV_1000840_3j3q-k9-P3_7629_0",
           "Outcome": 1,
           "ReceivedTime": "2017-01-29T00:57:36",
           "ReportDeadline": "2017-01-19T00:57:36",
           "SentTime": "2017-01-19T22:33:33",
           "ServerState": 5,
           "ValidateState": 1,
           "FileDeleteState": 2
        },

That's right: the ReceivedTime lies 10 days in the future! (... and the ReportDeadline is more than 21 hours earlier than the SentTime!)
It must be some temporary problem, because when I noticed it it was already 15 minutes later, so I decided to take another stab at downloading a new JSON file and now, 22 minutes later, the ReceivedTime and the ReportDeadline have changed to more acceptable values :-) as follows:
        {
           "AppName": "fahv",
           "ClaimedCredit": 3.422033411926014,
           "CpuTime": 0.03998752683401108,
           "ElapsedTime": 0.048217497766017914,
           "ExitStatus": 0,
           "GrantedCredit": 3.422033411926014,
           "DeviceId": *******,
           "DeviceName": "*****",
           "ModTime": 1484868028,
           "WorkunitId": 1924524908,
           "ResultId": 1208051036,
           "Name": "FAHV_1000840_3j3q-k9-P3_7629_0",
           "Outcome": 1,
           "ReceivedTime": "2017-01-19T23:14:25",
           "ReportDeadline": "2017-01-29T22:33:33",
           "SentTime": "2017-01-19T22:33:33",
           "ServerState": 5,
           "ValidateState": 1,
           "FileDeleteState": 2
        },

Here's a diff (produced by vimdiff) concerning this particular result:
localtime: 2017-01-20 00:59:11         localtime: 2017-01-20 01:22:03
+ +--  8 lines: {-------------------------------------|+ +--  8 lines: {-------------------------------------
             "DeviceName": "*****",                   |             "DeviceName": "*****",
             "ModTime": 1484868028,                   |             "ModTime": 1484868028,
             "WorkunitId": 1924524908,                |             "WorkunitId": 1924524908,
             "ResultId": 1208051036,                  |             "ResultId": 1208051036,
             "Name": "FAHV_1000840_3j3q-k9-P3_7629_0",|             "Name": "FAHV_1000840_3j3q-k9-P3_7629_0",
             "Outcome": 1,                            |             "Outcome": 1,
             "ReceivedTime": "2017-01-29T00:57:36",   |             "ReceivedTime": "2017-01-19T23:14:25",   
             "ReportDeadline": "2017-01-19T00:57:36", |             "ReportDeadline": "2017-01-29T22:33:33",
             "SentTime": "2017-01-19T22:33:33",       |             "SentTime": "2017-01-19T22:33:33",
             "ServerState": 5,                        |             "ServerState": 5,
             "ValidateState": 1,                      |             "ValidateState": 1,
             "FileDeleteState": 2                     |             "FileDeleteState": 2
          },                                          |          },

----------------------------------------
[Edit 2 times, last edit by adriverhoef at Jan 20, 2017 1:07:50 AM]
[Jan 20, 2017 1:04:57 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Something weird with the XML API
Found another strange case, caught at 2017-01-19 17:59:04 (UTC):
        {
           "AppName": "fahv",
           "ClaimedCredit": 11.91591994130389,
           "CpuTime": 0.34928473830223083,
           "ElapsedTime": 0.4161713421344757,
           "ExitStatus": 0,
           "GrantedCredit": 11.91591994130389,
           "DeviceId": *******,
           "DeviceName": "*****",
           "ModTime": 1484847829,
           "WorkunitId": 1923968843,
           "ResultId": 1207412794,
           "Name": "FAHV_1000813_3j3q-_I-P3_Rigid_0786_0",
           "Outcome": 1,
           "ReceivedTime": "2017-01-19T17:43:41",
           "ReportDeadline": "2017-01-19T17:41:17",
           "SentTime": "2017-01-29T08:23:30",
           "ServerState": 5,
           "ValidateState": 1,
           "FileDeleteState": 2
        },
Here, the SentTime lies 10 days in the future, 10 days beyond the ReceivedTime and ReportDeadline.

One hour later, everything was looking normal again:
        {
           "AppName": "fahv",
           "ClaimedCredit": 11.91591994130389,
           "CpuTime": 0.34928473830223083,
           "ElapsedTime": 0.4161713421344757,
           "ExitStatus": 0,
           "GrantedCredit": 11.91591994130389,
           "DeviceId": *******,
           "DeviceName": "*****",
           "ModTime": 1484847829,
           "WorkunitId": 1923968843,
           "ResultId": 1207412794,
           "Name": "FAHV_1000813_3j3q-_I-P3_Rigid_0786_0",
           "Outcome": 1,
           "ReceivedTime": "2017-01-19T17:43:41",
           "ReportDeadline": "2017-01-29T06:36:06",
           "SentTime": "2017-01-19T06:36:06",
           "ServerState": 5,
           "ValidateState": 1,
           "FileDeleteState": 2
        },
(SentTime and ReportDeadline are now showing a more reasonable value. :-) )
[Jan 20, 2017 2:09:37 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Something weird with the XML API
Searching for more anomalies I found 95 occurrences in total since 25-12-2016, where SentTime lies way beyond ReceivedTime:
SentTime             ReportDeadline       ReceivedTime         Name
2016-12-26T08:50:42  2017-01-05T08:50:42  2016-12-25T02:31:47  FAHV_1000257_3j3q-_i-P1_0753_0
2017-01-05T12:02:04  2017-12-26T13:48:21  2016-12-26T13:23:08  FAHV_1000260_3j3q-_i-P1_14228_0
2016-12-26T12:02:04  2017-01-05T12:02:04  2016-12-26T11:32:24  FAHV_1000260_3j3q-_i-P1_2944_0
2016-12-26T12:32:22  2017-12-25T23:45:34  2016-12-04T23:45:34  FAHV_1000261_3j3q-_i-P1_20479_0
2017-01-05T07:53:31  2017-01-05T13:39:13  2016-12-26T17:28:26  FAHV_1000262_3j3q-_i-P1_22984_0
2017-01-05T12:26:06  2016-12-26T14:56:55  2016-12-26T17:47:26  FAHV_1000263_3j3q-_i-P1_24025_0
2016-12-26T09:02:39  2017-01-05T09:02:39  2016-12-26T04:06:13  FAHV_1000257_3j3q-_i-P1_14950_0
2016-12-29T09:58:48  2017-01-08T09:58:48  2016-12-29T07:08:50  FAHV_1000300_3j3y-iy-P1_7715_0
2016-12-29T09:56:43  2017-01-08T09:56:43  2016-12-29T05:48:37  FAHV_1000300_3j3y-iy-P1_7011_0
2016-12-29T13:04:57  2017-01-08T13:04:57  2016-12-26T02:44:31  FAHV_1000303_3j3y-iy-P1_Rigid_1670_0
2016-12-31T14:31:05  2017-01-10T14:31:05  2016-12-29T11:32:32  FAHV_1000323_3j3y-_o-P1_0948_0
2016-12-31T13:51:53  2017-01-10T13:51:53  2016-12-31T04:04:02  FAHV_1000322_3j3y-_o-P1_19487_0
2017-01-01T14:56:56  2017-01-11T00:28:35  2017-01-01T02:04:39  FAHV_1000335_3j3y-_o-P1_Rigid_18124_0
2017-12-31T04:04:02  2017-01-11T00:28:35  2017-01-01T02:02:30  FAHV_1000335_3j3y-_o-P1_Rigid_18167_0
2017-01-11T00:35:09  2017-01-11T00:58:53  2017-01-01T15:53:08  FAHV_1000335_3j3y-_o-P1_Rigid_7854_0
2017-01-11T09:36:20  2017-01-05T18:50:27  2017-01-03T01:51:36  FAHV_1000281_3j3q-_i-P1_Rigid_4505_3
2017-01-13T05:12:21  2017-01-03T06:26:13  2017-01-03T10:37:55  FAHV_1000356_3j3y-3S-P2_8992_0
2017-01-03T07:20:37  2017-01-13T07:20:37  2017-01-03T02:07:06  FAHV_1000355_3j3y-3S-P2_23834_0
2017-01-13T01:49:55  2017-01-13T11:33:54  2017-01-03T12:38:55  FAHV_1000359_3j3y-3S-P2_4586_0
2017-01-13T02:54:33  2017-01-03T04:13:38  2017-01-03T12:07:26  FAHV_1000359_3j3y-3S-P2_11925_0
2017-01-13T15:11:40  2017-01-13T16:26:59  2017-01-03T18:04:07  FAHV_1000365_3j3y-3S-P2_27045_0
2017-01-04T00:54:27  2017-01-14T00:54:27  2017-01-03T19:05:02  FAHV_1000374_3j3y-3S-P2_Rigid_13189_0
2017-01-04T14:47:27  2017-01-14T14:47:27  2017-01-04T04:50:40  FAHV_1000383_3j3y-3S-P2_Rigid_7601_0
2017-01-04T14:47:25  2017-01-14T14:47:25  2017-01-04T00:04:33  FAHV_1000383_3j3y-3S-P2_Rigid_7630_0
2017-01-05T00:49:21  2017-01-15T00:49:21  2017-01-04T23:41:01  FAHV_1000394_4xfx_P2_14632_0
2017-01-14T21:16:13  2017-01-04T22:48:11  2017-01-05T01:31:58  FAHV_1000393_4xfx_P2_24962_0
2017-01-15T00:49:21  2017-01-05T02:10:23  2017-01-05T01:08:45  FAHV_1000393_4xfx_P2_19807_0
2017-01-05T00:17:47  2017-01-15T00:17:47  2017-01-05T00:07:07  FAHV_1000393_4xfx_P2_26578_0
2017-01-05T00:02:56  2017-01-15T00:02:56  2017-01-04T19:09:10  FAHV_1000393_4xfx_P2_19814_0
2017-01-05T02:52:42  2017-01-15T02:52:42  2017-01-05T02:06:43  FAHV_1000397_4xfx_P2_14890_0
2017-01-15T03:05:39  2017-01-15T04:30:24  2017-01-05T05:44:04  FAHV_1000399_4xfx_P2_3996_0
2017-01-05T05:25:07  2017-01-15T05:25:07  2017-01-05T04:42:36  FAHV_1000401_4xfx_P2_18674_0
2017-01-05T05:42:00  2017-01-05T04:11:16  2017-01-05T04:11:16  FAHV_1000401_4xfx_P2_5338_0
2017-01-14T09:23:49  2017-01-04T14:47:04  2017-01-05T14:57:55  FAHV_1000408_4xfx_P2_Rigid_25323_0
2017-01-15T06:28:45  2017-01-05T09:56:47  2017-01-05T17:43:06  FAHV_1000412_4xfx_P2_Rigid_16127_0
2017-01-15T08:31:21  2017-01-05T14:16:14  2017-01-05T17:43:06  FAHV_1000412_4xfx_P2_Rigid_15343_0
2017-01-06T08:39:36  2017-01-16T08:39:36  2017-01-06T05:15:46  FAHV_1000428_3j3y-_6-P2_18349_1
2017-01-06T08:41:56  2017-01-16T08:41:56  2017-01-06T00:59:39  FAHV_1000430_3j3y-_6-P2_22637_0
2017-01-06T11:52:18  2017-01-16T11:52:18  2017-01-06T07:42:42  FAHV_1000436_3j3y-_6-P2_4423_0
2017-01-07T04:54:47  2017-01-17T00:46:02  2017-01-07T01:26:58  FAHV_1000452_3j3y-_6-P2_Rigid_17121_2
2017-01-07T01:44:42  2017-01-17T01:44:42  2017-01-06T19:25:30  FAHV_1000450_3j3y-_6-P2_Rigid_11531_0
2017-01-16T12:00:19  2017-01-06T13:44:38  2017-01-07T06:18:54  FAHV_1000450_3j3y-_6-P2_Rigid_1155_0
2017-01-07T14:43:15  2017-01-17T05:02:23  2017-01-07T08:21:36  FAHV_1000465_3j3q-8O-P2_7043_0
2017-01-07T14:39:06  2017-01-17T14:39:06  2017-01-06T12:53:27  FAHV_1000465_3j3q-8O-P2_3410_0
2017-01-17T18:15:58  2017-01-07T19:22:21  2017-01-08T10:09:20  FAHV_1000482_3j3q-8O-P2_Rigid_27599_0
2017-01-18T15:33:57  2017-01-18T17:07:08  2017-01-08T17:30:29  FAHV_1000495_3j3y-c4-P2_0715_0
2017-01-18T15:19:24  2017-01-18T14:47:10  2017-01-08T17:07:08  FAHV_1000491_3j3y-c4-P2_3493_0
2017-01-08T17:17:50  2017-01-18T17:17:50  2017-01-07T18:24:35  FAHV_1000471_3j3q-8O-P2_Rigid_11887_2
2017-01-08T17:17:49  2017-01-18T17:17:49  2017-01-07T17:32:51  FAHV_1000471_3j3q-8O-P2_Rigid_11875_2
2017-01-09T15:04:19  2017-01-19T00:19:06  2017-01-09T05:41:17  OET1_0004340_x3MWP_rig_81063_0
2017-01-08T23:32:05  2017-01-18T23:32:05  2017-01-08T09:03:33  OET1_0004340_x3MWP_rig_23769_1
2017-01-09T00:16:57  2017-01-19T00:16:57  2017-01-08T10:17:47  OET1_0004340_x3MWP_rig_72680_0
2017-01-09T02:52:59  2017-01-19T02:52:59  2017-01-09T01:16:53  OET1_0004342_x3MWP_rig_91200_0
2017-01-20T02:09:06  2017-01-19T11:10:19  2017-01-09T20:54:28  FAHV_1000511_3j3y-c4-P2_Rigid_12501_0
SentTime             ReportDeadline       ReceivedTime         Name
2017-01-20T16:20:34  2017-01-20T16:05:35  2017-01-10T22:32:37  FAHV_1000541_3j3y-_M-P2_Rigid_12841_0
2017-01-11T21:03:37  2017-01-21T01:41:46  2017-01-11T10:52:25  FAHV_1000551_3j3y-_M-P2_Rigid_0910_0
2017-01-11T00:33:54  2017-01-21T00:33:54  2017-01-10T15:28:12  FAHV_1000550_3j3y-_M-P2_Rigid_1037_0
2017-01-11T02:42:30  2017-01-20T15:21:01  2017-01-10T20:29:52  FAHV_1000540_3j3y-_M-P2_Rigid_11365_0
2017-01-21T00:33:53  2017-01-11T09:51:45  2017-01-11T10:52:25  FAHV_1000551_3j3y-_M-P2_Rigid_0978_0
2017-01-10T17:03:04  2017-01-20T17:03:04  2017-01-09T18:32:13  FAHV_1000542_3j3y-_M-P2_Rigid_3422_0
2017-01-10T17:40:09  2017-01-20T17:40:09  2017-01-09T17:26:38  FAHV_1000543_3j3y-_M-P2_Rigid_13188_0
2017-01-11T06:01:25  2017-01-20T15:30:18  2017-01-10T21:32:28  FAHV_1000540_3j3y-_M-P2_Rigid_14868_0
2017-01-10T15:06:40  2017-01-19T17:31:59  2017-01-09T20:54:28  FAHV_1000539_3j3y-_M-P2_Rigid_6617_0
2017-01-20T14:51:36  2017-01-21T02:42:30  2017-01-11T11:53:00  FAHV_1000551_3j3y-_M-P2_Rigid_8923_0
2017-01-20T00:07:00  2017-01-10T09:25:44  2017-01-10T21:32:28  FAHV_1000541_3j3y-_M-P2_Rigid_12834_0
2017-01-11T09:51:45  2017-01-20T17:16:00  2017-01-11T03:43:13  FAHV_1000542_3j3y-_M-P2_Rigid_7497_0
2017-01-10T21:32:29  2017-01-20T21:32:29  2017-01-10T16:07:43  FAHV_1000547_3j3y-_M-P2_Rigid_14409_0
2017-01-11T02:42:30  2017-01-20T15:21:01  2017-01-10T20:29:52  FAHV_1000551_3j3y-_M-P2_Rigid_8963_0
2017-01-10T22:32:37  2017-01-20T22:32:37  2017-01-10T07:47:52  FAHV_1000520_3j3y-c4-P2_Rigid_1446_2
2017-01-11T05:45:34  2017-01-21T05:45:34  2017-01-10T16:05:35  FAHV_1000554_3j3y-_M-P2_Rigid_5391_0
2017-01-11T04:44:59  2017-01-21T04:44:59  2017-01-11T00:38:49  FAHV_1000553_3j3y-_M-P2_Rigid_6771_0
2017-01-20T21:32:29  2017-01-20T15:21:16  2017-01-10T21:03:37  FAHV_1000540_3j3y-_M-P2_Rigid_11454_0
2017-01-21T03:43:14  2017-01-11T12:54:09  2017-01-11T08:49:51  FAHV_1000549_3j3y-_M-P2_Rigid_11811_0
2017-01-11T07:47:52  2017-01-20T16:05:35  2017-01-10T21:32:28  FAHV_1000541_3j3y-_M-P2_Rigid_12806_0
2017-01-11T17:44:01  2017-01-21T11:53:01  2017-01-11T17:25:17  FAHV_1000573_3j3q-_w-P2_Rigid_2069_0
2017-01-12T17:12:52  2017-01-22T17:12:52  2017-01-11T13:44:29  FAHV_1000595_3j3y-_y-P2_6043_0
2017-01-21T23:38:44  2017-01-12T02:44:24  2017-01-11T16:05:06  FAHV_1000568_3j3q-_w-P2_2192_0
2017-01-22T15:32:55  2017-01-12T16:58:39  2017-01-12T19:51:02  FAHV_1000595_3j3y-_y-P2_6021_0
2017-01-22T16:45:31  2017-01-21T17:54:57  2017-01-11T16:05:06  FAHV_1000568_3j3q-_w-P2_2196_0
2017-01-22T17:14:58  2017-01-12T19:51:03  2017-01-12T19:13:45  FAHV_1000595_3j3y-_y-P2_6041_0
2017-01-23T23:52:00  2017-01-14T01:05:22  2017-01-14T06:57:10  FAHV_1000638_4xfx_P3_10617_0
2017-01-14T05:44:54  2017-01-24T05:44:54  2017-01-07T15:19:02  FAHV_1000639_4xfx_P3_7693_0
2017-01-14T08:07:23  2017-01-24T08:07:23  2017-01-14T00:06:14  OET1_0004355_x3MWT_rig_63391_0
2017-01-25T17:01:38  2017-01-25T17:01:38  2017-01-16T03:53:05  OET1_0004380_x3MX2p_rig_3046_0
2017-01-24T09:08:50  2017-01-25T04:38:58  2017-01-16T03:53:05  OET1_0004380_x3MX2p_rig_30285_0
2017-01-17T01:17:55  2017-01-27T01:17:55  2017-01-16T23:04:11  FAHV_1000729_3j3q-k9-P3_7688_0
2017-01-18T23:22:52  2017-01-28T04:06:03  2017-01-18T01:10:52  FAHV_1000773_3j3y-gO-P3_5131_0
2017-01-29T08:23:30  2017-01-19T17:41:17  2017-01-19T17:43:41  FAHV_1000813_3j3q-_I-P3_Rigid_0786_0
2017-01-19T22:33:32  2017-01-29T22:33:43  2017-01-19T20:00:19  FAHV_1000840_3j3q-k9-P3_7345_0
2017-01-28T23:17:10  2017-01-29T19:40:54  2017-01-19T23:16:24  FAHV_1000835_3j3q-k9-P3_3665_0
2017-01-28T05:07:59  2017-01-29T22:33:33  2017-01-19T23:08:03  FAHV_1000840_3j3q-k9-P3_7636_0
2017-01-19T22:33:32  2017-01-29T22:33:32  2017-01-19T04:16:43  FAHV_1000840_3j3q-k9-P3_7626_0
2017-01-19T19:40:54  2017-01-29T19:40:54  2017-01-19T05:03:40  FAHV_1000835_3j3q-k9-P3_3644_0
2017-01-19T22:33:32  2017-01-29T22:33:32  2017-01-19T03:01:26  FAHV_1000840_3j3q-k9-P3_7508_0
2017-01-29T06:19:08  2017-01-29T20:43:09  2017-01-20T05:12:24  FAHV_1000837_3j3q-k9-P3_2569_0
SentTime             ReportDeadline       ReceivedTime         Name
[Jan 20, 2017 1:42:30 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2346
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Version 1.18 of my script 'wcgresults'
Posting a new version of my script called "wcgresults". By running it at regular intervals — say once per hour scheduled through crontab — you can collect a flat database of your results.

New in this version:
* option -q
- with this option it is possible to spot the number of jobs in progress per device.
Examples, run from a terminal screen:
$ wcgresults -dq
18:37:32 device_a 9 device_b 3 device_c 12
There is more information if you double the 'q':
$ while sleep 3600; do wcgresults -dqq; done
15:08:34 device_a mcm1:9 oet1:4 zika:1 device_b scc1:3 device_c mcm1:9
16:08:36 device_a mcm1:5 oet1:6 zika:3 device_b scc1:3 device_c mcm1:11
17:08:38 device_a mcm1:3 oet1:7 zika:3 device_b scc1:3 device_c mcm1:12

* option -O (= capital letter O)
- let the program explain about the meaning of each Outcome
* option -S (= capital letter S)
- let the program explain about the meaning of each ServerState
* option -V (= capital letter V)
- let the program explain about the meaning of each ValidateState

Running the program with the option -f will add records to a logfile that you specify.

In your crontab you can put (where N is the value of the minute within the hour) e.g.:
N * * * * $HOME/bin/wcgresults -df $HOME/tmp/wcg.log

Select the text below and save it to a file called "wcgresults" in your directory $HOME/bin (and do "chmod +x ~/bin/wcgresults").

---8< --- SNIP SNIP SNIP --- CUT HERE ---8<
#!/bin/sh
#
# wcgresults - use WCG's API to get a list of recent results
# https://www.worldcommunitygrid.org/help/viewTopic.do?shortName=api
#
#	VERSION: 1.18
#	Author: Adri Verhoef

Prog=${0##*/}

#############
# CONFIGURE #
#############
TMPDIR=/var/tmp					# Directory for storing temporary files
URLBASE=https://www.worldcommunitygrid.org	# Don't change this, unless you know what you're doing
MEMBER="youraccountname"			# The name of your account with which you're crunching
VERIFY="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" 	# Your verification code on 'My Profile'
MEMBER1="anotheraccount"
VERIFY1="11111111111111111111111111111111"
########################
# END OF CONFIGURATION #
########################

usage () {
	printf "%s\n" \
		"Usage: $Prog [options] [-d | file ...]" \
		" Options:" \
		"  -a           [see -f] record also those results that got 0 points all in all" \
		"  -b sortby    sort results by the value of ’sortby’" \
		"               (0 = DeviceId, 1 = SentTime, 2 = ReportDeadline, 3 = ReceivedTime (default), 4 = CpuTime)" \
		"  -d           download a new status file before calculating averages" \
		"  -D dir       download the new status file(s) to directory “dir”" \
		"  -E           only show the textvalue of each defined ExitStatus >= 192" \
		"  -f logfile   record results that went through the validator to file ’logfile’, sorted by CpuTime" \
		"  -l limit     define the number of results to download (default: 250)" \
		"  -O           only show the textvalue of each defined Outcome" \
		"  -o offset    define the number of results to skip when downloading (default: 0)" \
		"  -q           only show the number of jobs in progress per device (”-qq” will also list the apps)" \
		"  -S           only show the textvalue of each defined ServerState" \
		"  -s state     only select results with ”ValidateState” value ’state’" \
		"               (0 = pending validation, 1 = valid, 2 = invalid," \
		"                4 = pending verification, 5 = results failed to validate within given deadline)" \
		"  -u user      the name of your account on World Community Grid (default: you; ’user’ = 1 ... for other account(s))" \
		"  -V           only show the textvalue of each defined ValidateState" \
		"  -w           only select results that are still 'in progress'" \
		;
}

# Options
OPTS=hab:D:dEf:l:Oo:qSs:u:Vw
export opt_a=0
export opt_b=3		# latest received
export opt_D=$TMPDIR
export opt_d=0
export opt_E=0
export opt_f=""
export opt_l=250
export opt_o=0
export opt_O=0
export opt_q=0		# show the queue
export opt_S=0
export opt_s=""
export opt_u=""
export opt_V=0
export opt_w=0
while getopts $OPTS opt; do
	case $opt in
		h)	usage; exit 0;;
		[adEOqSVw])	eval let opt_$opt++;;
		[a-zA-Z0-9])	eval opt_$opt='$OPTARG';;
		?)	usage; exit 1;;
	esac
done
shift $(($OPTIND - 1))

case $opt_b in [01234]) case $opt_b in 0) opt_b=DeviceId;; 1) opt_b=SentTime;; 2) opt_b=ReportDeadline;; 3) opt_b=ReceivedTime;; 4) opt_b=CpuTime;; esac; SortBy="&SortBy=$opt_b";; *) SortBy="";; esac
case $opt_o in 0) Offset="";; *) Offset="&Offset=$opt_o";; esac
case $opt_q in 0) ;; *) opt_w=1;; esac # see below
case $opt_s in [01245]) ValidateState="&ValidateState=$opt_s";; *) ValidateState="";; esac
case $opt_u in "") ;; *) eval MEMBER=\$MEMBER$opt_u; eval VERIFY=\$VERIFY$opt_u;; esac
case $opt_w in 0) ServerState=5;; *) ServerState=4;; esac

TMPFILE=$opt_D/$Prog.`date +%Y-%m-%dT%H:%M:%S`

PRINT_LONG_FORMAT='
	printf "%-20s %6d", $txt_ValidateState[$ValidateState], $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState};
	printf " %6.1f", $cc=($combi_DeviceName_AppName_ValidateState_ClaimedCredit{$DeviceName}{$AppName}{$ValidateState} / $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState});
	printf "/%-6.1f", $g=($combi_DeviceName_AppName_ValidateState_GrantedCredit{$DeviceName}{$AppName}{$ValidateState} / $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState});
	printf " %7.2f", $ct=($combi_DeviceName_AppName_ValidateState_CpuTime{$DeviceName}{$AppName}{$ValidateState} / $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState});
	printf "/%-7.2f", $combi_DeviceName_AppName_ValidateState_ElapsedTime{$DeviceName}{$AppName}{$ValidateState} / $combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState};
	printf "\n";
'
ADDITIONAL_HEADER='(ValidateState)             (Claim/Grant)    (Cpu/Elapsed)'
PRINT_LONG_HEADER='
	printf "%-16s %-6s %4d\t'"$ADDITIONAL_HEADER"'\n", $DeviceName, $AppName, $combi_DeviceName_AppName{$DeviceName}{$AppName};
'

calculate_or_record () {
perl -wne '
	BEGIN {
		$txt_Outcome[0] = "(waiting)"; # in progress
		$txt_Outcome[1] = "success";
		$txt_Outcome[2] = ""; # UNDEFINED
		$txt_Outcome[3] = "error";
		$txt_Outcome[4] = "no reply";
		$txt_Outcome[5] = ""; # UNDEFINED
		$txt_Outcome[6] = "validation error";
		$txt_Outcome[7] = "abandoned";
		$txt_ServerState[4] = "Still in progress";
		$txt_ServerState[5] = "Already reported back to the server";
		$txt_ValidateState[0] = "Pending validation";
		$txt_ValidateState[1] = "Valid";
		$txt_ValidateState[2] = "Invalid";
		$txt_ValidateState[3] = "In progress"; # UNDEFINED, see option -w
		$txt_ValidateState[4] = "Pending verification";
		$txt_ValidateState[5] = "Too late";
		#str_ExitStatus[192] = "EXIT_STATEFILE_WRITE";
		$txt_ExitStatus[192] = "The client will exit if it cannot write to the client_state.xml file. Make sure you have permission to write to the BOINC Data directory.";
		#str_ExitStatus[193] = "EXIT_SIGNAL";
		$txt_ExitStatus[193] = "The client, Manager and/or application will exit when getting the exit signal.";
		#str_ExitStatus[194] = "EXIT_ABORTED_BY_CLIENT";
		$txt_ExitStatus[194] = "Task was aborted by the BOINC client.";
		#str_ExitStatus[195] = "EXIT_CHILD_FAILED";
		$txt_ExitStatus[195] = "The science application running the task failed for unknown reasons.";
		#str_ExitStatus[196] = "EXIT_DISK_LIMIT_EXCEEDED";
		$txt_ExitStatus[196] = "The amount of disk space that the task could use was more than the limit set to the task.";
		#str_ExitStatus[197] = "EXIT_TIME_LIMIT_EXCEEDED";
		$txt_ExitStatus[197] = "The amount of time that the task can run was estimated to be far greater than the time limit set to the task.";
		#str_ExitStatus[198] = "EXIT_MEM_LIMIT_EXCEEDED";
		$txt_ExitStatus[198] = "The amount of memory that the task could use was greater than the limit set to the task.";
		#str_ExitStatus[199] = "EXIT_CLIENT_EXITING";
		$txt_ExitStatus[199] = "The task was ended when the client was exited.";
		#str_ExitStatus[200] = "EXIT_UNSTARTED_LATE";
		$txt_ExitStatus[200] = "Task was aborted due to it having not started and already past the deadline.";
		#str_ExitStatus[201] = "EXIT_MISSING_COPROC";
		$txt_ExitStatus[201] = "A coprocessor that was detected earlier went missing. Work for it is being aborted.";
		#str_ExitStatus[202] = "EXIT_ABORTED_BY_PROJECT";
		$txt_ExitStatus[202] = "Tasks on the computer were aborted by the project.";
		#str_ExitStatus[203] = "EXIT_ABORTED_VIA_GUI";
		$txt_ExitStatus[203] = "Applications exit with this code after you told them to abort.";
		#str_ExitStatus[204] = "EXIT_UNKNOWN";
		$txt_ExitStatus[204] = "Unknown exit.";
		if ($ENV{opt_E} == 1) {
			printf "Exit\tMeaning\n";
			for ($i = 192; $i <= 204; $i++) {
				printf "%4d\t%s\n", $i, $txt_ExitStatus[$i];
			}
			exit(0);
		}
		if ($ENV{opt_O} == 1) {
			printf "Outc\tMeaning\n";
			for ($i = 0; $i <= 7; $i++) {
				printf "%4d\t%s\n", $i, $txt_Outcome[$i] unless $txt_Outcome[$i] eq "";
			}
			exit(0);
		}
		if ($ENV{opt_S} == 1) {
			printf "Server-\nState\tMeaning\n";
			for ($i = 4; $i <= 5; $i++) {
				printf "%4d\t%s\n", $i, $txt_ServerState[$i];
			}
			exit(0);
		}
		if ($ENV{opt_V} == 1) {
			printf "Validate-\nState\tMeaning\n";
			for ($i = 0; $i <= 5; $i++) {
				printf "%4d\t%s\n", $i, $txt_ValidateState[$i] unless $i == 3;
			}
			exit(0);
		}
		if ($ENV{opt_f} ne "") {
			if (open(LOGFILE, $ENV{opt_f})) {
				while (<LOGFILE>) {
					chomp;
					@t = split(/\t/);
					$GRANTEDCREDIT = 5;
					#$MODTIME = 8;
					$NAME = 11;
					$VALIDATESTATE = 17;
					$grantedc{$t[$NAME]} = $t[$GRANTEDCREDIT];
					#$modtime{$t[$NAME]} = $t[$MODTIME]; # If a workunit is user aborted, it will have its ModTime changed later on and we do not want to record that. So forget this. :-)
					$workunit{$t[$NAME]} = $t[$VALIDATESTATE];
				}
				close(LOGFILE);
			}
			open(LOGFILE, ">>$ENV{opt_f}") || die "$ENV{Prog}: cannot open logfile ’$ENV{opt_f}’ for writing.\n";
		}
		$outcome_varies = 0;
		$result_matches = 0;
	}
	chomp;
	if (/"ResultsReturned": "([^"]+)",/) { print "Results returned: $1\n" unless $ENV{opt_f} ne ""; }
	elsif (/"AppName": "([^"]+)",/) { $AppName = $1; }
	elsif (/"ClaimedCredit": ([^,]+)/) { $ClaimedCredit = $1; }
	elsif (/"CpuTime": ([^,]+)/) { $CpuTime = $1; }
	elsif (/"ElapsedTime": ([^,]+)/) { $ElapsedTime = $1; }
	elsif (/"ExitStatus": ([^,]+)/) { $ExitStatus = $1; }
	elsif (/"GrantedCredit": ([^,]+)/) { $GrantedCredit = $1; }
	elsif (/"DeviceId": ([^,]+)/) { $DeviceId = $1; }			# only used for LOGFILE
	elsif (/"DeviceName": "([^"]+)",/) { $DeviceName = $1; }
	elsif (/"ModTime": ([^,]+)/) { $ModTime = $1; }				# only used for LOGFILE
	elsif (/"WorkunitId": ([^,]+)/) { $WorkunitId = $1; }			# only used for LOGFILE
	elsif (/"ResultId": ([^,]+)/) { $ResultId = $1; }			# only used for LOGFILE
	elsif (/"Name": "([^"]+)",/) { $Name = $1; }
	elsif (/"Outcome": ([^,]+)/) { $Outcome = $1; }
	elsif (/"ReceivedTime": "([^"]+)",/) { $ReceivedTime = $1; }		# only used for LOGFILE
	elsif (/"ReportDeadline": "([^"]+)",/) { $ReportDeadline = $1; }	# only used for LOGFILE
	elsif (/"SentTime": "([^"]+)",/) { $SentTime = $1; }			# only used for LOGFILE
	elsif (/"ServerState": ([^,]+)/) { $ServerState = $1; }
	elsif (/"ValidateState": ([^,]+)/) { $ValidateState = $1; }
	elsif (/"FileDeleteState": ([^,]+)/) { $FileDeleteState = $1; }		# only used for LOGFILE
	if (/"([^"]+)": ("([^"]+)"|[0-9]+(\.[0-9]+)?)$/) { # last item in record, line not ending in comma
		if ($ENV{opt_f} ne "") {
			# ServerState == 5: results that have already been reported back to the server
			# ValidateState == 0: pending validation
			# ValidateState == 4: pending verification
			if ($ServerState == 5 && $ValidateState != 0 && $ValidateState != 4 || $ENV{opt_w} > 0) {
				if (exists $workunit{$Name}) {
					if ($workunit{$Name} != $ValidateState) {
						print STDERR "$Name has its ValidateState changed from $workunit{$Name} to $ValidateState\n"; # Hopefully changed from ”Too Late” into something better. (Or probably opt_w>0)
					} elsif ($grantedc{$Name} != $GrantedCredit) {
						print STDERR "$Name has its GrantedCredit changed from $grantedc{$Name} to $GrantedCredit\n"; # Sometimes an ”Invalid” result (with credit ”0.0”) was granted some credit after all.
					} else {
						next; # already recorded in LOGFILE; the names of workunits vary per device, so no need to use $DeviceId or $WorkunitId to distinguish between them
					}
				}
				do {
					next if $ClaimedCredit + $CpuTime + $ElapsedTime + $GrantedCredit == 0.0; # probably: 192 <= $ExitStatus <= 204
				} if $ENV{opt_a} + $ENV{opt_w} == 0; # opt_a=1: record all WUs, also those that got 0 points.
				$ReceivedTime = "" if $ENV{opt_w} > 0;
				printf LOGFILE "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", $AppName, $ClaimedCredit, $CpuTime, $ElapsedTime, $ExitStatus, $GrantedCredit, $DeviceId, $DeviceName, $ModTime, $WorkunitId, $ResultId, $Name, $Outcome, $ReceivedTime, $ReportDeadline, $SentTime, $ServerState, $ValidateState, $FileDeleteState;
				$workunit{$Name} = $ValidateState; # for feeding from file ...
				$grantedc{$Name} = $GrantedCredit; # ... to avoid duplicates (when feeding a whole bunch of files at the same time)
			}
		} else {
			if ($Outcome != 1) {
				if ($Outcome == 0) {
					$ValidateState = 3; # In progress
					$ExitStatus = "-";
				}
				$fmt0 = "%-16s %-6s %-9s %-20s %-6s";
				$fmt1 = "%s    %s";
				$fmt2 = "%6.1f/%-6.1f %7.2f/%-7.2f ";
				$fmt3 = "   %s\n";
				printf "$fmt0 $fmt1$fmt3", "(DeviceName)", "(App)", "(Outcome)", "(ValidateState)", "(Exit)", "(Claim/Grant)", "(Cpu/Elapsed)", "(Name)" if ++$outcome_varies == 1;
				if ($ExitStatus ne "-") {
					if ($ExitStatus >= 192 && $ExitStatus <= 204) {
						if (not exists $DidExitStatus[$ExitStatus]) {
							$DidExitStatus[$ExitStatus] = 1;
							printf "\tExitStatus $ExitStatus: $txt_ExitStatus[$ExitStatus]\n";
						}
					}
				}
				printf "$fmt0 $fmt2$fmt3", $DeviceName, $AppName, $txt_Outcome[$Outcome], $txt_ValidateState[$ValidateState], $ExitStatus, $ClaimedCredit, $GrantedCredit, $CpuTime, $ElapsedTime, $Name;
			}
			if ($ServerState == 5) {
				# results that have already been reported back to the server
				$combi_DeviceName_AppName{$DeviceName}{$AppName}++;
				$combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}{$ValidateState}++; # number of validated results
				$combi_DeviceName_AppName_ValidateState_ClaimedCredit{$DeviceName}{$AppName}{$ValidateState} += $ClaimedCredit;
				$combi_DeviceName_AppName_ValidateState_CpuTime{$DeviceName}{$AppName}{$ValidateState} += $CpuTime;
				$combi_DeviceName_AppName_ValidateState_ElapsedTime{$DeviceName}{$AppName}{$ValidateState} += $ElapsedTime;
				$combi_DeviceName_AppName_ValidateState_GrantedCredit{$DeviceName}{$AppName}{$ValidateState} += $GrantedCredit;
				$result_matches++ if $ENV{opt_s} eq "" || $ENV{opt_s} eq $ValidateState;
			}
		}
	}
	END {
		if ($result_matches > 0) {
			printf "%s %s %-6s %s %s\n", "=" x 16, " " x 4, " (Number)", " " x 20, "(Number)" unless $outcome_varies == 0;
			foreach $DeviceName (keys %combi_DeviceName_AppName) {
				#print "$DeviceName:\n";
				foreach $AppName (keys %{$combi_DeviceName_AppName{$DeviceName}}) {
					if ($ENV{opt_s} eq "") {
						'"$PRINT_LONG_HEADER"'
						foreach $ValidateState (keys %{$combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}}) {
							printf "\t\t\t\t";
							'"$PRINT_LONG_FORMAT"'
						}
					} else {
						foreach $ValidateState (keys %{$combi_DeviceName_AppName_ValidateState_count{$DeviceName}{$AppName}}) {
							if ($ENV{opt_s} eq $ValidateState) {
								printf "%-16s %-6s %4d\t", $DeviceName, $AppName, $combi_DeviceName_AppName{$DeviceName}{$AppName};
								'"$PRINT_LONG_FORMAT"'
							}
						}
					}
				}
			}
		}
	}
' "$@"
}

do_opt_q() {
	perl -wne '
		use Term::ANSIColor qw(:constants); # "REVERSE", "RESET"
		use POSIX; # strftime()
		chomp;
		@t = split(/\t/);
		# column 0: AppName
		# column 7: DeviceName
		$app{$t[7]}{$t[0]}++;
		$dev{$t[7]}++;
		END {
			if ($ENV{opt_d} == 0) {
				printf "-";
			} else {
				print strftime "%T", localtime; # HH:MM:SS
			}
			foreach $devkey (sort keys %dev) {
				printf " %s%s%s", REVERSE, $devkey, RESET;
				if ($ENV{opt_q} == 1) {
					printf " %d", $dev{$devkey};
				} else {
					foreach $appkey (sort keys %{$app{$devkey}}) {
						printf " %s:%d", $appkey, $app{$devkey}{$appkey};
					}
				}
			}
			printf "\n";
		}
	' $opt_f
	rm -f $opt_f
}

wcgget() {
	[ -n "$opt_f" ] && Offset="&Offset=$skipm"
	wget $QUIET -O $TMPFILE "$URLBASE/api/members/$MEMBER/results?code=$VERIFY&format=json&ServerState=$ServerState$ValidateState$SortBy$Offset&Limit=$opt_l" && [ -s $TMPFILE ] &&
	calculate_or_record $TMPFILE
	if [ -n "$opt_f" ]; then
		mv $TMPFILE $TMPFILE.$skipm
		[ -s $TMPFILE.$skipm ] || exit 1
	fi
}

wcgget_cumulative() {
	[ $opt_w = 0 ] &&
	SortBy="&SortBy=CpuTime" # CpuTime is rather unique for each WU in order to prevent fetching duplicate WU-entries from the database (resulting from overlap in two or more fetches) using other key selections such as SentTime.
				 # (https://www.worldcommunitygrid.org/help/viewTopic.do?shortName=api says "Default is SentTime.")
	wcgget
	let avail=`perl -wne 'print if s/\s+"ResultsAvailable": "([^"]+)",\s+/$1/' $TMPFILE.$skipm` #     "ResultsAvailable": "395",
	let toget=$avail-$opt_l
	[ $opt_q -gt 0 ] && rm -f $TMPFILE.$skipm
	while [ $toget -gt 0 ]; do
		let skipm=$skipm+$opt_l # let wcget() deal with the Offset
		wcgget
		let toget=$toget-$opt_l # decrease the number of available results
		[ $opt_q -gt 0 ] && rm -f $TMPFILE.$skipm
	done
}

if [ $opt_E$opt_O$opt_V != 000 ]; then
	opt_d=0
fi
if [ $opt_q -gt 0 ]; then
	opt_f=/tmp/$Prog.$$ # name of the LOGFILE; let do_opt_q() do the cleanup
fi
if [ $opt_d = 0 ]; then
	calculate_or_record "$@"
	if [ $opt_q -gt 0 ]; then
		do_opt_q
	fi
else
	let skipm=0 # offset
	if [ -n "$opt_f" -o $opt_q -gt 0 ]; then
		QUIET=-q
		wcgget_cumulative
		if [ $opt_q -gt 0 ]; then
			do_opt_q
		fi
	else
		QUIET=""
		wcgget
	fi
fi

----------------------------------------
[Edit 3 times, last edit by adriverhoef at Feb 1, 2017 7:15:02 PM]
[Feb 1, 2017 6:27:53 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 286
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Version 1.18 of my script 'wcgresults'

Posting a new version of my script called "wcgresults"

Thanks adriverhoef for producing that script - it has proved very reliable over the last 10 days.. using the -dq option and graphing the results...
http://www.sraellis.tk/frame-17-wcg_wip.html
----------------------------------------
Run Time Stats https://grassmere-productions.no-ip.biz/
----------------------------------------
[Edit 1 times, last edit by TonyEllis at Feb 13, 2017 7:05:40 PM]
[Feb 11, 2017 11:17:01 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 65   Pages: 7   [ Previous Page | 1 2 3 4 5 6 7 | Next Page ]
[ Jump to Last Post ]
Post new Thread