Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 7
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 2662 times and has 6 replies Next Thread
thunder7
Senior Cruncher
Netherlands
Joined: Mar 6, 2013
Post Count: 241
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
monitoring progress on linux?

Is there any script or tool to answer questions like

how much points did I gain in projects X, Y and Z yesterday?

did all installations work as expected yesterday?

Did the average amount of time / results per day for a certain project move or not?

from linux?

Wget or curl need to get past the password barrier - anybody solved that?
[Mar 31, 2017 11:22:10 AM]   Link   Report threatening or abusive post: please login first  Go to top 
thunder7
Senior Cruncher
Netherlands
Joined: Mar 6, 2013
Post Count: 241
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: monitoring progress on linux?

For those who wish to do something like this, this is how I got started:

wget --quiet --save-cookies /tmp/boinc_cookies.txt --keep-session-cookies --post-data 'j_username=thunder7&j_password=wouldntyouliketoknow' https://www.worldcommunitygrid.org/j_security_check -O /tmp/boinc_login.html

wget --quiet --load-cookies /tmp/boinc_cookies.txt https://secure.worldcommunitygrid.org/ms/viewMyMemberPage.do -O contribution.html

wget --quiet --load-cookies /tmp/boinc_cookies.txt https://secure.worldcommunitygrid.org/ms/device/viewStatisticsByDevice.do -O device_statistics.html
[Apr 1, 2017 5:17:16 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2356
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: monitoring progress on linux?

I just wrote a script to download statistics from your "My Contribution" page.
Many thanks to World Community Grid Tech @knreed for help on this matter (see this posting).

This program will read statistics from your "My Contribution" page and output one line containing something like:
2017-04-01 00:06:03	7:274:19:28:31	19,919	17,435,920	8,393	61,289	3,629	10:039:15:59:27	8,314	22,758,229	4,685	70,792	2,906
(Date+Time, your Total Run Time+Rank, your Points+Rank, your Results Returned+Rank
and for your Team also Total Run Time+Rank, Points+Rank and Results Returned+Rank).

If you specify the option -x, the program will output one line with the current statistics for all your WCG-projects along the lines of:
2017-04-01 12:06:02  scc1=0:032:18:25:57=45,445=137  zika=1:188:13:50:25=3,572,290=9,090  hst1=0:059:01:44:44=386,082=211 …
(line shortened to fit pagewidth)

If you want to save the output to a file on a daily basis, then add one line to your crontab, looking something like this:
27 8 * * * bin/wcgstats >> WCG.stats # Insert the directory where the file WCG.stats will be located.

Save the code below to a file called 'wcgstats', chmod +x it and move it to your ~/bin directory. Then change the values of USERNAME=... and PASSWORD=... when/while editing the file to reflect the credentials of your account.
--- CUT HERE --- snip 8< --- snip 8< --- snip 8< --- snip 8< --- snip 8< --- snip 8<
#!/bin/sh
#
# wcgstats - get recent statistics from WCG's website
#
# VERSION: 1.0.2
# Author: Adri Verhoef

Prog=${0##*/}

#############
# CONFIGURE #
#############
TMPDIR=/tmp # Directory for storing temporary files
URLBASE=www.worldcommunitygrid.org # Don't change this, unless you know what you're doing
HTTPS="https:"// # Don't change this, unless you know what you're doing
USERNAME="youraccountname" # The name of your account with which you're crunching
PASSWORD="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" # The password of your account to read My Contribution
USERNAME1="anotheraccount" # the name of the account for user 1
PASSWORD1="????????????????????????????????" # password of the account for user 1
########################
# END OF CONFIGURATION #
########################

usage () {
printf "%s\n" \
"Usage: $Prog [options]" \
" Options:" \
" -D dir download files to directory ’dir’" \
" -u user the name of your account on World Community Grid (default: you; ’user’ = 1 ... for other account(s))" \
" -x show extended statistics (per project) instead of basic statistics; add another ’x’ to also show basic statistics" \
;
}

# Options
OPTS=hD:u:x
export opt_D=$TMPDIR
export opt_u=""
export opt_x=0
while getopts $OPTS opt; do
case $opt in
h) usage; exit 0;;
[x]) eval let opt_$opt++;;
[a-zA-Z0-9]) eval opt_$opt='$OPTARG';;
?) usage; exit 1;;
esac
done
shift $(($OPTIND - 1))

case $opt_u in "") ;; *) eval USERNAME=\$USERNAME$opt_u; eval PASSWORD=\$PASSWORD$opt_u;; esac

POST_DATA="j_username=$USERNAME&j_password=$PASSWORD"

TMPDIR=$opt_D
TIMESTAMP=`date +%Y-%m-%dT%H:%M:%S`
COOKIES=$opt_D/$Prog.$$.cookies.$TIMESTAMP
TMPFILE=$opt_D/$Prog.$$.webpage.$TIMESTAMP

rm -f $COOKIES

umask 0077
QUIET=-q
MY_CHECK=-
MY_STATS=$TMPFILE.html
wget $QUIET -O $MY_CHECK --save-cookies $COOKIES --keep-session-cookies --post-data "$POST_DATA" $HTTPS$URLBASE/j_security_check >/dev/null
wget $QUIET -O $MY_STATS --load-cookies $COOKIES $HTTPS$URLBASE/ms/viewMyMemberPage.do

trio='\t%s\t%s\t%s' # basic statistics (My Contribution + My Team)
xtra='\t%s=%s=%s=%s' # extended statistics (per project)
perl -wne '
BEGIN { $state = 0; $span_seq = 0; $this_or_that = 0; $proj_seq = 0; }
sub save_span() {
if ($span_seq == 0) { die "PROGRAM ERROR: \$span_seq == 0"; }
if ($span_seq == 1) {
($runtime[$this_or_that],$rt_rank[$this_or_that]) = $span =~ /([^ ]+) +\(#([^\)]+)\)/;
} elsif ($span_seq == 2) {
($gen_pts[$this_or_that],$gp_rank[$this_or_that]) = $span =~ /([^ ]+)[\t\n ]+\(#([^\)]+)\)/;
} elsif ($span_seq == 3) {
($results[$this_or_that],$rr_rank[$this_or_that]) = $span =~ /([^ ]+) +\(#([^\)]+)\)/;
} elsif ($span_seq == 4) {
($m,$d,$y,$t) = $span =~ /([0-9]+)\/([0-9]+)\/([0-9]+) +([012][0-9]:[0-6][0-9]:[0-6][0-9])/;
$updated = sprintf "%02d-%02d-%02d %s", 2000 + $y, $m, $d, $t;
}
do {
$span_seq = 0;
$this_or_that = 0 if ++$this_or_that == 2; # reset to select member (0) or team (1)
} if $span_seq < 10;
}
do { $state = 1; next; } if /<!-- START Member and Team Statistics Summaries -->/;
do { $state = 2; next; } if /<!-- START Member Stats by Projects -->/;
do { $state = 3; next; } if /<!-- END Member Stats by Projects -->/;
next if $state == 0;
next if $state == 3;
do { $span_seq = 1; next; } if /<span>Total Run Time \(y:d:h:m:s\) \(Rank\)<\/span>/;
do { $span_seq = 2; next; } if /'\''\/help\/viewTopic\.do\?shortName=points#177'\''/;
do { $span_seq = 3; next; } if /<span>Results Returned \(Rank\)<\/span>/;
do { $span_seq = 4; next; } if /<span>Statistics Last Updated:<\/span>/;
do { $span_seq = 5; $p_inshort[$proj_seq] = $1; $trio = 0; next; } if /"\/stat\/viewProject\.do\?projectShortName=([^"]+)"/;
next if $span_seq == 0; # nothing to find here
if (/<span>([^<]+)<\/span>/) {
$span = $1;
if ($state == 2) {
if ($trio == 0) { $p_gen_pts[$proj_seq] = $span; }
elsif ($trio == 1) { $p_results[$proj_seq] = $span; }
elsif ($trio == 2) { $p_runtime[$proj_seq] = $span; $proj_seq++; }
else { $span_seq = 0; }
$trio++;
} else {
save_span();
}
next;
} elsif (/<span>([^<]+)/) {
$span = $1;
next;
} elsif (/([^<]+)<\/span>/) {
$span .= $1;
save_span();
next;
}
END {
die "Statistics unavailable.\n" if $state == 0;
printf "%s", $updated;
printf "'"$trio$trio$trio$trio"'",
$runtime[0], $rt_rank[0], $gen_pts[0], $gp_rank[0], $results[0], $rr_rank[0],
$runtime[1], $rt_rank[1], $gen_pts[1], $gp_rank[1], $results[1], $rr_rank[1] if $ENV{opt_x} != 1;
do {
for ($n = 0; $n < $proj_seq; $n++) {
printf "'"$xtra"'", $p_inshort[$n], $p_runtime[$n], $p_gen_pts[$n], $p_results[$n];
}
} if $ENV{opt_x} > 0; # show extended statistics (for all projects)
printf "\n";
}
' $MY_STATS
rm -f $COOKIES $MY_STATS

exit 0


To answer questions like "how much points did I gain in projects X, Y and Z yesterday?" and "Did the average amount of time / results per day for a certain project move or not?" you could use the logfile, "WCG.stats"; the answer to crossing the password barrier is also answered in this script. The question "did all installations work as expected yesterday?" could also be answered by the script 'wcgresults' that can be used to record results that passed the validator.
----------------------------------------
[Edit 5 times, last edit by adriverhoef at Apr 7, 2017 5:58:54 PM]
[Apr 1, 2017 6:11:44 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2356
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: monitoring progress on linux?

Here is an updated version of the script that will show the main statistics from your My Contribution page.

What has changed?
- Option -c was introduced to remove the comma from numbers > 999 (if needed).
- If you haven't joined a team, zeroes will be shown instead of error messages.


This program will read statistics from your "My Contribution" page and output one line containing something like:
2017-04-13 12:06:02  0:103:07:20:26  354,018  64,015  544,464  127  586,042  0:207:14:41:52  177,037  128,031  272,928  255  293,084
(Date+Time, your Total Run Time+Rank, your Points+Rank, your Results Returned+Rank
and for your Team also Total Run Time+Rank, Points+Rank and Results Returned+Rank).

If you specify option -x, the program will output one line with the current statistics for all your WCG-projects along the lines of:
2017-04-01 12:06:02  scc1=0:032:18:25:57=45,445=137  zika=1:188:13:50:25=3,572,290=9,090  hst1=0:059:01:44:44=386,082=211 …
(line shortened to fit pagewidth)

If you want to save the output of the program to a file on a daily basis, you could add one line to your crontab, looking something like this:
27 8 * * * bin/wcgstats >> WCG.stats # You might want to insert the directory where the file WCG.stats will be located.

Save the code below to a file called 'wcgstats', chmod +x it and move it to your ~/bin directory. Then change the values of USERNAME=... and PASSWORD=... when/while editing the file to reflect the credentials of your account.
--- CUT HERE --- snip 8< --- snip 8< --- snip 8< --- snip 8< --- snip 8< --- snip 8<
#!/bin/sh
#
# wcgstats - get recent statistics from WCG's website
#
# VERSION: 1.2
# Author: Adri Verhoef

Prog=${0##*/}

#############
# CONFIGURE #
#############
TMPDIR=/tmp # Directory for storing temporary files
URLBASE=www.worldcommunitygrid.org # Don't change this, unless you know what you're doing
HTTPS="https:"// # Don't change this, unless you know what you're doing
USERNAME="youraccountname" # The name of your account with which you're crunching
PASSWORD="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" # The password of your account to read My Contribution
USERNAME1="anotheraccount" # the name of the account for user 1
PASSWORD1="????????????????????????????????" # password of the account for user 1
########################
# END OF CONFIGURATION #
########################

usage () {
printf "%s\n" \
"Usage: $Prog [options]" \
" Options:" \
" -c omit ',' in numbers (e.g. for compatibility reasons)" \
" -D dir download intermediate files to directory ’dir’" \
" -u user the name of your account on World Community Grid (default: you; ’user’ = 1 ... for other account(s))" \
" -x show extended statistics (per project) instead of basic statistics; add another ’x’ to also show basic statistics" \
;
}

# Options
OPTS=hcD:u:x
export opt_c=0
export opt_D=$TMPDIR
export opt_u=""
export opt_x=0
while getopts $OPTS opt; do
case $opt in
h) usage; exit 0;;
[cx]) eval let opt_$opt++;;
[a-zA-Z0-9]) eval opt_$opt='$OPTARG';;
?) usage; exit 1;;
esac
done
shift $(($OPTIND - 1))

case $opt_u in "") ;; *) eval USERNAME=\$USERNAME$opt_u; eval PASSWORD=\$PASSWORD$opt_u;; esac

POST_DATA="j_username=$USERNAME&j_password=$PASSWORD"

TMPDIR=$opt_D
TIMESTAMP=`date +%Y-%m-%dT%H:%M:%S`
COOKIES=$opt_D/$Prog.$$.cookies.$TIMESTAMP
TMPFILE=$opt_D/$Prog.$$.webpage.$TIMESTAMP

umask 0077
QUIET=-q
MY_CHECK=-
MY_STATS=$TMPFILE.html

rm -f $COOKIES $MY_STATS
wget $QUIET -O $MY_CHECK --save-cookies $COOKIES --keep-session-cookies --post-data "$POST_DATA" $HTTPS$URLBASE/j_security_check >/dev/null
wget $QUIET -O $MY_STATS --load-cookies $COOKIES $HTTPS$URLBASE/ms/viewMyMemberPage.do

trio='\t%s\t%s\t%s' # basic statistics (My Contribution + My Team)
xtra='\t%s=%s=%s=%s' # extended statistics (per project)
perl -wne '
BEGIN {
$state = 0;
$span_seq = 0;
$proj_seq = 0;
for ($n = 1; $n <= 3; $n++) { $seen[$n] = 0; }
for ($n = 0; $n <= 1; $n++) {
$runtime[$n] = 0;
$rt_rank[$n] = 0;
$gen_pts[$n] = 0;
$gp_rank[$n] = 0;
$results[$n] = 0;
$rr_rank[$n] = 0;
}
}
sub save_span() {
if ($span_seq == 0) { die "PROGRAM ERROR: \$span_seq == 0"; }
if ($span_seq == 1) {
($runtime[$seen[$span_seq]],$rt_rank[$seen[$span_seq]]) = $span =~ /([^ ]+) +\(#([^\)]+)\)/;
} elsif ($span_seq == 2) {
($gen_pts[$seen[$span_seq]],$gp_rank[$seen[$span_seq]]) = $span =~ /([^ ]+)[\t\n ]+\(#([^\)]+)\)/;
} elsif ($span_seq == 3) {
($results[$seen[$span_seq]],$rr_rank[$seen[$span_seq]]) = $span =~ /([^ ]+) +\(#([^\)]+)\)/;
} elsif ($span_seq == 4) {
($m,$d,$y,$t) = $span =~ /([0-9]+)\/([0-9]+)\/([0-9]+) +([012][0-9]:[0-6][0-9]:[0-6][0-9])/;
$updated = sprintf "%02d-%02d-%02d %s", 2000 + $y, $m, $d, $t;
}
$seen[$span_seq]++;
$span_seq = 0;
}
do { $state = 1; next; } if /<!-- START Member and Team Statistics Summaries -->/;
do { $state = 2; next; } if /<!-- START Member Stats by Projects -->/;
do { $state = 3; next; } if /<!-- END Member Stats by Projects -->/;
next if $state == 0;
next if $state == 3;
do { $span_seq = 1; next; } if /<span>Total Run Time \(y:d:h:m:s\) \(Rank\)<\/span>/;
do { $span_seq = 2; next; } if /'\''\/help\/viewTopic\.do\?shortName=points#177'\''/;
do { $span_seq = 3; next; } if /<span>Results Returned \(Rank\)<\/span>/;
do { $span_seq = 4; next; } if /<span>Statistics Last Updated:<\/span>/;
do { $span_seq = 5; $p_inshort[$proj_seq] = $1; $trio = 0; next; } if /"\/stat\/viewProject\.do\?projectShortName=([^"]+)"/;
next if $span_seq == 0; # nothing to find here
if (/<span>([^<]+)<\/span>/) {
$span = $1;
if ($state == 2) {
if ($trio == 0) { $p_gen_pts[$proj_seq] = $span; }
elsif ($trio == 1) { $p_results[$proj_seq] = $span; }
elsif ($trio == 2) { $p_runtime[$proj_seq] = $span; $proj_seq++; }
else { $span_seq = 0; }
$trio++;
} else {
save_span();
}
next;
} elsif (/<span>([^<]+)/) {
$span = $1;
next;
} elsif (/([^<]+)<\/span>/) {
$span .= $1;
save_span();
next;
}
END {
die "Statistics unavailable.\n" if $state == 0;
do {
for ($n = 0; $n < 2; $n++) {
$rt_rank[$n] =~ s/,//g; $gen_pts[$n] =~ s/,//g; $gp_rank[$n] =~ s/,//g; $results[$n] =~ s/,//g; $rr_rank[$n] =~ s/,//g; # v1.1
}
} if $ENV{opt_c} > 0;
printf "%s", $updated;
printf "'"$trio$trio$trio$trio"'",
$runtime[0], $rt_rank[0], $gen_pts[0], $gp_rank[0], $results[0], $rr_rank[0],
$runtime[1], $rt_rank[1], $gen_pts[1], $gp_rank[1], $results[1], $rr_rank[1] if $ENV{opt_x} != 1;
do {
for ($n = 0; $n < $proj_seq; $n++) {
do {
$p_gen_pts[$n] =~ s/,//g; $p_results[$n] =~ s/,//g; # v1.1
} if $ENV{opt_c} > 0;
printf "'"$xtra"'", $p_inshort[$n], $p_runtime[$n], $p_gen_pts[$n], $p_results[$n];
}
} if $ENV{opt_x} > 0; # show extended statistics (for all projects)
printf "\n";
}
' "$MY_STATS"
rm -f "$COOKIES" "$MY_STATS"

exit 0

[Apr 13, 2017 9:10:31 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 291
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: monitoring progress on linux?

Thanks adriverhoef for that script...

by using v1.2 and this command "wcgstats.sh -x -c | tr '\t' '\n' | tr '=' ' ' > /tmp/xxxxx" a nice file like this is created...

2017-04-14 00:06:02
scc1 2:004:13:54:30 2887784 6897
zika 12:090:01:58:13 11982781 31046
hst1 3:092:06:25:15 4767435 2412
fahb 10:006:11:50:05 14391101 5716
oet1 10:005:08:53:47 9784941 41464
mcm1 10:153:06:14:34 12705295 17505
faah 10:009:00:58:51 10420645 24432
beta 0:333:06:25:51 615216 1013
ugm1 10:006:08:06:03 8312095 12864
cfsw 2:022:01:31:03 1575607 9833
sn2s 7:243:21:25:05 8806702 12064
gfam 9:101:21:29:23 11946397 15011
dsfl 3:022:03:05:44 2897076 3610
c4cw 2:004:00:45:26 1120436 2537
cep2 5:018:08:30:30 5570888 5599
dddt2 0:216:16:24:16 383669 988
hcmd2 2:003:18:51:00 1545925 6437
flu1 0:001:01:54:08 1496 2
hfcc 2:013:23:57:16 1342277 1748
cep1 0:001:11:38:09 2298 1
rice 0:178:19:19:30 371927 580
hcc1 3:081:03:08:29 1075789 5661
hpf2 2:197:14:28:13 1713591 1977

This makes it easy to extract the project names and fields required to be graphed using mrtg and added to my web-site
The end result here... http://www.sraellis.tk/frame-17-wcg_progress.html
----------------------------------------
[Apr 14, 2017 8:13:41 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2356
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: monitoring progress on linux?

It's very nice to hear that you like it, Tony.

And if you want to make use of only one 'tr' command, you could even write it like this:
wcgstats -x -c | tr '\t=' '\n '
nerd (Japanese wizkid?)

By using mrtg everyone can see something's growing on your machine wink ... Awesome! (I've never used mrtg)
[Apr 14, 2017 8:41:15 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 291
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: monitoring progress on linux?

Thanks adriverhoef - have applied your 'tr' hint. Here in Australia the daily statistics updates happens around midday - that just happened to be when your post was seen and I copied and pasted your script - took me a few seconds to realize why it was failing smile - needed to be patient for the WCG batch job to finish before being able to progress with this little project sad
----------------------------------------
----------------------------------------
[Edit 1 times, last edit by TonyEllis at Apr 14, 2017 9:37:23 AM]
[Apr 14, 2017 9:36:12 AM]   Link   Report threatening or abusive post: please login first  Go to top 
[ Jump to Last Post ]
Post new Thread