Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 20
Posts: 20   Pages: 2   [ Previous Page | 1 2 ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 3597 times and has 19 replies Next Thread
sptrog1
Master Cruncher
Joined: Dec 12, 2017
Post Count: 1592
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

The question remains::
WHY so little research
Is grid computing becoming less attractive to researchers doing humanitarian and medical research of the type that runs on the WCG? Are the questions to be answered not solvable by parallel processing in small chunks?
The projects on this site seem to be dwindling. Why are there no new ones? Why do the ones that were running seem to flame out before completion? Are the ongoing costs unsustainable? Is it difficult to program the work units? The work units seem to be sparse.
Are the startup costs beyond the reach of the prospective researchers?
Maybe the opinions of the Jurisca group on all of this would be the most cogent.
[Jul 24, 2025 6:46:47 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Unixchick
Veteran Cruncher
Joined: Apr 16, 2020
Post Count: 1293
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

I think they need to have a stable base before asking for new projects. You can read their status updates to see that they are moving to new hardware, and are updating the software to make it more stable, but also easier to add new projects. The updates are not as frequent as I would like I admit. I think the hardware updates are not on the time schedule estimated, but they don't have control of that.

I get that you wish it was faster, don't we all. We are all here because we believe in the power of distributed computing and we want to help the science.

If you look at the latest ARP update, It said that AI didn't work as well as what we are helping them do.

My guess is that they are continuing to work on the software and waiting on the hardware upgrade. I love how MCM and ARP are currently running full tilt. They are definitely having weekend issues, but they come back strong on Monday.

I'm guessing they are having to spend more time than they would like on hardware issues. The upgrade was due in early July, and still says that on the hardware upgrade page. It is already mid to late July and the hardware page (which is not Jurisica Lab's) hasn't been updated. Maybe Jurisica Lab doesn't have info to give us, yet.
[Jul 24, 2025 7:25:08 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Link64
Senior Cruncher
Joined: Feb 19, 2021
Post Count: 206
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

The projects on this site seem to be dwindling. Why are there no new ones?
Actually they are working on new project called "Mapping Arthritis Markers" and the first stage of beta testing has been completed a while ago. Like Unixchick I guess we need to wait for the migration to new servers before they continue with that as the current ones obviously can't even cope with the load generated by MCM and ARP, IIRC they have slowed down the release of new ARP WUs a lot, otherwise they were killing the servers instantly in the past and stil everything breaks down every few days.
----------------------------------------

[Jul 24, 2025 8:23:03 PM]   Link   Report threatening or abusive post: please login first  Go to top 
sptrog1
Master Cruncher
Joined: Dec 12, 2017
Post Count: 1592
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

Perhaps it is time to ask this question again:
Is the active research active? Can WCG infrastructure support it? Where are the problems?
[Nov 3, 2025 1:47:56 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TPCBF
Master Cruncher
USA
Joined: Jan 2, 2011
Post Count: 2173
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

Perhaps it is time to ask this question again:
Is the active research active? Can WCG infrastructure support it? Where are the problems?
Well, where have you been the last two months?
[Nov 3, 2025 5:12:13 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Garrulus glandarius
Advanced Cruncher
Romania
Joined: Apr 10, 2025
Post Count: 88
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

Perhaps it is time to ask this question again:
Is the active research active? Can WCG infrastructure support it? Where are the problems?
Well, where have you been the last two months?


I was also surprised at how many people don't check their rigs for weeks/months and even then they rush to ask a question which has been answered weeks/months ago dozens of times in multiple threads... Oh well, the important thing is that we keep crunching an the projects lives on.
----------------------------------------

[Nov 4, 2025 6:57:45 AM]   Link   Report threatening or abusive post: please login first  Go to top 
sptrog1
Master Cruncher
Joined: Dec 12, 2017
Post Count: 1592
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

I am sorry for incurring the dissappointment of the cogniscenti. I was hoping to have had a more positive response.
I therefore conclude from the response now received that the Active research projects listed are moribund since there has been no news posted of evidence of their interest or activity. I also conclude that there is no capability beyond the Mapping Cancer project at present other than what sems to be working now judging from the Jurisca status report on their website.
If there is a more positive outlook, a concise answer would be appreciated.
[Nov 4, 2025 8:42:28 AM]   Link   Report threatening or abusive post: please login first  Go to top 
MJH333
Senior Cruncher
England
Joined: Apr 3, 2021
Post Count: 300
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

If there is a more positive outlook, a concise answer would be appreciated.
sptrog1,
The latest update (for 28 October) on the Jurisica Lab website, Operational Status tab, includes the following:

  • Regarding uploads that span the downtime for migration, we will reconcile validation and credit for these workunits as soon as the production path for MCM1 described above is running. We should be able to use the new components to do that, after walking the filesystems where those uploads live, double-checking the list that need validation and crediting in the database, and pursuing a similar "reprocessing" path which worked well to re-attempt validation and crediting of the test MCM1 batches.
  • Then, we will begin testing beta30/MAM1, and ARP1 using the new system, which we expect to progress much faster now that we have ironed out the logic with MCM1.
I take this to mean that the new Mapping Arthritis Markers project will start properly soon, and that the Africa Rainfall Project will restart soon.
Of course, there is a question about how soon is "soon".
Cheers,
Mark
----------------------------------------
[Edit 1 times, last edit by MJH333 at Nov 4, 2025 9:30:10 AM]
[Nov 4, 2025 9:27:53 AM]   Link   Report threatening or abusive post: please login first  Go to top 
dylanht
World Community Grid Tech
Joined: Jul 1, 2021
Post Count: 35
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

"Soon" is the plan. For a more positive outlook, the MAM1 beta application is a platform for biomarker discovery that internally I dubbed "MDMG" for Mapping Disease Markers on GPUs. MAM1 is able to train and evaluate neural networks via the LibTorch backend, and I am a few compiler errors away from getting an NVIDIA GPU-enabled build working. LibTorch is not the only major dependency of interest for that application though - I forked and made serializeable a small set of numerical optimization techniques from the ensmallen library, so the search for gene signatures to train and evaluate with neural networks will be more directed with MAM1/MDMG. We plan to backport this application to MCM1 "soon". We also build Apache Arrow into the project to get access to Parquet as an IO format. That is exciting because while the input/output code path I have sent out in beta has been JSON-only so far IIRC, I am almost done with the Parquet IO path that will let us treat result files collectively as a database as soon as the results hit our servers using something like DuckLake. From there, we can query to adjust the next batch of workunits to perform more directed/promising searches adaptively in "real-time", per batch. The reason that works, is the new protobuf schema batch plans. When I need to change something about the next batch of workunits for MCM1, like rsc_fpops_est, I adjust the batch plan defaults or override them in the script that submits new batch plans to a Kafka topic that the app-specific createWork daemons are subscribed to. Wiring up a queue of batch plans that depend on how well signatures from previous batches did should let us hone in on promising areas of the search space more quickly, and get better signatures faster, and analyze results quickly compared to the current setup for MCM1.

A bit about ARP1, which has always been a bit of a sore spot for the grid's architecture because of the data dependency between workunit generations that touch in the grid over sub-saharan Africa that defines the problem space geographically. By mapping the subregions of the problem space to the same nodes on the backend - so instead of boinc fanout buckets we use the geographic long/lat of the workunit in sub-saharan Africa - we can now route from HAProxy to individual partitions that split up the work of input file creation, batch database updates, and upload handling, only cross-fetching to each partition the "halo" required to calculate the next frontier for a node's assigned "region"/partition. We are hopeful that this will be the template design for partitioned workunit management on the backend for problem spaces like ARP1 with significant data dependency in the compute DAG, opening up new classes of problems for the WCG to crunch beyond the tail-latency insensitive embarassingly parallel problems which are the most natural fit for BOINC (we're hoping to upgrade BOINC as well so we can start taking advantage of the features of the recent clients released this year). Add to that the new object storage we're going to have access to and the massively increased NFS capacity at Nibi, many issues with stability of old hardware and the old cloud environment gone, and we are hopeful that the new backend architecture and MAM1 application will be the start of a turnaround for WCG.

I also plan to exploit the new protobuf "schemas" and all the metrics we get from the Kafka state management and docker stats for BOINC daemon containers to provide new research partners with a dashboard to login and manage workunit creation, download, and possibly some pre-defined analytics, which will hopefully open up the grid for rapid turnaround and short-run projects in addition to the traditional "a lot to do over a couple of years" projects that are the best fit. There is a lot to figure out with that, but I think easing the onboarding process should be a priority once we make good on all the above and tackle the rest of the backlog.
[Nov 4, 2025 5:54:06 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Hans Sveen
Veteran Cruncher
Norge
Joined: Feb 18, 2008
Post Count: 981
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Constipation or Starvation

Hi!

Thank You dylanht for the update, made a link to it on Project Status thread!!

Hans S.
----------------------------------------
[Edit 1 times, last edit by Hans Sveen at Nov 4, 2025 6:26:41 PM]
[Nov 4, 2025 6:10:00 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 20   Pages: 2   [ Previous Page | 1 2 ]
[ Jump to Last Post ]
Post new Thread