| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
| World Community Grid Forums
|
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 20
|
|
| Author |
|
|
sptrog1
Master Cruncher Joined: Dec 12, 2017 Post Count: 1592 Status: Offline Project Badges:
|
The question remains::
WHY so little research Is grid computing becoming less attractive to researchers doing humanitarian and medical research of the type that runs on the WCG? Are the questions to be answered not solvable by parallel processing in small chunks? The projects on this site seem to be dwindling. Why are there no new ones? Why do the ones that were running seem to flame out before completion? Are the ongoing costs unsustainable? Is it difficult to program the work units? The work units seem to be sparse. Are the startup costs beyond the reach of the prospective researchers? Maybe the opinions of the Jurisca group on all of this would be the most cogent. |
||
|
|
Unixchick
Veteran Cruncher Joined: Apr 16, 2020 Post Count: 1293 Status: Offline Project Badges:
|
I think they need to have a stable base before asking for new projects. You can read their status updates to see that they are moving to new hardware, and are updating the software to make it more stable, but also easier to add new projects. The updates are not as frequent as I would like I admit. I think the hardware updates are not on the time schedule estimated, but they don't have control of that.
I get that you wish it was faster, don't we all. We are all here because we believe in the power of distributed computing and we want to help the science. If you look at the latest ARP update, It said that AI didn't work as well as what we are helping them do. My guess is that they are continuing to work on the software and waiting on the hardware upgrade. I love how MCM and ARP are currently running full tilt. They are definitely having weekend issues, but they come back strong on Monday. I'm guessing they are having to spend more time than they would like on hardware issues. The upgrade was due in early July, and still says that on the hardware upgrade page. It is already mid to late July and the hardware page (which is not Jurisica Lab's) hasn't been updated. Maybe Jurisica Lab doesn't have info to give us, yet. |
||
|
|
Link64
Senior Cruncher Joined: Feb 19, 2021 Post Count: 206 Status: Offline Project Badges:
|
The projects on this site seem to be dwindling. Why are there no new ones? Actually they are working on new project called "Mapping Arthritis Markers" and the first stage of beta testing has been completed a while ago. Like Unixchick I guess we need to wait for the migration to new servers before they continue with that as the current ones obviously can't even cope with the load generated by MCM and ARP, IIRC they have slowed down the release of new ARP WUs a lot, otherwise they were killing the servers instantly in the past and stil everything breaks down every few days.![]() |
||
|
|
sptrog1
Master Cruncher Joined: Dec 12, 2017 Post Count: 1592 Status: Offline Project Badges:
|
Perhaps it is time to ask this question again:
Is the active research active? Can WCG infrastructure support it? Where are the problems? |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 2173 Status: Offline Project Badges:
|
Perhaps it is time to ask this question again: Well, where have you been the last two months?Is the active research active? Can WCG infrastructure support it? Where are the problems? |
||
|
|
Garrulus glandarius
Advanced Cruncher Romania Joined: Apr 10, 2025 Post Count: 88 Status: Offline Project Badges:
|
Perhaps it is time to ask this question again: Well, where have you been the last two months?Is the active research active? Can WCG infrastructure support it? Where are the problems? I was also surprised at how many people don't check their rigs for weeks/months and even then they rush to ask a question which has been answered weeks/months ago dozens of times in multiple threads... Oh well, the important thing is that we keep crunching an the projects lives on. ![]() ![]() |
||
|
|
sptrog1
Master Cruncher Joined: Dec 12, 2017 Post Count: 1592 Status: Offline Project Badges:
|
I am sorry for incurring the dissappointment of the cogniscenti. I was hoping to have had a more positive response.
I therefore conclude from the response now received that the Active research projects listed are moribund since there has been no news posted of evidence of their interest or activity. I also conclude that there is no capability beyond the Mapping Cancer project at present other than what sems to be working now judging from the Jurisca status report on their website. If there is a more positive outlook, a concise answer would be appreciated. |
||
|
|
MJH333
Senior Cruncher England Joined: Apr 3, 2021 Post Count: 300 Status: Offline Project Badges:
|
If there is a more positive outlook, a concise answer would be appreciated. sptrog1,The latest update (for 28 October) on the Jurisica Lab website, Operational Status tab, includes the following:
Of course, there is a question about how soon is "soon". Cheers, Mark [Edit 1 times, last edit by MJH333 at Nov 4, 2025 9:30:10 AM] |
||
|
|
dylanht
World Community Grid Tech Joined: Jul 1, 2021 Post Count: 35 Status: Offline Project Badges:
|
"Soon" is the plan. For a more positive outlook, the MAM1 beta application is a platform for biomarker discovery that internally I dubbed "MDMG" for Mapping Disease Markers on GPUs. MAM1 is able to train and evaluate neural networks via the LibTorch backend, and I am a few compiler errors away from getting an NVIDIA GPU-enabled build working. LibTorch is not the only major dependency of interest for that application though - I forked and made serializeable a small set of numerical optimization techniques from the ensmallen library, so the search for gene signatures to train and evaluate with neural networks will be more directed with MAM1/MDMG. We plan to backport this application to MCM1 "soon". We also build Apache Arrow into the project to get access to Parquet as an IO format. That is exciting because while the input/output code path I have sent out in beta has been JSON-only so far IIRC, I am almost done with the Parquet IO path that will let us treat result files collectively as a database as soon as the results hit our servers using something like DuckLake. From there, we can query to adjust the next batch of workunits to perform more directed/promising searches adaptively in "real-time", per batch. The reason that works, is the new protobuf schema batch plans. When I need to change something about the next batch of workunits for MCM1, like rsc_fpops_est, I adjust the batch plan defaults or override them in the script that submits new batch plans to a Kafka topic that the app-specific createWork daemons are subscribed to. Wiring up a queue of batch plans that depend on how well signatures from previous batches did should let us hone in on promising areas of the search space more quickly, and get better signatures faster, and analyze results quickly compared to the current setup for MCM1.
A bit about ARP1, which has always been a bit of a sore spot for the grid's architecture because of the data dependency between workunit generations that touch in the grid over sub-saharan Africa that defines the problem space geographically. By mapping the subregions of the problem space to the same nodes on the backend - so instead of boinc fanout buckets we use the geographic long/lat of the workunit in sub-saharan Africa - we can now route from HAProxy to individual partitions that split up the work of input file creation, batch database updates, and upload handling, only cross-fetching to each partition the "halo" required to calculate the next frontier for a node's assigned "region"/partition. We are hopeful that this will be the template design for partitioned workunit management on the backend for problem spaces like ARP1 with significant data dependency in the compute DAG, opening up new classes of problems for the WCG to crunch beyond the tail-latency insensitive embarassingly parallel problems which are the most natural fit for BOINC (we're hoping to upgrade BOINC as well so we can start taking advantage of the features of the recent clients released this year). Add to that the new object storage we're going to have access to and the massively increased NFS capacity at Nibi, many issues with stability of old hardware and the old cloud environment gone, and we are hopeful that the new backend architecture and MAM1 application will be the start of a turnaround for WCG. I also plan to exploit the new protobuf "schemas" and all the metrics we get from the Kafka state management and docker stats for BOINC daemon containers to provide new research partners with a dashboard to login and manage workunit creation, download, and possibly some pre-defined analytics, which will hopefully open up the grid for rapid turnaround and short-run projects in addition to the traditional "a lot to do over a couple of years" projects that are the best fit. There is a lot to figure out with that, but I think easing the onboarding process should be a priority once we make good on all the above and tackle the rest of the backlog. |
||
|
|
Hans Sveen
Veteran Cruncher Norge Joined: Feb 18, 2008 Post Count: 981 Status: Offline Project Badges:
|
Hi!
----------------------------------------Thank You dylanht for the update, made a link to it on Project Status thread!! Hans S. [Edit 1 times, last edit by Hans Sveen at Nov 4, 2025 6:26:41 PM] |
||
|
|
|