| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Community
Forum: Caring and Sharing
Thread: AAAS: Machine learning 'causing science crisis' |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 6
|
|
| Author |
|
|
R.West
Cruncher Joined: Jan 17, 2018 Post Count: 11 Status: Offline Project Badges: 
|
This weekend I read the following article on the BBC website:
AAAS: Machine learning 'causing science crisis': https://www.bbc.com/news/science-environment-47267081 One of the quotes that got my attention was this one: “There is general recognition of a reproducibility crisis in science right now. I would venture to argue that a huge part of that does come from the use of machine learning techniques in science.” The “reproducibility crisis” in science refers to the alarming number of research results that are not repeated when another group of scientists tries the same experiment. It means that the initial results were wrong. One analysis suggested that up to 85% of all biomedical research carried out in the world is wasted effort. Now my question is if World Community Grid is aware of this risk and if already completed studies have ever shown this issue when the results were used as input in the actual labs for verification. |
||
|
|
sptrog1
Master Cruncher Joined: Dec 12, 2017 Post Count: 1557 Status: Offline Project Badges: 
|
I wonder also.
It would seem that the present projects are "screening". That is eliminating unpromising possibilities. As such, the search for reproducable results is left to the future. The feedback from the scientific community has been sparse and justifiably long term as there should be a lot of work to do in the follow up. However the climate studies coming up seem to be more definitive and the results may be indeed consigned to the "Journal of Irreproducable Results". (Maybe there should be such a publication). However, I believe the results will be valid enough. Did you know that there if a difference in weather prediction between the European model and the American model given the same data. In the past,"Computing for sustainable water" seems to have been a wash. "Computing for clean water" produced some suprising results. And the results of "African climate at home" seem to be undiscoverable. |
||
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7545 Status: Offline Project Badges: 
|
“There is general recognition of a reproducibility crisis in science right now. I would venture to argue that a huge part of that does come from the use of machine learning techniques in science.” The “reproducibility crisis” in science refers to the alarming number of research results that are not repeated when another group of scientists tries the same experiment. It means that the initial results were wrong. One analysis suggested that up to 85% of all biomedical research carried out in the world is wasted effort. (Emphasis added) It does not necessarily mean the initial results were wrong. It may mean the followup results are wrong. Or it may mean all of the variables in the followup experiment were not identical to the original experiment. It may mean the initial conditions or inputs varied between the two experiments. The equipment used to monitor the experiments may have calibrated differently. In short, what I am postulating is the experiments were not identical. If the results are close (a relative term) in similarity then that may be good enough to draw the same conclusion. Another way to look at the same issue, for example, is to do the experiment 100 times. If 98 of the experiments are within the test parameters, but 2 are not, it is possible the two bad ones occurred second and third in the sequence of experiments, which would not make the first experiment invalid. I am not a research scientist, but I do know some who worked in labs who have encountered this issue and have strived to find the cause, but there are many variables to consider. Cheers
Sgt. Joe
*Minnesota Crunchers* |
||
|
|
pvh513
Senior Cruncher Joined: Feb 26, 2011 Post Count: 260 Status: Offline Project Badges:
|
In very crude terms, machine learning is about finding patterns in a large data set without any predetermined conceptions of what those patterns should be. It should be up to the AI to figure out what those patterns are. The theory is that this approach is bias-free. The practice turns out to be quite different (e.g. google "racial bias in AI"). Another problem is that machine learning essentially creates a black box that nobody understands (not even the people who programmed it). The BBC article may highlight a similar problem in this approach. My opinion is that machine learning is an immature technique that still has a lot to learn from its failings.
I do not think machine learning is what WCG does. They have a more classic, deterministic modeling approach. Something like: take molecule X and predict some characteristic of that molecule, then repeat the procedure for molecule Y, Z, etc. That approach is not fool-proof either of course (as sptrog1 pointed out), but should avoid the typical problems of machine learning. Regarding Sgt. Joe's remarks. He is completely right that the statement "it means that the initial results were wrong" is incorrect. If you cannot reproduce results, either the first or the second experiment is wrong, or both are wrong. But his definition of reproducibility is a bit too strict. In very broad terms, science is about finding the laws of nature, and the assumption is that these laws are invariable. From this comes the requirement of reproducibility. The laws of nature should not have changed between experiment 1 and 2, and hence you should find the same laws. So even if you have two completely different data sets, you should still be able to find the same laws. But this of coarse glosses over a lot of technical details... |
||
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7545 Status: Offline Project Badges:
|
Here is an interesting article about statistical analysis which relates to the significance of results and reproducibility. There are also links to other related articles on the page.
----------------------------------------Cheers
Sgt. Joe
*Minnesota Crunchers* |
||
|
|
Jack007
Master Cruncher CANADA Joined: Feb 25, 2005 Post Count: 1604 Status: Offline Project Badges: 
|
why is this in caring and sharing?
---------------------------------------- |
||
|
|
|