Guest Essay by Kip Hansen — 18 October 2022
Every time someone in our community, the science skeptic or Realists® community, speaks out about uncertainty and how it affects peer-reviewed scientific results, they are immediately accused to being Science Deniers or of trying to undermine the entire field of Science.
I have written again and again here about how the results of the majority of studies in climate science vastly underestimate the uncertainty of their results. Let me state this as clearly as possible: Any finding that does not honestly include a frank discussion of the uncertainties involved in the study, beginning with the uncertainties of the raw data and then all the way through the uncertainties added by each step of data processing, is not worth the digital ink used to publish it.
A new major multiple-research-group study, accepted and forthcoming in the Proceedings of the National Academy of Sciences, is set to shake up the research world. This paper, for once, is not written by John P.A. Ioannidis, of “Why Most Published Research Findings Are False” fame.
The paper is: “Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Idiosyncratic Uncertainty”. [ or as .pdf here ].
This is good science. This is how science should be done. And this is how science should be published.
First, who wrote this paper?
Nate Breznau et many many al. Breznau is at the University of Bremen. For co-authors, there is a list of 165 co-authors from 94 different academic institutions. The significance of this is that this is not the work of a single person or a single disgruntled research group.
What did they do?
The research question is this: “Will different researchers converge on similar findings when analyzing the same data?”
They did this:
“Seventy-three independent research teams used identical cross-country survey data to test an established social science hypothesis: that more immigration will reduce public support for government provision of social policies.”
What did they find?
“Instead of convergence, teams’ numerical results varied greatly, ranging from large negative to large positive effects of immigration on public support.”
Another way to look at this is to look at the actual numerical results produced by the various groups, asking the same question, using identical data:
The discussion section starts with the following:
“Discussion: Results from our controlled research design in a large-scale crowdsourced research effort involving 73 teams demonstrate that analyzing the same hypothesis with the same data can lead to substantial differences in statistical estimates and substantive conclusions. In fact, no two teams arrived at the same set of numerical results or took the same major decisions during data analysis.”
Want to know more?
If you really want to know why researchers who are asking the same question using the same data arrive at wildly different, and conflicting, answers you will really have to read the paper.
How does this relate to The Many-Analysts Approach?
Last June, I wrote about an approach to scientific questions named The Many-Analysts Approach.
The Many-Analysts Approach was touted as:
“We argue that the current mode of scientific publication — which settles for a single analysis — entrenches ‘model myopia’, a limited consideration of statistical assumptions. That leads to overconfidence and poor predictions. …. To gauge the robustness of their conclusions, researchers should subject the data to multiple analyses; ideally, these would be carried out by one or more independent teams.“
This new paper, being discussed today, has this to say:
“Even highly skilled scientists motivated to come to accurate results varied tremendously in what they found when provided with the same data and hypothesis to test. The standard presentation and consumption of scientific results did not disclose the totality of research decisions in the research process. Our conclusion is that we have tapped into a hidden universe of idiosyncratic researcher variability.”
And, that means, for you and I, that neither the many-analysts approach or the many-analysis-teams approach will not solve the Real World™ problem that is presented by the inherent uncertainties of the modern scientific research process – “many-analysts/teams” will use slightly differing approaches, different statistical techniques and slightly different versions of the available data. The teams make hundreds of tiny assumptions, mostly considering each as “best practices”. And because of these tiny differences, each team arrives at a perfectly defensible results, sure to pass peer-review, but each team arrives at different, even conflicting, answers to the same question asked of the same data.
This is the exact problem we see in CliSci every day. We see this problem in Covid stats, nutritional science, epidemiology of all types and many other fields. This is a separate problem from the differing biases affecting politically- and ideologically-sensitive subjects, the pressures in academia to find results in line with current consensuses in one’s field and the creeping disease of pal-review.
In Climate Science, we see the mis-guided belief that more processing – averaging, anomalies, krigging, smoothing, etc. — reduces uncertainty. The opposite is true: more processing increases uncertainties. Climate science does not even acknowledge the simplest type of uncertainty – original measurement uncertainty – but rather wishes it away.
Another approach sure to be suggested is that the results of the divergent findings should now be subjected to averaging or finding the mean — a sort of consensus — of the multitude of findings. The image of results shows this approach as the circle with 57.7% of the weighted distribution. This idea is no more valid than the averaging of chaotic model results as is done in Climate Science — in other words, worthless.
Pielke Jr. suggests in a recent presentation and follow-up Q&A with the National Association of Scholars that getting the best real experts together in a room and hashing these controversies our is probably the best approach. Pielke Jr. is an acknowledged fan of the approach used by the IPCC – but only long as their findings are untouched by politicians. Despite that, I tend to agree that getting the best and most honest (no-dog-in-this-fight) scientists in a field, along with specialists in statistics and evaluation of programmatic mathematics, all in one virtual room with orders to review and hash out the biggest differences in findings might produce improved results.
Don’t Ask Me
I am not an active researcher. I don’t have an off-the-cuff solution to the “ Three C’s” — the fact that the world is 1) Complicated, 2) Complex, and 3) Chaotic. Those three add to one another to create the uncertainty that is native to every problem. This new study adds in another layer – the uncertainty caused by the multitude of tiny decisions made by researchers when analyzing a research question.
It appears that the hope that the many-analysts/many-analysis-teams approaches would help resolve some of the tricky scientific questions of the day has been dashed. It also appears that it may be that when research teams that claim to be independent arrive at answers that have the appearance of too-close-agreement – we ought to be suspicious, not re-assured.
# # # # #
Author’s Comment:
If you are interested in why scientists don’t agree, even on simple questions, then you absolutely must read this paper, right now. Pre-print .pdf is here.
If it doesn’t change your understanding of the difficulties of doing good honest science, you probably need a brain transplant. …. Or at least a new advanced critical thinking skills course.
As always, don’t take my word for any of this. Read the paper, and maybe go back and read my earlier piece on Many Analysts.
Good science isn’t easy. And as we ask harder and harder questions, it is not going to get any easier.
The easiest thing in the world is to make up new hypotheses that seem reasonable or to make pie-in-the-sky predictions for futures far beyond our own lifetimes. Popular Science magazine made a business-plan of that sort of thing. Today’s “theoretical physics” seems to make a game of it – who can come up with the craziest-yet-believable idea about “how things really are”.
Thanks for reading.
# # # # #