For over a decade now, there’s a growing awareness in the academic world that many research results are significantly less reliable than is often assumed, and this includes the results of some leading studies. This certainly applies to psychology, the field in which Peder Isager has a degree, but also to medical science, for example. “According to current estimates, the chance that you’ll arrive at the same findings when you replicate a specific study in these fields, is no more than roughly fifty percent,” Isager says.

“Nevertheless, it’s not uncommon within the field of psychology for hundreds of follow-up studies to build upon only a small number of groundbreaking studies with findings that could not be replicated at all afterwards.” He cites the concept of ‘ego depletion’ as an example, the intuitive idea that willpower draws upon a reserve of mental energy that we can temporarily run out of when we need too much of it. “It is said that willpower functions like a muscle that can become fatigued.”

This theory has supposedly been demonstrated, among other ways, in experiments with test persons who were asked to solve an unsolvable puzzle. One group of test persons had to resist a plate of freshly baked cookies while solving the puzzle. That group gave up trying to solve the puzzle much faster than the other control groups. “This study was replicated a couple of years ago, but this time around, the effect of ego depletion wasn’t visible at all. As a result, that research line collapsed like a house of cards.”

Negative results

How is it possible that follow-up studies didn’t demonstrate much sooner that this concept was built on quicksand? The most important reason, Isager says, is the large influence of what is known as ‘publication bias.’ “Journals are rarely willing to publish studies with negative results. Because of this, scientists are always looking for new, positive results.” When a study doesn’t demonstrate the expected effect, the results are often filed away in a drawer. That is why only one percent of all publications in psychology consists of the kind of replication studies you need for safeguarding the quality of the field, according to Isager, who pursued a PhD in the Human-Techology Interaction group at TU/e's department of Industrial Engineering and Innovation Sciences.

Incidentally, even when a study doesn’t result in a confirmation of the research hypothesis, you can often still distill a new, significant result from research data, he stresses. “That’s not necessarily fraud; science just happens to be messier than you would like.” That is why more and more people are calling for a preregistration of studies. Preregistration is the practice of sharing both the exact method and the hypotheses before an experiment is conducted. This should prevent scientists from misusing their own research data.

Replication value

In short: it is to be expected that the academic world has even more skeletons in its closet. And the sooner they are revealed the better, before  scientific literature becomes even more contaminated with insights that are in fact unproven. “The essential question is: what is the best place to start,” Isager says. “Because there isn’t enough time and money to replicate every study. That is why I searched for the best criteria for determining which studies are the most important candidates for replication. Can we select studies for replication based on an objective replication value?”

Generally speaking, such a replication value should be based on both the importance of a study and the uncertainty we have about the study results. “In psychology, the size of a study, meaning the number of individuals that participate, is an important contributor to uncertainty. The actual importance of psychological research is often more difficult to determine objectively.”

Life years

Economic and medical sciences do however have some very obvious objective criteria for determining the potential value of a replication study. Think of the number of healthy life years gained, or the amount of money than can be earned. In psychology, the importance of a study is more abstract generally speaking. “That is why we decided to use citation score to determine the importance of a study. It’s not a perfect indicator, but an important one when it comes to determining an article’s influence within the field.”

Even though Isager managed to lay a foundation for an objective replication value, the time has now come to test it in practice, he believes. “For that, you should compare our formula to a ‘gold standard,’ such as human lives or dollars, in situations where this is possible. But that proves to be difficult.” He admits that he would prefer to do it himself. “Even though this is not the kind of project for a single person.”

Gospel

For now, he teaches at Oslo New University College, a place where he found refuge since the outbreak of the pandemic over two years ago. “We had to stay inside anyway, so I decided that I would rather stay close to my girlfriend and my family in Norway. Since then, I’ve returned to Eindhoven a couple of times during a few months. Now that I’ve completed my thesis, I’m working fulltime in Oslo to set up a new master’s program.”

One thing he plans to do in that function is to bring the replication gospel into practice. “Replicating a classic study is highly instructive to students. And very exciting, because we know many findings do not replicate, so who can tell how the results will come out! I will definitely recommend it to my students wholeheartedly.”