![[Translate to English:]](/fileadmin/_processed_/9/5/csm_doorlichtteam_shutterstock_ce49ace574.jpg)
- Research
- 30/06/2020
Successful test with ‘red team’ for criticism on research plan
Criticism from colleagues on the set-up of scientific studies shouldn’t take place through peer review after articles are submitted for publication, but prior to a research project instead - when that set-up can still be revised. That is the idea behind a test designed by associate professor Daniël Lakens with international colleagues: they recruited a so-called ‘red team’ that got paid to shoot holes in a research protocol. The result: 22 applications, 107 proposed improvement points and 660 paid out dollars for charity.
The stable door is closed only after the horse has bolted. In a way, that’s what happens when scientific publications are peer reviewed: obviously, a study with a flawed set-up is all but a complete waste of time. Quite apart from the fact that in many instances no one actually closes the stable door at all, simply because the animal's disappearance goes unnoticed by the reviewers; because these are - unpaid - researchers with busy jobs. The commotion surrounding the coronavirus hasn’t improved things; several publications were retracted, including those published in esteemed journals, because afterwards it turned out that serious mistakes were made - in short, the review process is anything but watertight.
That situation is the reason why social psychologist Daniël Lakens, who has been committed to reliable science his entire career, decided to publish a column in Nature in which he argued for the deployment of a ‘red team’ when scientists set up a research project. This way, you organize your own criticism, he explains, by inviting outsiders to take a critical look at each phase of a research project, and perhaps even rewarding them financially for each error they detect. “That’s a very common practice in the field of software in particular,” he says,” where hackers get paid for exposing bugs or leaks in software.”
Lakens’ plea struck a chord with his colleagues, both within TU/e and outside the university, and it was soon decided to simply give it a try. “We decided to recruit five people from various backgrounds and to give them two hundred dollars each as payment. In addition to that, we would pay them an extra hundred dollars for each error they found, which they could spend on a charity of their choice.” The subject of their ‘red team pilot’ was an already completed study by Nicholas Coles, a postdoc at Harvard University with whom Lakens had worked before.
“Nicholas had investigated to what extent the effect that you feel happier when you smile depends on the smile itself, or on the fact that someone tells you that smiling makes you feel happier. A major journal had deemed that study not interesting enough because the effect as such had already been sufficiently proven; and so, we decided to use this research as a study case.” It would have been more logical in principle to set up a new research project, Lakens agrees. “But that would have been too costly and time consuming for a test like this; we just wanted to find out quickly whether we could even manage to form such a red team.”
They weren’t disappointed by the response to their call, Lakens says. “We got 22 applications, ranging from established scientists to a student from Brazil. We randomly selected five persons with various areas of expertise, such as statistics, theoretical background and reproducibility of the method. They spent an average of twelve hours on their analysis and presented 107 potential points for improvement, which we had evaluated by Ruben Arslan, a colleague who works in Berlin. He concluded that there were six critical errors and a few smaller ones, accounting for a total of 660 dollars for charity.”
One of the crucial improvement points was the fact that the test persons were instructed by people who were familiar with the purpose of the research project,” Lakens explains. “The study didn’t have a double-blind set-up, as we say, and that can influence the outcome. Nicholas decided to conduct a complementary study to deal with these points of criticism, based on the red team’s remarks.”
All in all, the experiment was as success, the social psychologist believes. “We could just as well have received zero responses, or the team’s remarks could have been useless to us. The participants were very positive and it seems that the extra reward for each detected error to motivate them isn’t even necessary. I’m convinced that these people invested more time than the average peer reviewer, also because they get paid for it.”
After this successful test, Lakens is considering whether he can collaborate with a journal and present the criticisms of red teams next to the official reports of peer reviewers. “When we do that for a larger number of publications, we will certainly be able to reach conclusions about the usefulness of red teams for science.”
Discussion