The long-awaited survey by the RIVM, the Trimbos Institute and the GGD recently painted a worrying picture of the wellbeing of students. It emerges from the ‘Monitor of mental health and substance use among higher education students’ that at the time of the lockdown in the third wave of COVID-19 two out of three students felt emotionally drained. A quarter were sometimes tired of life and then wanted to go to sleep and not wake up. They also drank and smoked cannabis a lot.

The results shocked student organisations and higher education institutions. Education Minister Ingrid van Engelshoven used the report to stress that a new lockdown of higher education would pose huge risks for the mental health of students.

But are students really so depressed? Han van der Maas, Professor and Head of Psychological Methods at UvA, thinks not. “The survey that is scaring people so much is not valid”, he wrote in the Dutch national newspaper De Volkskrant. “Only 11.7 percent of the 250 thousand students approached completed the questionnaires. Students with mental issues are probably much more likely to respond.”

Weak spot

“In recent years I have seen a lot of surveys that, just like this one, have in many respects been conducted well”, says Van der Maas. “But they have one very weak spot and that is the way in which the sample was composed. People are invited via a website or by letter to complete a questionnaire, but the response rate is often so low that you don’t know what self-selection has taken place.”

In his view that applies to the RIVM report as well. “It most likely leads to a substantial distortion of the results. The roughly 90 percent of students that did not complete the 20-minute questionnaire would probably have responded very differently to questions about loneliness and being tired of life.”

He refers to international studies on happiness. “In 2019 the Netherlands was once again in the global top 5, with a score of 7.5. And of course we are now in the grip of COVID-19 and that doesn’t make life any more enjoyable. But the score of 6 that students gave in the RIVM survey – which uses the same survey method – is the same level as Mexico. People there are living in very different conditions from our students in the time of COVID-19.”

Cautious

Health psychologist Peter van der Velden, senior researcher at Centerdata and former Professor of Victimology at Tilburg University, shares Van der Maas’s critical view. “The high level of publicity surrounding the survey is not justified by its quality. It’s perfectly possible to get such a low response but the researchers then have to be very cautious with the results. Especially if you don’t take into account how things were going with those students before COVID-19. Then you don’t actually know whether the problems have increased or decreased.”

Van der Velden and his colleagues are studying wellbeing and COVID-19. “We’re doing that by using the LISS panel, which is based on a random sample of the Dutch population. The panel has more than 6,500 members. What we saw in December 2020 is that young people have more mental health issues than elderly people, but that was also the case in the year prior to COVID-19. So yes, young people have more mental health issues than elderly people, but that isn’t solely because of COVID-19.”

In his view, the five-yearly survey among young people, which his institute conducted once again in December, shows a slight increase in mental health issues. “But nothing is happening that is of immediate or major concern. The same emerges from the annual survey by Statistics Netherlands, which is representative as well. Nothing indicates that things are going horribly wrong with youngsters of that age in the Netherlands.”

UvA professor Van der Maas would be interested in conducting follow-up research with the questionnaire of the RIVM survey. “I would be inclined to conduct some of the survey in the old-fashioned manner – on the campus of an institution, among around a thousand students. It’s a lot of work but the non-response would be a lot lower. If the same numbers were to emerge, I would have to admit that I was wrong, but I have the feeling that you would really get different data from it. With a response rate of 75 percent you would be able to check the present survey result properly.”

Too easy

In a written reply the creators of the Monitor admit that the low response rate is a weak point in their research. “We describe this at various points in the report and have (…) been cautious in drawing our conclusions.”

The researchers do not rule out the possibility that a relatively large number of students with mental health issues completed the questionnaire. “If a bias of that kind arose, it might have contributed to a more negative image of students’ mental health than is actually the case in the general student population.”

They say that it is hard to assess how great that possible distortion is. “However, in our view it is too easy – and not correct – to ascribe the findings solely to that possibility.”

To underline that, they refer to another survey by the RIVM and Statistics Netherlands (only available in Dutch) showing that during the third wave of COVID-19 young adults “suffered considerably more from mental health issues than the general population”.

Finally, they stress that almost 30 thousand students from 15 different institutions took part in the survey. “So it isn’t a fact – as has been stated – that ‘few students’ completed the questionnaire.”