These are the so-called Rubicon grants offered by NWO, which researchers can apply for after obtaining their doctorate. The grants have been around since 2005 and are quite popular. They have acquired a good reputation. And yet the effects of these grants are rather difficult to measure, says Marielle Non of the CPB.

The NWO selection committees rank applicants from the strongest to the weakest. In a study for the CPB, Non looked at those applicants who ranked in the middle, just above or below the cut-off, she told the audience of an online seminar about her study. These are the applicants who either just made it or just missed it.

Effects

There’s no simple metric for comparing all the applicants, she pointed out. It may well be that the best researchers would have had successful careers anyway, while the weakest wouldn’t go so far. If you want to isolate the effect of the grant, there has to be a basis for comparing different researchers to each other.

However, surprisingly enough, for those ranked around the cut-off, there is hardly any discernible difference between the four winners and the four losers. How much did they publish in the years after receiving their grant? What was the impact of their articles? Who has stopped publishing completely?

There are no significant discernible differences. Sometimes the outcomes seemed to weigh a bit more in favour of the winners, but not enough. NWO committees have no trouble selecting the top candidates, but for those in the middle the choice is somewhat arbitrary, at least if you look at the figures. In her study for CPB, Non checked her outcomes in all kinds of ways, but whatever she tried the differences remained negligible.

Everything isn’t quantifiable

Some things just can’t be measured, she points out. For some individual researchers, the grant may be extremely important. Moreover, you couldn’t say that international experience is not meaningful just because the best researchers would have had successful careers anyway.

On the other hand, her findings are consistent with the often heard plea that a lottery system should be used for the pool of grant applicants who rank closely together. That might feel more justified than a rejection which isn’t really based on anything, someone suggested after the presentation had ended.

Another audience member raised practical objections. For such a lottery the committee would have to use two cut-offs: one for those who absolutely should be selected and one for those who should clearly be dropped. That would be extra work for results that would not be significantly different, and it probably wouldn’t put an end to the discussion about how to evaluate researchers just above or just below the cut-off.