Paul van der Corput flips his laptop open and shows a demo of the application ICLIC, which he built for his PhD. “A large amount of metadata is already being added automatically to digital photos,“ he explains, all the while scrolling back and forth through the application to show everything it can do. “Where the photo was taken, for example, and when, with what camera settings and which device.”

Moreover, software now exists that can automatically recognize a pictorial image, explains the computer scientist. “This kind of open software was improving all the time while I was studying for my PhD; it is being developed at universities. It now works with tens of thousands of key words, all of which are given a value between  ‘0’ and ‘1’.” If the picture definitely shows a car, person or chimpanzee, those key words are assigned a ‘1’. If there is any doubt, the score will be lower. “Like this, you can categorize hundreds of thousands of images very quickly.”

Flickr

In addition, Van der Corput also developed an application that can compare multiple image collections: PICTuReVis. He tested this software using all the images on the Flickr photo-sharing site that were demonstrably made in the Netherlands in 2015 – a half million photos made by some ten thousand different people. In the blink of an eye the application shows him which photos come from a certain person, and that the number of photos peaks in vacation periods. Next, he selects a number of key words related to the theme 'sailing' and shows that most of these were made during a couple of days in August, and in Amsterdam. “Sail Amsterdam 2015”, he declares not without pride.

It becomes really spectacular when he starts searching using cycling terms. After a couple of clicks, a map of the Netherlands appears with a ribbon of brightly colored dots running from Utrecht to Zeeland. Each dot marks a photo of a bicycle; this is the route of the second stage of the Tour de France, which that year started in the Netherlands. So on that day the entire route, from Utrecht to Neeltje Jans, was lined with people who uploaded their cycle photos to Flickr.

Investigation

Of course, there's also a serious side to his project, Van der Corput explains. “If you look at the photos from Sail, you'll see for example that two people took almost identical photos at almost exactly the same moment. Elsewhere in the collection the same thing happens again with these two people. From this you can deduce that they probably know each other.” And that is potentially interesting information for investigative services if one of the two is already under suspicion.

So it only made sense that the National Police and the Netherlands Forensic Institute were part of the user panel for the NWO project in which Van der Corput did his doctoral work. “Both institutions tested my applications using their own data. They were enthusiastic and said they wanted to take it further.”

Another side to the coin

Although Van der Corput is convinced that in the hands of investigative services his system can benefit society, he realizes there's another side to the coin. With this in mind, he draws the conclusions in his thesis to a close with a personal remark. “You can also see my work as a warning to consumers,” he explains. “It shows how powerful recognition and visualization software has already become. Where photos on Flickr are concerned, you know that they are public property, but the issue is less clear cut for Facebook. No one knows what Facebook itself does with them, and whether the US government is watching. Personally I am very wary and will always save my private photos locally, never in the cloud. Be aware that all those photos remain saved in principle forever and that they reveal a great deal about you. Including things you would rather not share with everyone.”

Photo | Bart van Overbeeke