The project name: Enigma, after the coding machine that got famous during the second world war in nazi-Germany. “In this project, we have built a real-time 3D reconstruction system (hardware and software) that allows soldiers and commanders to enter and operate in unknown buildings”, Bondarau explains proudly. “The soldiers go in with a stereo camera located on the shoulder built on a rig and with a laptop in their backpack. The algorithmic system automatically reconstructs the 3D structure of the building as they move forward and sends the model via wifi or radio protocol to the commander. The latter can be outside in a van or on the other end of the world, that doesn’t matter. He then sees the soldier moving trajectory inside the 3D model and can give him instructions.”

Stereo camera

Three years ago, Bondarau and professor Peter de With (also in the VCA group, ed.) received a request from the Ministry of Defence saying they need an equipment that allows soldiers to enter unknown environments, and helps to see the indoor 3D structure. Soon after, Bondarau and his PhD student Xin Liu started the project, together with ViNotion. “We were quite free in the selection of sensors to use, but they could not be active, meaning they could not send out any kind of radiation as that would make soldiers discoverable. But we needed 3D sensors for the project and most are active, emitting IR light. That was a problem. The only option was a stereo camera, which is a system containing two cameras several centimeters apart from each other. While the stereo camera has also allowed us to reconstruct depth, the quality of this data is very poor. Half of the obtained data points is simply noise due to the passive nature of the stereo camera. To detect and remove this noise, we have developed a set of 3D filtering algorithms. This appeared to be the most challenging research part of the project.”

Harskamp

Bondarau shows the presentation with many 3D models of the building gathered on the training site of Harskamp, close to Arnhem. The 3D models were rendered by the developed coordinator app and every time the soldier moves, this 3D model structure is updated. The images did not only contain the building but also all the routes the soldier has taken and the current position of the soldier. The system even automatically discovers if a soldier goes upstairs and creates different 3D models (regarding levels) for different floors to keep things organized. The system is also trained to semantically segment different objects. It can now distinguish between walls, windows, doors and certain big furniture, humans and guns. “The latter two still need improvement. For the Ministry of Defence it is important that the system can recognize those parts of the building and indicate them. Windows are areas where potential danger can come from. That also goes for doors, which at the same time can be a means of escape for the soldier. Therefore these are very relevant for the commander to give good instructions.”

But sending the data to the commander without wifi can cause another problem. “Sometimes you’re lucky to have wifi and that allows you to send bigger files. But without wifi, you need to use radio protocol and the images the system makes need to be converted into AutoCAD-like files for the smaller bandwidth of radio protocol.” Unexpectedly, Bondarau had to deal with this issue at the university as well: “Due to corona, we could not present the final tool at Harskamp and had to do it at TU/e instead. The commander was in Flux and a colleague and myself would play the soldiers and walk through MetaForum to reconstruct the sixth floor. We tested it the day before the presentation and the morning before the presentation and it worked perfectly. But then during the presentation, the wifi at Flux stopped working. We were very surprised as it normally always works… Luckily, we recorded the process and the results of two tests before the actual presentation, so we could still show them the video.”

Boys’ dream

The testing of the system at Harskamp really made an impression on Bondarau. “The army was training and shooting while we were there. Machine guns around us, tanks and many bullet casings on the floor. It was strange, as if we were in a war situation. But at the same time it was also very impressive and a bit of a boys’ dream to see this all up close.”

The new system can really help soldiers, as until now they don’t have anything but radio when entering an unknown building. Major of the Marines at the Ministry of Defence Roel Driessen, who was involved in this project, says in a personal capacity that he is reserving judgement about the implementation of this technology: “You have to see this as a feasibility study, it is great that we know that it is possible, but not all results are directly used in practice. It's helpful for us if a commander outside a building can see what's going on inside, especially if he doesn't have a map of the building. The passive sensors used in this project are of great value. We get the information from cameras that do not emit radiation. This is important because if a camera does emit infrared radiation, everyone can see from the sky that you are present on the ground. Preventing this was also the main challenge in this project.”

Too much data

The system the VCA group made is based on deep learning, a technique in which you teach an algorithm certain rules based on big data and then the system applies those rules to new situations. “We were lucky to have already some existing open data availabe of algorithms knowing how to detect doors and windows. But, for example, stairs were insufficiently present in open datasets so we had to teach the system how to recognize those.”

“The system still needs some improvement. It needs to learn to recognize people and weapons better and execute faster at lower computation costs. The ministry wants an updated map every two seconds a soldier is in a building. With small houses this requirement is easily met, but in bigger buildings, like Metaforum or Atlas, where they also practiced, the system produces map updates only once every five seconds. “Even the most expensive and strong laptop that we bought for this project, couldn’t process that much data that fast to meet the requirement, so we still need a solution for that real-time issue.”