Metaverse in the industry: remote maintenance with the avatar

The avatar’s head hovers over the machine. The spooky hands point to the knife that needs to be replaced and the screw that needs to be unscrewed. In the headphones of the data glasses, the voice explains step by step what needs to be done and answers questions. We are in the AR world, which stands for augmented reality, an extension of reality through displayed digital information or: Industry 4.0 plus.

The machine, a granulator for plastics recycling, is very real and is located in the factory building of the medium-sized mechanical engineering company ZWi Technologies in the Troisdorf industrial park near Bonn. The technician who gives the instructions to the machine operator as an avatar sits in an office, also wears data glasses and has the digital twin of the granulator in front of him in the room. With the joysticks in his hands, he marks components on the three-dimensional image. At the same time, he sees on video what the machine operator’s camera is recording: the machine through someone else’s eyes, so to speak.

ZWi has been involved in the research project since the beginning of 2021 5G industrial city park Troisdorf together with the film producer Kuraray, the Fraunhofer Institute for Applied Information Technology (FIT) and other research institutions. With funding from the Federal Ministry for Digital Affairs and Transport (BMVD), a 5G campus network is being set up in the industrial park and applications are being developed for which real-time data transmission is important. One of them is a so-called 5G toolbox for medium-sized companies: a service package consisting of hardware and software for complex remote maintenance tasks.

They don’t just want to play

While virtual worlds used to be reserved exclusively for gamers, companies are now increasingly using them for marketing, recruiting, meetings and training. “The Metaverse is the next big thing on the Internet,” says Leif Oppermann, who researches technical solutions for Mixed and Augmented Reality, or MR/AR for short, at the FIT in St. Augustin near Bonn. He sees great potential for assistance systems in production, maintenance and logistics in the merging of virtual and real worlds. The Troisdorf research project is not yet a metaverse in its purest form, but at least combines several of its elements: “This means that two spatially separated people can work on the digital twin and on the real machine at the same time and communicate with the help of avatars as well as audio, video and gestures.”

Digital twins, avatars, data glasses and other wearable devices are the components of Metaverse, the virtual parallel universe in which real things and people have their copies, can interact and transcend space and time. There are several platforms like Decentraland, Omniverse, Roblox and sand box, which, however, do not form a universal metaverse accessible to all. Tech companies like Facebook parent Meta are investing billions in the technology, but many smaller companies are also tinkering with their own virtual environments.

No frills this time

Older people will still remember Second Life: Every self-respecting company secured a plot of land in the virtual parallel world and invited to meetings and conferences there, to which imaginatively styled avatars flocked. The boom didn’t last long. Data glasses were still unknown in the early 2000s, the Internet was neither fast nor wireless and down-to-earth industrial companies saw no added value in virtual bells and whistles. Then the pandemic hit and catapulted remote services to the fore. Apprentices had to learn at a distance, service technicians had to commission systems and fix problems remotely.

“During the Corona period, we helped each other out with videos,” says Lukas Odenthal, project manager for development and construction at ZWi. The Troisdorf medium-sized company builds special machines mainly for plastics producers: There are only a few of these very special systems in the world. Consequently, there are no standard solutions for any technical problems. If a customer had one, he called ZWi, described the situation and made a video of it. “We sent our instructions back via video: that wasn’t very efficient.”

communicating tools

With the new tool box via 5G, communication should take place in real time and without loss of information. Even more: “We are in the process of putting tools in there that can actually communicate with the data glasses,” says Odenthal. So the smart wrench could automatically tighten the screw with the exact torque you need. This requires a digital twin of the machine with a 1:1 image on the internet. Creating this does not mean any additional effort, says the project manager: “The data is available because we have been designing the machines with computer aid for a long time.” Even the stock could be retrofitted with twins. “It’s just not possible with old machines that were still drawn on paper with ink.” And the avatars? “We noticed that they simplify things. You see the avatar’s point of view, you see what their hands are pointing at instead of marking the spot with arrows.”

The Japanese specialty chemicals group Kuraray produces films for laminated glass at the Troisdorf site. Kuraray uses the ZWi granulators to shred the film residue and reuse it as a raw material directly in production. With the help of the new technology, the employees receive an exact picture of the systems, enriched by operating instructions and construction plans. This allows faults to be rectified and new workers to be quickly trained for daily cleaning. “Then it would even be possible for our technicians to work from home,” believes site manager Holger Stenzel. As a cooperation partner, the Institute for Work Health (ILAG) checks how well employees feel in the mix of the real and virtual world.

Real time needs 5G

According to a recent survey by the digital association Bitkom, almost a third of German companies are critical of the Metaverse topic, but a quarter are open-minded. The idea is that a medium-sized company that currently only has its own website can use the toolbox to jump straight into the Metaverse. If service technicians do not have to travel all over the world, it saves costs, time and CO2 and remedies the shortage of skilled workers. “For some problems I would actually have to send three people to the customer: an electrician, a mechanical engineer and a computer scientist,” says Odenthal. “Virtually, I can just turn them on.”

So that the mobile devices deliver pin-sharp images, the videos do not stutter and the information arrives without delay, the latest 5G generation of mobile communications is required. A 5G campus network offers locally limited super-fast data transmission in the required bandwidth. The FIT already has such a test network at its location in St. Augustin, in which it tests the devices. In the industrial park near Bonn, the subsidiary of Stadtwerke Troisdorf, TroiLine, is building a 5G campus network of radio base stations and fiber optic connections. Due to delivery problems, however, the work is taking longer than planned. In mid-2024, Oppermann estimates, it should be ready.