Using the name lifeClipper3, researchers at the Institute for Research in Art and Design in Basel have created a film-like immersive experience that mixes the real world with virtual reality.
Led by Swiss design researcher Jan Torpus, this outdoor walking experiment brings virtual elements to the landscape of Sankt Johanns Park on the banks of the Rhine. With the help of special equipment, users undergo an experience that changes character depending on the direction they take, what they look at, or even how they breathe.
As the user walks through the park, the grass turns into a desert, fish-like animals fly through the air, and lying in the middle of the meadow is a huge green worm into whose mouth participants can enter.
“It always makes me happy when people don’t know if something is real or not,” says Torpus.
“I want to take people out of what they take for granted. It’s like the Matrix idea that behind all you see around you, there is something that is reality.”
The dual experience of being in the park but feeling you are part of a movie is possible thanks to a sophisticated equipment set-up. It involves a laptop, a GPS receiver to locate the user, a video camera, a head-mounted display with goggles, 3D surround-sound headphones, a vision sensor that determines where the user is looking, and a biofeedback sensor that measures the user’s breathing.
A total of five software programs run at the same time, continually communicating with each other and allowing the user to interact with the park by means of position, vision and breathing.
The creators of lifeClipper3 scanned the whole park with an infrared camera before making a 3D model on a 1:1 scale, but overlapping the two datasets was tricky.
“The most difficult part was that the real and the virtual spaces always had to coincide, which is almost impossible,” says Torpus, who led a multidisciplinary team of engineers, IT specialists, musicians and new media artists.
And what’s the aim of such a creative and seemingly bizarre project? Torpus says that lifeClipper3 exists somewhere between art and research. It enables experimentation with new media design, but it also facilitates the examination of human perception and emotions in situations where people are taken away from their everyday lives and thrust into an augmented reality.
Torpus also believes that, like all new art forms, it will take time for augmented reality to be accepted.
“It’s like cinema. At the beginning no one respected it as art; it was just a new technical discovery, but today it is [art],” says Torpus.
But there are stumbling blocks. Often users have struggled to understand what they are viewing through the goggles or just can’t immerse themselves fully in the adventure.
“lifeClipper3 goes too far sometimes,” agrees Torpus, who insists that this element is part of the research project, which is not open to the public.
He adds that the experience does not work for people when scenes, elements or tricks stray too far from cultural norms or the best-known natural laws, or are too far divorced from the type of media literacy garnered from films or video games.
“The continuum between real and virtual is very different [from other experiences people may have had] because you don’t have that in movies or video games,” he explains. The trick is to avoid virtual unreality by finding a balance that allows users to fully immerse themselves in the scene to such an extent that they no longer know if what they are seeing or experiencing is real or virtual.
The equipment also needs to become less invasive. Carrying around 10 kilos of gear on your back gets tiring, while the head-mounted display offers limited vision. In order to avoid tripping or running into people, benches or trees, Torpus acts as a sort of Sherpa, a reminder to users that they are, after all, still in the real world.
lifeClipper 3 is the last of a series of augmented reality projects Jan Torpus has worked with for the last eight years. The two previous experiments dealt with the past and the future of the city of Basel. The projects were not unlike the plot of the novel ‘Virtual Light’ by famed American author William Gibson, with Basel stepping in for San Francisco. Torpus says they were better understood by participants because they were closer to their everyday lives.
Although Torpus admits it could be years before the new technology enjoys widespread use, he is confident of an eventual breakthrough.
“lifeClipper is the first sketch for a new medium,” he says.
“These could be the immersive movies of tomorrow”.