Sound is becoming an increasingly important component of AR/VR

Yaniv Elmadawi is vice president of technology solutions and services at Agora.

As we spend more of our lives in digital spaces, users want more natural experiences in those spaces. For those who are building these digital spaces, delivering realistic sound is essential. VR and AR enabled sound has been a challenge for the industry. However, AI is driving the evolution of virtual audio, and stakeholders should take some time to look (and listen) closely to just how far the most advanced technology providers have come.

We perceive the physical world through our senses, and our senses are not static. They react to the environments we find ourselves in, even in ways we are not consciously aware of. However, we do notice at some level when elements of a virtual environment respond in ways that the human mind registers as unnatural; we get that “uncanny valley” feeling. In an AR or VR enabled environment, users expect believability and realize when they don’t get it. In virtual reality and augmented reality, we need to create experiences that better reflect the person-to-person and person-to-environment connections we’re used to.

When looking at the practical applications of AR and VR, live events like concerts and sporting events often come up. These are environments where sound, with its naturally dynamic nature, is a central element of the experience. Part of the experience is based on the user’s position in relation to the sound source: the stage, the playing field, the speakers, etc.

Beyond entertainment, there are also business applications to be explored. What AR and VR can do for an arena event, they can also do for a conference. There are reasons why a conference experience is often more engaging and enjoyable than a webinar, and the dynamic audio you get from a 3D multi-room space is one of those reasons.

Industry leaders have spoken about how AR and VR enhance the shopping experience when trying products. What would you like to try on, say, a Bluetooth speaker? The consumer may want to hear how it sounds from different distances or in the types of spaces where a Bluetooth speaker might be useful. The consumer may want to know if it sounds as good on the beach as it does while listening in the shower.

Realistic audio is expanding in VR and AR as companies and creators learn to implement it. AR/VR technology works by introducing two main components: simulating an immersive sound environment (that is, simulating what the ear would hear while moving through a real-life small room, large hall, scene urban, wild environment, etc.), and noise suppression and human voice extraction.

Spatial soundscapes are designed to create more compelling and immersive experiences. However, today there are technical challenges in implementing effective spatial soundscapes. The most important is latency.

Audio latency is the time it takes for audio to reach the headphones. In a digital world where audio and visual communications are delivered simultaneously to the end user, any latency leads to complications such as desynchronization. In games, for example, an explosion or a car engine spinning at full speed in a curve that is out of sync with the visuals takes the player away from the immersive intent of the experience. Just like in a digital virtual environment built for work collaboration, any latency disrupts the intended person-to-person connections with colleagues.

Overcoming this challenge involves using a real-time network that sits on top of the in-person Internet and can provide a wide range of coverage around the world. AI algorithms can monitor these networks in real time and choose the most efficient routing path automatically, delivering ultra-low latency of 400 milliseconds or less.

The technological process for creating realistic sound experiences works similar to how the film industry creates visual effects. In movies, the actors first perform in front of a green screen. Human performances are drawn from those shots. Computer graphics are then implemented to blend those performances into a virtual environment, and lighting and effects are altered throughout the scene to make it appear real. Extracting human voice in a noise-cancelled studio environment works the same way.

Immersive soundscapes arise by leveraging multiple AI-powered technologies to address background noise issues. This technology is already being used in gaming, social networking, and workplace communications. Noise-cancelling headphones can help the listener regulate the soundscape at the receiving end, but AI-powered solutions can also eliminate distracting noise at the audio source end. Taken together, this technology can deliver state-of-the-art experiences for users by eliminating effects at either end that otherwise make the sound experience seem cluttered.

To move forward with this shift towards AI-powered immersive sound, organizations must first look internally to ensure the right structure is in place. The technology leadership overseeing the AI-powered solution must have a good understanding of how the technology works and have the ability to fix any potential issues that may arise.

Even more important, consider any challenges that may arise prior to implementation, have plans to quickly fix any issues, and limit any disruption the customer might feel would take them out of the immersive experience. There must be a strong communication strategy in place so that new and existing customers can understand how interaction with the business can improve and change as AI-powered sound is added to the mix.

When we make audio and video interact naturally in virtual environments, we can open up the AR/VR market to new audiences. Many consumers are still wary of exploring virtual spaces deeper than a superficial level; frankly, many people found the “uncanny valley” effect of less sophisticated AR/VR experiences too much to ignore. AI is driving creators and companies out of that valley, and we’ll know it when we hear the echo from the top of the hill.

The Forbes Technology Council is an invite-only community for world-class CIOs, CTOs, and technology executives. Do I qualify?