[Interview] Movie-Quality Audio From the Comfort of Your Home: Meet the Leaders of Next-Generation 3D Audio Technology

For viewers to enjoy content such as movies, sports games and live events to the fullest, home TVs not only need excellent picture quality but impressive sound quality as well. Understanding this, industry leaders are pursuing 3D audio development to add a deeper level of immersion to entertainment experiences — bringing to life thrilling action scenes and capturing the quietest details of ambient noise.

Put simply, 3D audio makes listeners feel as if they are really in the middle of the action. Typically only supported at specific venues such as movie theatres or recording studios, 3D audio is making its way into homes to give viewers a new way to experience their favourite content.

Samsung Electronics’ advanced research institute Samsung Research has been striving to popularise 3D audio since 2020. To this end, Samsung worked with Google to jointly develop Immersive Audio Model and Formats (IAMF) — an advanced 3D spatial audio technology that was adopted by the Alliance for Open Media (AOM)^[1] in October 2023. Samsung Newsroom sat down with the team that developed IAMF to hear the story of how the technology came to life.

▲ The Visual Technology Team at Samsung Research led the development of a standard for audio technology. (From left) SungHee Hwang, JeongHoon Park and WooHyun Nam

A New Way To Enjoy 3D Audio at Home

Ambient noise exists everywhere — from the sound of shoes scraping on the sidewalk to the gentle hum of cars driving down the road. While this noise may seem unnecessary in media, it is vital in adding depth and a sense of realness. 3D audio works to seamlessly blend ambient noise with conversations and other sound effects to create a more vibrant and authentic entertainment experience.

“3D audio makes you feel as if you are really in the heart of the action by adjusting the strength, movement and vibrance of the sound,” said WooHyun Nam from the Visual Technology Team at Samsung Research. “In this way, viewers are able to enjoy a more full-bodied sound that captures the 3D aspects of the world around us.”

▲ SungHee Hwang from Samsung Research’s Visual Technology Team

Despite the advantages of 3D audio, it has been difficult to apply the technology to home audio devices due to technological limitations. “3D sound information from content cannot be interpreted properly by home audio systems such as TV speakers or sound bars, resulting in a slightly limited audio experience that lacks detail from the original content,” said SungHee Hwang from the Visual Technology Team at Samsung Research.

To resolve this, Samsung worked diligently with Google to develop an audio solution that would allow viewers to experience content audio as intended. “If the 3D audio data can be read by device manufacturers, they can adjust the sound in audio devices — allowing for immersive audio experiences with standard TV speakers or sound bars at home,” said Nam. “By adjusting the audio to match the home device environment, listeners can experience audio just as the creator intended without any distortions or loss in quality.”

A uniform standard is required to send and receive audio data smoothly between creators and device manufacturers. “Samsung and Google’s respective expertise in devices and content made the companies ideal partners to create IAMF technology,” said JeongHoon Park, Executive Vice President and Head of Visual Technology Team at Samsung Research. “By coming together to develop this unprecedented technology, we are paving the way for consumers to enjoy 3D audio in their homes.”

Three Characteristics of IAMF Technology: Vertical Audio, AI-Based Audio and Customised Audio

IAMF technology offers three distinct features that enhance the audio experience.

▲ WooHyun Nam from Samsung Research’s Visual Technology Team

1. Ability To Express Sound Vertically

Previous open-source audio codecs only supported horizontal sound expression. With IAMF technology, audio can now be expressed vertically so that sound is increasingly multi-directional. “IAMF makes sound more realistic, by allowing listeners to hear audio in front, behind or to either side and also above or below them,” said Nam. “As such, when IAMF technology is applied to home TV speakers and sound bars, it allows listeners to hear sounds such as birds flying over their head on their TVs at home.”

2. AI-Based Scene Analysis and 3D Audio Effects

IAMF utilises AI and deep-learning technology to analyse scenes and emphasise certain aspects of the content — adjusting audio levels for more enhanced sound throughout the viewing experience. “In TV and film, there are certain scenes where the soundtrack or background music is the main focus,” said Nam. “IAMF will balance the sound in these instances. Similarly, the technology will fine-tune audio when there is character dialogue to allow the listener to focus on the conversation.”

In addition, IAMF technology provides optimal sound despite changes in the device environment. “By adjusting the scene analysis audio data according to the device environment, IAMF technology enables listeners to enjoy the content’s original sound quality on standard home TVs,” he added.

3. Highly Customised Audio

Users will be able to freely adjust sound according to their preferences with IAMF technology. Whether viewers are looking to amplify sound effects from an action scene or enhance dialogue, IAMF gives them the flexibility to customise content audio for a more personalised experience.

▲ IAMF analyses data from content and allows viewers to adjust and customise audio settings. When watching a sports game, users can directly choose whether to emphasise the commentator’s voice or the sounds of the game itself.

3D Audio Across the Industry With Open-Source IAMF

Open source is crucial to creating a unified standard across the industry. IAMF is the first open source-based audio technology standard adopted by AOM — meaning both corporate and independent content creators across the industry can access the technology and expand its use.

“In order to allow people to freely create content with 3D audio technology, related technology needs to be open to all,” said Nam. “Providing a complete open-source framework for 3D audio, from creation to delivery and playback, will allow for even more diverse audio content experiences in the future.”

Similarly, Park highlighted how IAMF technology will have a large impact on the audio landscape moving forward. “Because we live in an era dominated by content creation, IAMF will help lead, expand and transform the 3D audio ecosystem,” he said.

Success Accomplished Through Cooperation

Research on IAMF began in 2020 and took nearly four years to complete. Through a combination of perseverance and hard work, the team was able to achieve success.

“The project involved many days of nonstop work, and there were times when we needed to work at night due to the time difference between our office and Google’s,” said Hwang. “Despite these challenges, I was proud in the end. It was a great opportunity to work with developers from other countries.”

The teams developed a sense of camaraderie along the way and built a solid foundation of trust and understanding. “A Google employee mentioned how this partnership with Samsung has been the most enjoyable collaboration he had completed so far,” Park recalled. “We were able to accomplish this project because our teams trusted each other. I am very proud of and thankful for our team members’ efforts and their cooperation.”

▲ JeongHoon Park, Corporate EVP and Head of the Visual Technology Team at Samsung Research

Additionally, Park praised each team member’s unique skills and expertise which helped contribute to the creation of IAMF. “WooHyun Nam was instrumental in proposing the technology development ideas while SungHee Hwang organised and documented our team’s ideas in a clear-cut manner,” he said.

A Vision for ‘Samsung Sound’

Upon its release, the Samsung Research team realised that the standardisation of 3D spatial audio marked a new era of sound technology.

“Thanks to IAMF, we can begin researching technologies that will open up a world of audio possibilities,” said Nam. In line with this, the Samsung Research team is currently developing an advanced version of IAMF technology that can be applied to different sectors such as mobile devices, the metaverse, video games and more.

▲ With their combined efforts and unrelenting persistence, the Visual Technology Team at Samsung Research was able to successfully create a new standard for 3D audio.

Given the success of IAMF, the Samsung Research team is motivated to create better audio technology for consumers. Each of the team members shared their aspirations for the future of 3D audio.

Nam touched on his desire to create audio technology that is more immersive. “I want to create a more advanced 3D audio technology that makes users feel like they are truly in the scene of a movie, TV show or live event,” he said. “I also hope to continue this research until 3D audio is applied to Samsung’s smartphones.”

More broadly, Hwang discussed how he hopes Samsung will be able to create unrivalled audio technology that will place it among the ranks of other leading audio companies. “My goal is to develop a technology that consumers will be able to easily distinguish as ‘Samsung Sound’ when they hear it,” he said. “I am optimistic that the IAMF standard is a stepping stone toward this dream.”

“My hope is that Samsung’s sound technology will enable consumers to enjoy an upgraded audio experience that is on par with that of current visual experiences,” said Park. “I hope that content creators will utilise Samsung’s audio technology to make 3D audio content accessible. Additionally, we aim to create a supportive environment that enables researchers to take on bold and exciting challenges such as expanding Samsung’s audio technology.”

Technological standards such as IAMF will make entertainment more immersive for viewers across the board. The Samsung Research team is paving the way for innovations that will change the future of audio.

^{[1] A non-profit industry consortium formed to develop and share technology for multimedia delivery, allowing creators to utilise the technology without worrying about cost. It is operated by an alliance of 38 companies including Samsung Electronics, Google, Amazon, Apple and Meta.}