Connecting live performances of the future at IBC2024
With artists and fans alike looking to reduce their environmental impact and support more sustainable events, d&b recently took part in an IBC2024 Accelerator challenge, exploring innovative ways to link venues for simultaneous live performances.
Part of the annual IBC show in Amsterdam, the Accelerator programme explores potential solutions to address critical industry challenges. At the beginning of the year, eight challenges were chosen, providing opportunities for collaboration and experimentation within the media, entertainment, and technology communities.
One of the projects, proposed and co-led by d&b, focused on creating seamless remote experiences by connecting artists and audiences in multiple locations by delivering synchronised ultra-low latency audio, video, light and media data (ULL-AVLM). This would enable artists performances expand real-time outreach to audiences by connecting with remote live experiences.
But for that to happen, remote performances must deliver natural, authentic and immersive experiences, ensuring equitable quality across all venues, regardless of performer location, while also enabling bi-directional interaction between performers and audiences.
“The need to synchronise the video, audio and motion capture data stood out as an important thread in the challenge of sharing performances across multiple venues,” explained Andy Hook, Director of Technology Strategy at d&b. “In addition to motion capture data to drive rendered graphics, we also include full lighting control data as well as telemetry required to generate spatial audio, adding to the immersive experience.”
The project was driven by cultural, media, broadcasting and academic institutions, including University of Strathclyde, Music Venue Trust, EBU, Royal Central School of Speech and Drama, Kings College London, University of Kent and TV2 along with the partnership of innovative AV technology and connectivity providers d&b, Neutral Wireless, Spectral Compute and Salsa Sound.
As first-time IBC Accelerator Champions, the team at the Music Venue Trust (MVT) were not sure what to expect from the project and how a cultural charity partner would fit in, according to Chris Sherrington, Policy & Strategy Support at the MVT. “But we were very happy to be invited to represent the grassroots music sector the project team helped ensure that we were able to actively engage” he said. As the project progressed, MVT grew in their presence and provided valuable contribution. “Over the course of the project it became clear that the potential applications were even more wide ranging than we first thought, and we were particularly excited to see the ways in which it could help to increase access to performance in all areas of the creative ecosystem from audiences to artists and crew.”
The Wrapper
While existing methodologies allow the transmission of audio, video, lighting, and auxiliary data to a secondary venue as individual elements, doing this in an asynchronous way with varying delay times makes maintaining real-time synchronisation and immersive interactivity unfeasible. The key challenge this initiative overcame was the ability to stream all these data sources together in real-time at the same latency.
Rather than try to build something entirely new, the team chose to use existing protocols to ensure the widest possible compatibility at different venues and with standard equipment. The team tried to reduce latency wherever possible and explored the use of private 5G networks to transmit data within venues.
Sam Yoffe, a Research Fellow at the University of Strathclyde and Senior Systems Engineer for Neutral Wireless, said the collaborators instead worked to wrap all audio, video and other data feeds into a standard MPEG-Transport Stream that keeps them synchronised until they are fed into their respective hardware at the remote sites.
To help reduce barriers to collaboration, learnings and traction, a rapid iteration approach was taken to help explore the ‘Wrapper’ solution. Yoffe felt that it promoted a very positive and active collaboration. “The regular testing and Proof of Concepts allowed us to build up and integrate the different components into the system, eventually covering the full AVLM spread.”
“It was great to see how quickly new ideas and concepts went from thought to implementation.” Michael Søndergaard, CEO at Spectral Compute, added.
As Sherrington explained “The hackathon and multi-platform online format allowed us to work with a wide variety of partners with minimal resource commitment and the project was really well managed.“
Ahead of the IBC2024 exhibition showcase, for the final Proof of Concept, the team delivered a full live demonstration of the technology. An interactive live performance was carried out between The Sound Lounge, a MVT grass roots venue in Sutton, south London, and d&b’s Immersive Technology Experience Centre (ITEC) at the Science Museum in central London. d&b’s Soundscape system at ITEC enabled a fully immersive object-based audio recreation of the performance, which was enhanced with real time visuals and lighting, all controlled from The Sound Lounge.
Low-cost solution
Sherrington also lauded the IBC2024 Accelerator team’s dedication to seeking a low-cost solution that would make it accessible and affordable to smaller venues and a wide range of acts. “Touring is expensive and lots of artists simply can’t afford to reach the number of smaller local venues they might have done previously due to a lack of grassroots funding support. Innovation in connected performances is one way of allowing venues to drive more ticket sales, empower artists with wider reach, and give creatives more access to advanced tools across sound, lighting and motion capture technologies.”
“There remain some unknown subtleties with the video and audio codecs, but the underlying principle and techniques are now established, and the system is very flexible.” Yoffe shared. “Building on this project’s success by extending the use of 5G connectivity to connect smaller and more remote venues marks an exciting next step in enabling ultra-low latency, real-time interactivity between locations.”
Ideally, widespread adoption of a standard ULL-AVLM protocol for immersive remote experiences could help struggling regional venues gain access to those top-tier artists looking to reduce their touring carbon footprint, but also give less well-known performers novel ways to reach more fans.
Now d&b and its partners are working hard to turn those tantalising possibilities into reality and are excited to showcase further developments.
“It’s been fantastic to see this breakthrough initiative come to life, and we’re excited to see how this type of technology can benefit the live entertainment industry” Søndergaard concluded. “We're so pleased with this project that we can't wait to continue collaborating on future expansion on the technology”
To learn more about this IBC Accelerator ‘Connecting Live Performances’ and listen to the project team’s proof-of-concept demonstration and Final Showcase Session at IBC2024, click here.