High Road Stories is the creative lead in a research project designed to take the use of volumetric video in musical XR experiences to the next level. In partnership with the Fraunhofer Heinrich-Hertz-Institut and Berlin Phil Media GmbH, we research and develop two use cases: an XR violin masterclass with a top musician from the Berlin Philharmonic, and multi-user pop concert experience.

In close collaboration the project partners are jointly exploring and realizing new forms of interaction with musicians, the virtual environment, and the music itself, with the goal of creating deeply immersive and participatory experiences.

Fraunhofer HHI sets standards in the lifelike 3D capture and processing of real people using volumetric video. Musicians and the finest details of movement and instruments are portrayed through novel Gaussian splatting methods in combination with extremely high-resolution cameras. To achieve this, the team of Fraunhofer HHI is developing a novel 3D recording system, AI-based reconstruction methods, and a scalable streaming format for modern head-mounted displays that enables high quality with reduced bandwidth requirements.

Creative lead High Road Stories is responsible for developing the user experience and visual design of both use cases, supported by the vast expertise in music education and communication of Berlin Phil Media GmbH.

Berlin Phil Media GmbH, in collaboration with Henrik Oppermann (Schallgeber), oversees the capture and playback of spatial audio and will set a new benchmark in the recording, processing, and playback of spatial audio with six degrees of freedom in XR. This allows users to move freely within the virtual space and perceive changes in sound based on their position. 

The consortium is committed to bringing the research use cases into commercial practice.

Funding for the project, which runs until September 2027, is provided by Investitionsbank Berlin, with financing from the European Regional Development Fund (ERDF) and the State of Berlin.