Immersive Multimedia Technologies I

This course describes volumetric data representation techniques used for capturing and rendering spatial videos from Apple Vision Pro [1] and/or Google Starline [2] for holographic video conferencing. Volumetric representations and their immersive rendering, recently published in the MPEG-I standards ("Moving Picture Experts Group – Immersive") will also be covered (compression itself is studied in INFO-H516).

INFO-H518 presents techniques primarily based on image processing—for example, creating a 3D perspective illusion by merging 2D images, cf. the image below (calculated from one single input image), extracting depth from stereo matching, etc.—and is therefore complementary to INFO-H502, which focuses on 3D graphical representations and their OpenGL rendering of content described by connected triangles. Both approaches can be used in virtual/augmented reality applications such as the Metaverse, but the former is preferred for free navigation within photorealistic content.

Such a high-quality, immersive, and photorealistic capture-rendering processing chain relies on precise calibration of the multi-camera system, optical distortion removal, image stitching to create a panoramic image, digital stereoscopy, alignment of two point clouds using Iterated Closest Point (ICP), and/or Structure from Motion (SfM) estimation for volumetric (3D) reconstruction from 2D camera images. Exercises using online software linked to IPOL publications ("Image Processing On-Line") will help students in mastering the various concepts (students don’t have to do much programming; the scientific study of IPOL papers will be paramount).

The proposed multi-camera capturing approaches are also used in visual effects studios, in recent driver assistance systems that provide a complete image of the vehicle's surroundings, and in immersive cinemas offering a 360° view of the film. Extensions to 3D light field displays that do not require stereoscopic viewing glasses (autostereoscopy, holography) will also be studied, and the student will implement a simplified version on a 3D display from the brand "Looking Glass" [3] (using its open-source drivers, IPOL and/or OpenCV modules and/or ad-hoc programming in C/C++ and/or Python).

Understand volumetric rendering techniques based on images (and video).

Required and Corequired knowledge and skills

C/C++ programming skills are recommended, though the exercises can be done in Python only (but the case study examples will mostly be in C, often without object-oriented concepts).

Theory lessons will explain the main ingredients of immersive 3D rendering technologies, complemented with exercises (mini-projects) for mastering their basics.

References, bibliography, and recommended reading

[1] Apple Vision Pro: The era of spatial computing,  https://www.apple.com/au/apple-vision-pro/

[2] Jason Lawrence, et. al., “Project Starline: A high-fidelity telepresence system,” ACM Trans. Graph., Vol. 40, No. 6, Article 242, December 2021, https://doi.org/10.1145/3478513.3480490

[3] Light Field displays, https://lookingglassfactory.com/displays-overview

Course notes

• Syllabus
• Université virtuelle

Additional information

Slides and notes (close to a complete syllabus) inspired by [1,2] and their bibliographic references, as well as video clips for some subjects.

Contacts

Prof. Gauthier Lafruit, LISA-VR

Campus

Solbosch

Method(s) of evaluation

• Oral examination

Oral examination

A report on the exercises must be submitted before the exam period.
The oral examination covers one theory question, as well as punctual questions on the exercises report.
This is no open book exam, though a single A4 page with formulae is allowed (besides of a copy of [2]).

Mark calculation method (including weighting of intermediary marks)

50% on the theory question, 50% on the exercises' questions (and report).

Language(s) of evaluation

• english
• (if applicable french, Dutch )