Visual Media Compression

Image compression is an essential technique for reducing the amount of information transmitted and is the cornerstone of smooth, lag-free streaming of video content (YouTube, cable TV, etc.). It also allows for the storage of over 100 times more videos on servers without significantly compromising quality.

This course primarily revisits the ISO (International Organization for Standardization) standards for JPEG (Joint Photographic Experts Group) image compression and MPEG (Moving Picture Experts Group) video compression — for example, YouTube, MPEG-4, MV-HEVC for DVD/Blu-ray and Apple Vision Pro's spatial video, digital cinema using JPEG2000, etc. — as well as the compression of 3D representations (for example, Google's "Draco") where 3D is often converted into 2D images, well-suited to the video codecs mentioned above.

Despite significant progress in compression performance over the past 30 years (an order of magnitude improvement, progressing from MPEG-4 AVC to HEVC and finally VVC), the basic architecture of the video codec (= the system that provides compression) remains quite similar, cf. the image below [1], and this is what will be studied in detail: exercises will highlight the different modules and performance metrics of a basic software package (much simpler than what is found in software such as ffmpeg and VLC, often used in this field). It's a bit like thermal combustion engines in cars, which still operate on the same principle of controlled combustion with a transformation into rotation by pistons (even if the first cars probably bounced more than they moved), but whose performance has increased through continuous refinement over more than a century of engineering.

The INFO-H516 lectures will introduce all the elements of the jargon and ecosystem of compression standardization (ISO Sub-Committee SC29) and its technical working groups (Working Groups WG1 to WG8). A reference source code (for a simple use case) will be made available to the group of students, and as is the case in all SC29/WGxx activities (in which the lecturer has participated for over two decades), students will be required to modify this code to (slightly) improve its performance. This is therefore a collaborative project led by a (technical) project manager, where each student will focus on specific software modifications to achieve a collective goal (even though the 12-hours project is advertised as "a personal project"), and a large part of this work will be completed during the exercises. This highly practical approach (requiring a minimum level of C/C++ programming skills) is the best way to understand the very specific world of compression (where theoretical concepts, such as the definition of entropy, offer little practical added value).

Alongside the exercises/project, the lectures will present different codec variants – primarily AVC (Advanced Video Codec), HEVC (High Efficiency Video Codec), and JPEG2000 (Cinema codec) – and their technical differences so that participants can appreciate the rationale behind this diverse family of codecs (despite a similar basic architecture). This will highlight the difficulty of finding "all-in-one" software online (something ffmpeg and VLC strive to achieve) and why it's often necessary to resort to workarounds when using recent codecs (which are not yet well-mastered/established), such as MV-HEVC (MultiView/stereo-HEVC) and its parametric extensions in the context of Apple's "Vision Pro," see course INFO-H518.

This course is therefore a hands-on, practical introduction (with simple use cases) for anyone who wants to delve into the world of image/video compression. It is also a unique opportunity to appreciate the technical decision-making process that is followed in these digital media standardization committees.

Understand a typical image or video coding pipeline, starting from source code of a very basic codec that will be gradually improved by adding new functionalities, following the JPEG/MPEG standardization workflow. 

Required and Corequired knowledge and skills

Students should well master C/C++ programming (Python for some data format conversions is also helpful).

An image/video coding pipeline is developed, starting from source code of a very basic codec that will be gradually improved by adding new functionalities following the JPEG/MPEG standardization workflow. Student groups will collaboratively reach the targets of the compression project, where the use of github (and a software coordinator) is recommended (not compulsory).

References, bibliography, and recommended reading

[1] University Pompeu Fabra in Barcelona, « Video Encoding Systems » course online, last update 24 Dec 2021, https://video-encoding-course.vercel.app/theory/2-MPEG
[2] Iain E.G. Richardson, "H.264 and MPEG-4 video compression: video coding for next-generation multimedia," Wiley, 2003, ISBN 0-470-84837-5
[3] D. S. Taubman, M. W. Marcellin, "JPEG2000 – Image Compression Fundamentals, Standards, and Practice," Kluwer Academic Publishers, 2002.

Contacts

Prof. Gauthier Lafruit, LISA-VR

Campus

Solbosch

Method(s) of evaluation

• Project
• Oral examination

Project

Oral examination

• Open book examination

• Open question with short answer
• Visual question

The project report must be submitted before the exam period.
The oral examination covers punctual questions on the report, as well as some complementary theory questions. 
Even though the project is done in group, the exam is individual (but the report can be submitted as a group report).

Mark calculation method (including weighting of intermediary marks)

50% on the project questions (and report), 50% on the complementary theory questions.

Language(s) of evaluation

• english
• (if applicable french, Dutch )