By Dr Amal Punchihewa
Mile High Video 2024 (MHV-2024), one of the leading video coding and streaming events since 2016, took place recently from February 11-14, during which the four-day conference addressed emerging technologies that technology developers believe would shape the future of broadcast and media delivery, including planning for online (only) media delivery and how to manage the limited bandwidth, quality and latency in signal processing.
More than 14 sessions filled with presentations over three full days explored the latest developments in video coding and streaming. Three keynotes — Evolution of Internet Streaming: 50 Years of Discovery, Next-Generation Networks, and AI Steers TV System on Chip’s to New Frontiers, addressed the evolution of streaming and coding.
Additionally, experts in respective fields shared ideas and insights during panel discussions that touched on topics such as Open Caching, Advanced Video Coding: The Future Outlook for The Next Five Years, and Recent Advances In AI-Based Technologies for Video Coding & Streaming.
Importantly, more than 58 presentations addressed two key technology areas — video streaming and video coding.
Each area is covered with sub-topics; viz. for streaming: content delivery network (CDN)-based video streaming – systems and applications; over-the-top (OTT) delivery, ad-supported video streaming, scalable video streaming; video quality and energy efficiency in video streaming; low-latency media and media over QUIC; and machine vision, storage, and cellular/wireless networks.
The video coding topics that were covered include recent advances in video coding standards and coding schemes, film grain technology, as well as immersive/3D applications and depth maps.
The challenges in video streaming arise out of managing quality of experience (QoE), latency, multi-video CDN systems and dynamic content. A presentation talked about how to infer video streaming QoE at scale using incremental statistics from CDN logs. To steer content including dynamic content, a standard for multi-CDN streaming would have to be presented.
In the same session, the availability, performance and cost characteristics of multi-video CDN systems were discussed with low-latency, low-loss, scalable throughput architecture and active switching.
A session dedicated to OTT delivery, ad-supported video streaming, and scalable video streaming discussed efficiency improvements of OTT delivery with state-of-the-art streaming optimisations. In recent times, there have been many services and discussions about ad-supported video services.
MHV-2024 discussed next-generation linear ad-supported service experiences across multi-distribution platforms (MDP), including UHD-2 (loosely known as 8K) displays and how UHD-1 content can be displayed on those displays. The use of advancing computing power for video super-resolution and optimising per-CPU encoding efficiency for scalable and low-cost live streaming spatial scalability were also presented.
Piracy and digital rights management (DRM) are key challenges for the media industry. For OTT and streaming, how security can be improved and how watermarking can be applied at scale were also discussed.
One of the main impairments in streaming is the loss of video quality. With more concern for climate change and sustainability, industry stakeholders are researching, studying and trying to minimise the carbon footprint created by broadcast and media operations. A session was devoted to video quality and energy efficiency in video streaming. As energy is a high-impact area that affects sustainability in media and broadcast, this is an important issue for video streaming.
Hence, an energy-aware ABR algorithm for green HTTP adaptive video streaming and how to empower a greener future by having strategies for sustainable video streaming and energy efficiency was presented. The use of AI in streaming to improve quality by combining Deep Learning (DL) and feature engineering for perceptual video quality assessment (PVQA) and standards-based neural-network post-filters for improved video quality was discussed.
On video quality, controlling quality for real-time variable bit rate encoding and assessment with texture information fusion for streaming applications was also presented.
Quick UDP Internet Connections (QUIC) is a new transport protocol that operates over User Datagram Protocol (UDP) and uses Transport Layer Security (TLS) 1.3 for encryption. It was designed to improve the performance, security, and efficiency of HTTP traffic and other applications. It establishes multiplexed connections between two endpoints and QUIC claims that it can handle packet loss and congestion better than Transmission Control Protocol (TCP). It was formally adopted as a standard by the Internet Engineering Task Force (IETF) in May 2021.
MHV2024 discussed low-latency media and media over QUIC through several presentations. AV1 for live streaming and real-time communications with high quality in low latency was presented among other presentations focusing on ultra-low latency video delivery over WebRTC data channels, Just Noticeable Difference (JND)-aware zero latency encoding scheme for HTTP adaptive live streaming, prioritisation in media-over-QUIC transport and enhancing adaptive video streaming with latency-aware dynamic resolution encoding.
Broadcasters often use weather and mobile cameras to augment their news services. There is also an increasing use of image and video capture for other horizontal industries with high spatial and temporal resolutions. We must therefore transport and store that data efficiently without losing quality for further processing to extract information. 5G networks are being rolled out with more connectivity with devices, high bandwidth and low latency.
Hence, machine vision, storage and cellular/wireless networks are closely linked with streaming and media delivery. An overview of visual signal compression towards machine vision, use of 5G for enhanced uplink video streaming, optimising mobile video streaming through collaboration between content providers and 5G networks and efficient re-compression for storage saving in video-on-demand (VoD) streaming services, and on-demand bandwidth boost for improving video streaming over cellular networks with network APIs was also presented.
Video quality assessment is still a challenge for the broadcast and media industry. Structural Similarity Index – SSIM has been used as a better proxy for mean-observer-score (MOS), which is a subjective assessment. Continuing research and development work is being done to assess video quality objectively.
Video coding sessions shared the recent advances in video coding standards and coding schemes. These include end-to-end Versatile Video Coding (VVC) encoding for energy-aware High Dynamic Range (HDR) rendering; review and evaluation of VVC fast partitioning search methods using a common baseline; AV1 deployment at Meta and next-generation AOM codec development; accelerated and adaptive non-local mean temporal filtering for improved High-Efficiency Video Coding (HEVC) coding efficiency; hardware accelerated unsupervised codec modernisation and revisit of the objective quality metric SSIM to emphasis error visibility over structural similarity.
Video coding will also need to focus on film grain technology. The Joint Video Experts Team (JVET) provided an update on film-grain synthesis technology for video applications and the Alliance for Open Media (AOM) presented advances in film-grain encoding in the AV1 codec. Other presentations included AV1-compatible film grain modelling for HEVC decoders, a novel quality assessment metric for film-grain preservation, and film grain handling tools for VVC-based open implementations.
Finally in video coding, immersive and 3D applications, depth maps were discussed under the presentations on how to optimise compression of immersive 3D content in HEVC, as well as the no-reference quality of experience model for dynamic point clouds in augmented reality (AR).
An AI-based monocular depth-map estimation applied to a video encoding pipeline was presented followed by enhancing virtual reality (VR) with 360° data analytics and customised content delivery, usage of multi-plane image information for distribution of lightweight 3DoF+ immersive content with conventional 2D decoder, and a comprehensive test-bed for evaluating remote rendering performance in the evolving metaverse.
Participants at MHV-2024 certainly took a deep dive and emerged exhausted … but gained a dazzling vision of how emerging streaming and coding advancements can enhance the online delivery platform.