Projects

ColorVideoVDP: A visual difference predictor for image, video and display distortions

Rafał K. Mantiuk⁽¹⁾, Param Hanji⁽¹⁾, Maliha Ashraf⁽¹⁾, Yuta Asano⁽²⁾, and Alexandre Chapiro⁽²⁾.

⁽¹⁾University of Cambridge, ⁽²⁾Meta Reality Labs

Presented at SIGGRAPH 2024, Technical Papers

ColorVideoVDP predicts the visibility of distortions for a pair of test and reference videos (or images) as seen on a display with a provided specification. The predictions are represented as a single quality value in Just-Objectionable-Difference (JOD) units, a distortion map video, and a distogram, which visualizes the distortions over time, separately for each channel and spatial frequency band.

Abstract

ColorVideoVDP is a video and image quality metric that models spatial and temporal aspects of vision for both luminance and color. The metric is built on novel psychophysical models of chromatic spatiotemporal contrast sensitivity and cross-channel contrast masking. It accounts for the viewing conditions, geometric, and photometric characteristics of the display. It was trained to predict common video-streaming distortions (e.g., video compression, rescaling, and transmission errors) and also 8 new distortion types related to AR/VR displays (e.g., light source and waveguide non-uniformities). To address the latter application, we collected our novel XR-Display-Artifact-Video quality dataset (XR-DAVID), comprised of 336 distorted videos. Extensive testing on XR-DAVID, as well as several datasets from the literature, indicate a significant gain in prediction performance compared to existing metrics. ColorVideoVDP opens the doors to many novel applications that require the joint automated spatiotemporal assessment of luminance and color distortions, including video streaming, display specification, and design, visual comparison of results, and perceptually-guided quality optimization.

Materials

• Paper:
  ColorVideoVDP: A visual difference predictor for image, video and display distortions.
  Rafal K. Mantiuk, Param Hanji, Maliha Ashraf, Yuta Asano, Alexandre Chapiro.
  In SIGGRAPH 2024 Technical Papers, Article 129
  [DOI] [paper PDF]
• Supplementary document [PDF]
• Code [Github]
• XR-DAVID dataset
• A webinar explaining ColorVideoVDP and also castleCSF

Results

• Comparison of quality metrics [link]
  This is a detailed report comparing the performance of quality metrics, including additional metrics that could not be included in the main paper. It includes RMSE, PLCC and SROCC performance measures.
• Results for synthetic distortions [link]
  The synthetic test cases reveal how metrics behave given some standard test conditions, such as increasing amount of contrast masking, or Gabors of different frequencies.
• Ablation studies:
  • Ablation on individual components of the model
  • Ablation on multiple masking models considered for the ColorVideoVDP

Related projects

• FovVideoVDP - Foveated Video Visual Difference Predictor
• DPVM - Deep Photometric Visual Metric
• HDR-VDP - A Visual Difference Predictor for High Dynamic Range Images
• castleCSF - A Contrast Sensitivity Function of Color, Area, Spatio-Temporal frequency, Luminance and Eccentricity - models contrast sensitivity in ColorVideoVDP
• ASAP - Active Sampling for Pairwise Comparisons - used to efficiently collect XR-DAVID dataset
• pwcmp - Bayesian pairwise comparison scaling - used to scale XR-DAVID subjective responses