Projects
Evaluating quality metrics through the lenses of psychophysical measurements of low-level vision
Dounia Hammou(1), Yancheng Cai(1), Pavan Madhusudanarao(2), Christos G. Bampis(2), Zhi Li(2) and Rafał K. Mantiuk(1).
(1)University of Cambridge, (2)Netflix, Inc.
Accepted to the 18th International Conference on Quality of Multimedia Experience (QoMEX), IEEE 2026.
Abstract
Image and video quality metrics, such as SSIM, LPIPS, and VMAF, aim to predict perceived visual quality and are often assumed to reflect principles of human vision. However, relatively few metrics explicitly incorporate models of human perception, with most relying on hand-crafted formulas or data-driven training to approximate perceptual alignment. In this paper, we introduce a set of tests for full-reference quality metrics that evaluate their ability to capture key aspects of low-level human vision: contrast sensitivity, contrast masking, and contrast matching. These tests provide an additional framework for assessing both established and newly proposed metrics. We apply the tests to 34 existing quality metrics and highlight patterns in their behavior, including the ability of LPIPS and MS-SSIM to predict contrast masking and the tendency of SSIM to overemphasize high spatial frequencies, which is mitigated in MS-SSIM, and the general inability of metrics to model supra-threshold contrast constancy. Our results demonstrate how these tests can reveal properties of quality metrics that are not easily observed with standard evaluation protocols.
Materials
- Paper:
Evaluating quality metrics through the lenses of psychophysical measurements of low-level vision.
Dounia Hammou, Yancheng Cai, Pavan Madhusudanarao, Christos G. Bampis, Zhi Li, Rafał K. Mantiuk.
In 2026 IEEE International Conference on Quality of Multimedia Experience (QoMEX).
[paper PDF] - Code [Github]
- Complete results for 34 metrics
Related projects
- ColorVideoVDP-ML - ColorVideoVDP-ML: Visual Difference Predictor with a neural regressor for image and video quality prediction
- ColorVideoVDP - A visual difference predictor for image, video and display distortions
- FovVideoVDP - Foveated Video Visual Difference Predictor
- DPVM - Deep Photometric Visual Metric
- HDR-VDP - A Visual Difference Predictor for High Dynamic Range Images
- castleCSF - A Contrast Sensitivity Function of Color, Area, Spatio-Temporal frequency, Luminance and Eccentricity - models contrast sensitivity in ColorVideoVDP
- elaTCSF - elaTCSF: A Temporal Contrast Sensitivity Function for Flicker Detection and Modeling Variable Refresh Rate Flicker
- VFM-HVS - Do computer vision foundation models learn the low-level characteristics of the human visual system?