Metric comparison

The results in the plot below are reported for the testing data only. Only UPIQ and XR-DAVID were used for training (unless stated otherwise). The training set was also used to fit the JOD regression. The detailed reports (if present) show the entire dataset (both training and testing parts).



Table shows the results for test data

Metric/Variant RMSE PLCC SROCC
UPIQ XR-DAVID LIVEHDR KADID-10k LIVEVQA Average(1) UPIQ XR-DAVID LIVEHDR KADID-10k LIVEVQA Average(2) UPIQ XR-DAVID LIVEHDR KADID-10k LIVEVQA Average(3)
ColorVideoVDP → 0.6073 0.5944 0.3169 0.9262 0.8585 0.6607 0.9 0.8782 0.9357 0.8595 0.756 0.8773 0.8759 0.8913 0.9244 0.8605 0.7412 0.8694
IW-SSIM → 0.7247 0.7648 0.5393 0.9814 0.8747 0.777 0.8576 0.7964 0.7935 0.8402 0.7451 0.8102 0.8287 0.7726 0.7973 0.85 0.7318 0.7998
DSS → 0.6865 0.885 0.6889 0.9576 0.8131 0.8062 0.869 0.7314 0.8213 0.8487 0.7847 0.8165 0.8265 0.6878 0.8842 0.8604 0.7669 0.816
HaarPSI → 0.6223 1.007 0.526 0.9148 0.9801 0.81 0.895 0.7613 0.761 0.8629 0.6645 0.8041 0.8838 0.7399 0.7856 0.8849 0.6593 0.8071
GMSD → 0.6825 0.9896 0.6922 0.9647 0.8982 0.8454 0.883 0.7111 0.8576 0.8463 0.7287 0.8166 0.8259 0.6442 0.8719 0.8474 0.7209 0.796
VMAF (v0.6.3) → 0.7407 0.9531 0.4873 1.261 0.8747 0.8634 0.8441 0.7458 0.8158 0.7172 0.7451 0.7784 0.8108 0.6496 0.8767 0.721 0.7395 0.7718
FSIMc → 0.7423 0.7998 1.015 0.985 0.9023 0.889 0.8631 0.8441 0.5786 0.8389 0.7258 0.7886 0.8357 0.8057 0.5721 0.8538 0.727 0.775
FovVideoVDP 1.2 → 0.6917 1.111 0.6792 1.089 0.9105 0.8963 0.8678 0.7047 0.6407 0.7988 0.7198 0.7584 0.8338 0.6918 0.6901 0.8015 0.7062 0.7515
MDSI → 0.5473 0.8902 1.226 0.8833 0.9975 0.9088 0.9203 0.8556 0.7237 0.8729 0.6495 0.8276 0.8769 0.8079 0.75 0.8853 0.6432 0.8089
IQT → 0.7185 0.82 0.7471 1.161 1.155 0.9203 0.8573 0.7539 0.7396 0.767 0.4743 0.7392 0.8163 0.6827 0.7497 0.7509 0.4649 0.7095
VSI → 0.7552 0.9921 0.9999 0.9055 0.9533 0.9212 0.8512 0.8268 0.676 0.8658 0.6868 0.7945 0.8535 0.7659 0.695 0.878 0.6804 0.7882
MS-SSIM → 0.841 1.09 0.6606 1.11 0.9939 0.9391 0.7936 0.5882 0.7356 0.79 0.6524 0.7209 0.7885 0.5597 0.7371 0.7949 0.6299 0.7133
HDR-VDP-3 (v3.0.7) → 0.6993 1.206 0.7016 1.221 0.9593 0.9575 0.864 0.4614 0.6335 0.738 0.6819 0.6998 0.8202 0.4843 0.6142 0.741 0.6705 0.6822
VIF → 0.7442 0.9467 0.6691 1.359 1.114 0.9665 0.8455 0.7187 0.5961 0.6606 0.5285 0.6873 0.7084 0.5804 0.6361 0.6501 0.531 0.625
SSIM (luma) → 0.9215 1.222 0.6626 1.42 1.097 1.065 0.7457 0.4252 0.684 0.6199 0.5485 0.6163 0.7464 0.422 0.7204 0.613 0.5443 0.6228
LPIPS (VGG) → 0.7896 1.404 0.7406 1.254 1.19 1.076 0.847 0.3269 0.5923 0.7214 0.4204 0.62 0.7671 0.4074 0.6362 0.7201 0.4249 0.6118
PSNR (RGB) → 0.9874 0.9576 1.131 1.33 1.117 1.105 0.7038 0.6432 0.6772 0.6783 0.5245 0.6493 0.7091 0.62 0.7348 0.6757 0.498 0.6547
DeltaE ITP spatial → 1.041 1.313 0.5535 1.631 1.029 1.113 0.6572 0.35 0.756 0.4345 0.6202 0.5825 0.6588 0.3291 0.749 0.3988 0.5906 0.5654
CIEDE2000 → 1.076 1.218 0.6636 1.635 1.147 1.148 0.624 0.3487 0.6813 0.4299 0.4854 0.5251 0.6015 0.3076 0.6571 0.4239 0.4581 0.5003
DeltaE ITP → 1.144 1.169 0.6756 1.58 1.198 1.154 0.5582 0.3546 0.8061 0.4888 0.4067 0.5487 0.5492 0.3228 0.8293 0.4807 0.3921 0.5478
FUNQUE → 1.165 1.414 0.6209 1.714 0.8894 1.161 0.5435 0.1668 0.735 0.3543 0.735 0.5409 0.5468 0.1458 0.7159 0.3913 0.7244 0.5356
CIEDE2000 spatial → 0.977 1.312 0.9298 1.608 1.044 1.174 0.7042 0.3314 0.6834 0.4596 0.6051 0.5721 0.7113 0.3586 0.6865 0.4214 0.5772 0.5668
HDR-FLIP → 1.066 1.282 0.6868 1.638 1.204 1.175 0.6402 0.2511 0.5632 0.4263 0.3968 0.4665 0.6163 0.2193 0.5522 0.4333 0.3667 0.4482
NIQE → 1.201 1.362 0.9197 1.699 1.312 1.299 0.5001 0.1647 0.1414 0.3452 5.929e-16 0.2389 0.4501 0.2183 0.2602 0.3383
STRRED → 1.068 1.601 1.884 1.466 0.7719 1.358 0.6536 0.3521 0.6396 0.5862 0.8085 0.6293 0.6734 0.5243 0.702 0.5802 0.8052 0.6692
PIQE → 1.185 1.473 1.574 1.768 1.287 1.457 0.5195 0.163 0.5352 0.2151 0.1917 0.3364 0.45 0.1819 0.6202 0.2372 0.1822 0.3487
BRISQUE → 1.184 1.503 1.637 1.744 1.312 1.476 0.5587 0.004915 0.1672 0.2689 0.005037 0.2137 0.4954 0.09818 0.2512 0.2986 -0.0642 0.2245

Table shows the results for all data

Metric/Variant RMSE PLCC SROCC
UPIQ XR-DAVID LIVEHDR KADID-10k LIVEVQA Average(1) UPIQ XR-DAVID LIVEHDR KADID-10k LIVEVQA Average(2) UPIQ XR-DAVID LIVEHDR KADID-10k LIVEVQA Average(3)
ColorVideoVDP → 0.67 0.7768 0.6022 0.9262 0.8585 0.7667 0.8956 0.7843 0.8639 0.8595 0.756 0.8392 0.8876 0.8328 0.9071 0.8605 0.7412 0.8548
IW-SSIM → 0.755 0.8775 0.5601 0.9814 0.8747 0.8097 0.8654 0.7189 0.8279 0.8402 0.7451 0.8065 0.8352 0.6844 0.855 0.85 0.7318 0.8009
DSS → 0.7286 0.9588 0.701 0.9576 0.8131 0.8318 0.8752 0.6401 0.6707 0.8487 0.7847 0.7806 0.833 0.6371 0.8801 0.8604 0.7669 0.81
HaarPSI → 0.6793 1.006 0.5517 0.9148 0.9801 0.8265 0.8924 0.688 0.8032 0.8629 0.6645 0.7998 0.8828 0.654 0.8436 0.8849 0.6593 0.8072
GMSD → 0.7365 1.113 0.6538 0.9647 0.8982 0.8732 0.8756 0.6367 0.8212 0.8463 0.7287 0.796 0.8284 0.6036 0.8726 0.8474 0.7209 0.7919
VMAF (v0.6.3) → 0.8293 0.9432 0.6423 1.261 0.8747 0.9102 0.8348 0.7051 0.7841 0.7172 0.7451 0.7617 0.8078 0.666 0.8792 0.721 0.7395 0.7744
FSIMc → 0.746 1.088 0.9615 0.985 0.9023 0.9366 0.879 0.7933 0.7454 0.8389 0.7258 0.8042 0.8618 0.7706 0.7395 0.8538 0.727 0.7979
FovVideoVDP 1.2 → 0.7851 1.153 0.9061 1.089 0.9105 0.9688 0.8537 0.6647 0.6986 0.7988 0.7198 0.7563 0.8362 0.6516 0.7841 0.8015 0.7062 0.7637
MDSI → 0.6312 1.131 1.229 0.8833 0.9975 0.9744 0.9082 0.7742 0.7102 0.8729 0.6495 0.8044 0.8755 0.7626 0.8036 0.8853 0.6432 0.8097
IQT → 0.7829 1.082 0.7677 1.161 1.155 0.9897 0.8543 0.5673 0.8249 0.767 0.4743 0.7274 0.8151 0.5084 0.8432 0.7509 0.4649 0.7077
VSI → 0.7881 1.208 0.9298 0.9055 0.9533 0.9569 0.8575 0.7537 0.7772 0.8658 0.6868 0.7977 0.8699 0.714 0.7889 0.878 0.6804 0.8
MS-SSIM → 0.8941 1.204 0.6734 1.11 0.9939 0.9751 0.8047 0.5263 0.7631 0.79 0.6524 0.7212 0.8061 0.5003 0.7926 0.7949 0.6299 0.723
HDR-VDP-3 (v3.0.7) → 0.804 1.268 0.7215 1.221 0.9593 0.9948 0.8455 0.4499 0.7111 0.738 0.6819 0.7057 0.8081 0.5117 0.763 0.741 0.6705 0.7118
VIF → 0.8287 1.097 0.8287 1.359 1.114 1.045 0.835 0.6196 0.5853 0.6606 0.5285 0.6619 0.7078 0.5653 0.7262 0.6501 0.531 0.6423
SSIM (luma) → 1.094 1.278 0.7052 1.42 1.097 1.119 0.6882 0.4244 0.6968 0.6199 0.5485 0.6043 0.7156 0.4353 0.7718 0.613 0.5443 0.6308
LPIPS (VGG) → 0.8676 1.452 0.7083 1.254 1.19 1.094 0.8278 0.3374 0.6877 0.7214 0.4204 0.6333 0.7596 0.4261 0.7054 0.7201 0.4249 0.6279
PSNR (RGB) → 1.123 1.085 1.124 1.33 1.117 1.156 0.6686 0.5911 0.5945 0.6783 0.5245 0.6145 0.6891 0.5743 0.7494 0.6757 0.498 0.6457
DeltaE ITP spatial → 1.175 1.293 0.6051 1.631 1.029 1.146 0.6263 0.3698 0.7516 0.4345 0.6202 0.5771 0.6269 0.373 0.7763 0.3988 0.5906 0.5737
CIEDE2000 → 1.252 1.231 0.6671 1.635 1.147 1.186 0.5558 0.3787 0.7141 0.4299 0.4854 0.5242 0.5569 0.3486 0.7114 0.4239 0.4581 0.5123
DeltaE ITP → 1.322 1.134 0.7116 1.58 1.198 1.189 0.478 0.4036 0.6534 0.4888 0.4067 0.4923 0.4808 0.3413 0.7225 0.4807 0.3921 0.4976
FUNQUE → 1.318 1.452 0.627 1.714 0.8894 1.2 0.4988 0.1496 0.7628 0.3543 0.735 0.5386 0.5086 -0.005313 0.7662 0.3913 0.7244 0.5222
CIEDE2000 spatial → 1.132 1.309 0.8788 1.608 1.044 1.195 0.6591 0.3512 0.6855 0.4596 0.6051 0.5643 0.6678 0.4061 0.7469 0.4214 0.5772 0.5799
HDR-FLIP → 1.253 1.27 0.7308 1.638 1.204 1.219 0.5555 0.2943 0.6034 0.4263 0.3968 0.4626 0.5559 0.2432 0.6583 0.4333 0.3667 0.4642
NIQE → 1.371 1.219 0.9447 1.699 1.312 1.309 0.4141 0.27 0.1778 0.3452 5.929e-16 0.2462 0.3895 0.2814 0.2646 0.3383
STRRED → 1.126 1.486 1.817 1.466 0.7719 1.333 0.6777 0.4291 0.6454 0.5862 0.8085 0.6464 0.6974 0.5672 0.7847 0.5802 0.8052 0.7003
PIQE → 1.314 1.341 1.465 1.768 1.287 1.435 0.4985 0.2002 0.4508 0.2151 0.1917 0.3183 0.4334 0.2063 0.5337 0.2372 0.1822 0.3265
BRISQUE → 1.302 1.39 1.551 1.744 1.312 1.46 0.5505 0.1579 0.1371 0.2689 0.005037 0.2349 0.4899 0.1437 0.1682 0.2986 -0.0642 0.2154

(1)Average RMSE is computed as an average of RMSEs of individual datasets so that each dataset has the same influence on the average RMSE regardless of the number of conditions it contains.

(2-3)Average correlation coeffcients are computed by averaging individual correlation coefficients (in the Fisher's transorm space).


ColorVideoVDP

Test: RMSE = 0.6607 PLCC = 0.8773 SROCC = 0.8694

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



IW-SSIM

Test: RMSE = 0.777 PLCC = 0.8102 SROCC = 0.7998

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



DSS

Test: RMSE = 0.8062 PLCC = 0.8165 SROCC = 0.816

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



HaarPSI

Test: RMSE = 0.81 PLCC = 0.8041 SROCC = 0.8071

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



GMSD

Test: RMSE = 0.8454 PLCC = 0.8166 SROCC = 0.796

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



VMAF (v0.6.3)

Test: RMSE = 0.8634 PLCC = 0.7784 SROCC = 0.7718

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



FSIMc

Test: RMSE = 0.889 PLCC = 0.7886 SROCC = 0.775

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



FovVideoVDP 1.2

Test: RMSE = 0.8963 PLCC = 0.7584 SROCC = 0.7515

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



MDSI

Test: RMSE = 0.9088 PLCC = 0.8276 SROCC = 0.8089

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



IQT

Test: RMSE = 0.9203 PLCC = 0.7392 SROCC = 0.7095

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



VSI

Test: RMSE = 0.9212 PLCC = 0.7945 SROCC = 0.7882

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



MS-SSIM

Test: RMSE = 0.9391 PLCC = 0.7209 SROCC = 0.7133

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



HDR-VDP-3 (v3.0.7)

Test: RMSE = 0.9575 PLCC = 0.6998 SROCC = 0.6822

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



VIF

Test: RMSE = 0.9665 PLCC = 0.6873 SROCC = 0.625

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



SSIM (luma)

Test: RMSE = 1.065 PLCC = 0.6163 SROCC = 0.6228

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



LPIPS (VGG)

Test: RMSE = 1.076 PLCC = 0.62 SROCC = 0.6118

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



PSNR (RGB)

Test: RMSE = 1.105 PLCC = 0.6493 SROCC = 0.6547

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



DeltaE ITP spatial

Test: RMSE = 1.113 PLCC = 0.5825 SROCC = 0.5654

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



CIEDE2000

Test: RMSE = 1.148 PLCC = 0.5251 SROCC = 0.5003

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



DeltaE ITP

Test: RMSE = 1.154 PLCC = 0.5487 SROCC = 0.5478

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



FUNQUE

Test: RMSE = 1.161 PLCC = 0.5409 SROCC = 0.5356

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



CIEDE2000 spatial

Test: RMSE = 1.174 PLCC = 0.5721 SROCC = 0.5668

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



HDR-FLIP

Test: RMSE = 1.175 PLCC = 0.4665 SROCC = 0.4482

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



NIQE

Test: RMSE = 1.299 PLCC = 0.2389 SROCC = NaN

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



STRRED

Test: RMSE = 1.358 PLCC = 0.6293 SROCC = 0.6692

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



PIQE

Test: RMSE = 1.457 PLCC = 0.3364 SROCC = 0.3487

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA



BRISQUE

Test: RMSE = 1.476 PLCC = 0.2137 SROCC = 0.2245

Detailed report for:UPIQXR-DAVIDLIVEHDRKADID-10kLIVEVQA