Metrics comparison

The results in the plot below are reported for the testing data only. Only UPIQ and XR-DAVID were used for training (unless stated otherwise). The training set was also used to fit the JOD regression. The detailed reports (if present) show the entire dataset (both training and testing parts).



Table shows the results for test data

Metric/Variant SROCC PLCC RMSE
GAIM240 CGVQD BVI-HFR BVI-HFR-ext NerfBenchmark XR-DAVID LIVEVQA Average(1) GAIM240 CGVQD BVI-HFR BVI-HFR-ext NerfBenchmark XR-DAVID LIVEVQA Average(2) GAIM240 CGVQD BVI-HFR BVI-HFR-ext NerfBenchmark XR-DAVID LIVEVQA Average(3)
ColorVideoVDP-ML saliency_gaim240_cgvqd → 0.6696 0.4615 0.6258 0.5595 0.8253 0.8039 0.6544 0.6752 0.5442 0.2913 0.456 0.3863 0.8746 0.7672 0.6639 0.6112 0.9311 0.7788 1.84 1.754 2.595 0.7967 1.013 1.387
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_dataset_weighting SPEM → 0.312 0.4505 0.5964 0.5558 0.842 0.754 0.5365 0.6072 0.2576 0.4318 0.5645 0.486 0.8849 0.7677 0.5142 0.6023 1.04 0.6427 2.134 2.285 2.035 0.7956 1.252 1.455
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd SPEM → 0.2852 0.4505 0.5673 0.5244 0.8373 0.7465 0.4972 0.5885 0.2467 0.4076 0.5614 0.4811 0.8653 0.757 0.4811 0.5833 1.041 0.6523 2.02 2.154 2.197 0.8112 1.308 1.455
ColorVideoVDP-ML saliency_gaim240_cgvqd SPEM → 0.4808 0.4835 0.6585 0.574 0.7197 0.7311 0.3189 0.5825 0.3525 0.4417 0.6277 0.5086 0.6894 0.7321 0.3549 0.5464 1.011 0.608 1.475 1.476 3.267 0.872 1.523 1.462
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_dataset_weighting → 0.4215 0.4011 0.6711 0.6098 0.8439 0.8022 0.6799 0.6612 0.3592 0.3641 0.4763 0.4388 0.9419 0.8154 0.634 0.6471 1.013 0.6749 2.769 2.673 1.382 0.7306 1.088 1.476
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd → 0.3514 0.3462 0.6353 0.5832 0.8439 0.7797 0.6523 0.6312 0.331 0.2524 0.4878 0.4427 0.9309 0.8045 0.6098 0.6218 1.021 0.8349 2.536 2.482 1.684 0.7472 1.087 1.484
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_finetuned → 0.7818 0.5934 0.3694 0.3369 0.8602 0.6633 0.5986 0.6356 0.7229 0.5603 0.3507 0.3106 0.8799 0.71 0.521 0.62 0.8849 0.9069 2.689 2.735 1.799 0.9889 1.413 1.631
ColorVideoVDP-ML saliency_gaim240_cgvqd_finetuned SPEM → 0.6838 0.4945 0.416 0.3402 0.7664 0.4074 0.5706 0.5439 0.6416 0.4225 0.3969 0.3356 0.7352 0.3353 0.5687 0.5074 0.9351 0.8353 1.997 1.958 2.865 1.763 1.174 1.647
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_finetuned SPEM → 0.7641 0.544 0.4241 0.3658 0.8848 0.5331 0.5541 0.6194 0.6636 0.3639 0.4692 0.3964 0.8067 0.4832 0.5476 0.554 0.8832 0.9452 2.367 2.596 2.351 1.77 1.279 1.742
ColorVideoVDP ML-saliency → 0.6511 0.2363 0.4836 0.4359 0.757 0.5888 0.5896 0.5526 0.4259 0.09766 0.3142 0.2855 0.7527 0.567 0.5389 0.4504 1.185 2.328 3.164 3.2 2.715 1.427 1.634 2.236
ColorVideoVDP-ML saliency_gaim240_cgvqd_finetuned → 0.8957 0.5879 0.2225 0.1332 0.7122 0.4743 0.5757 0.5715 0.8735 0.6053 0.0008832 -0.03165 0.6716 0.465 0.5749 0.5154 0.7295 0.8657 4.478 4.517 3.269 1.332 1.258 2.35

Table shows the results for all data

Metric/Variant SROCC PLCC RMSE
GAIM240 CGVQD BVI-HFR BVI-HFR-ext NerfBenchmark XR-DAVID LIVEVQA Average(1) GAIM240 CGVQD BVI-HFR BVI-HFR-ext NerfBenchmark XR-DAVID LIVEVQA Average(2) GAIM240 CGVQD BVI-HFR BVI-HFR-ext NerfBenchmark XR-DAVID LIVEVQA Average(3)
ColorVideoVDP-ML saliency_gaim240_cgvqd → 0.6361 0.3534 0.6258 0.5595 0.8038 0.7117 0.6544 0.6369 0.6177 0.3828 0.456 0.3863 0.7689 0.6293 0.6639 0.5747 0.8522 0.9729 1.84 1.754 2.008 0.9932 1.013 1.348
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_dataset_weighting SPEM → 0.5878 0.2802 0.5964 0.5558 0.8361 0.7672 0.5365 0.622 0.582 0.2966 0.5645 0.486 0.8702 0.7162 0.5142 0.6083 0.8802 1.051 2.134 2.285 1.428 0.8597 1.252 1.413
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd SPEM → 0.579 0.2807 0.5673 0.5244 0.8376 0.7623 0.4972 0.6076 0.5771 0.2845 0.5614 0.4811 0.8591 0.7154 0.4811 0.5973 0.8832 1.062 2.02 2.154 1.482 0.8587 1.308 1.395
ColorVideoVDP-ML saliency_gaim240_cgvqd SPEM → 0.5545 0.293 0.6585 0.574 0.7134 0.6232 0.3189 0.5497 0.556 0.2835 0.6277 0.5086 0.6609 0.6028 0.3549 0.5247 0.8976 1.026 1.475 1.476 2.294 1.002 1.523 1.385
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_dataset_weighting → 0.5667 0.3561 0.6711 0.6098 0.8359 0.7922 0.6799 0.668 0.5404 0.3585 0.4763 0.4388 0.9248 0.7645 0.634 0.6432 0.941 1.013 2.769 2.673 1.157 0.7895 1.088 1.49
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd → 0.5358 0.3455 0.6353 0.5832 0.832 0.7693 0.6523 0.6451 0.5304 0.3016 0.4878 0.4427 0.9154 0.7324 0.6098 0.6233 0.934 1.074 2.536 2.482 1.274 0.8337 1.087 1.46
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_finetuned → 0.663 0.4752 0.3694 0.3369 0.8506 0.7826 0.5986 0.6182 0.6868 0.4739 0.3507 0.3106 0.8956 0.799 0.521 0.6281 0.7862 1.018 2.689 2.735 1.413 0.7425 1.413 1.542
ColorVideoVDP-ML saliency_gaim240_cgvqd_finetuned SPEM → 0.5939 0.405 0.416 0.3402 0.7094 0.5459 0.5706 0.5224 0.634 0.4386 0.3969 0.3356 0.6525 0.4481 0.5687 0.5054 0.8361 1.084 1.997 1.958 2.198 1.328 1.174 1.511
ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_finetuned SPEM → 0.6848 0.435 0.4241 0.3658 0.8695 0.7098 0.5541 0.6118 0.7131 0.4551 0.4692 0.3964 0.8765 0.6209 0.5476 0.6147 0.7585 1.246 2.367 2.596 1.626 1.122 1.279 1.571
ColorVideoVDP ML-saliency → 0.6346 0.235 0.4836 0.4359 0.6106 0.701 0.5896 0.5411 0.5633 0.2259 0.3142 0.2855 0.5047 0.6536 0.5389 0.4538 1.042 1.88 3.164 3.2 2.544 1.139 1.634 2.086
ColorVideoVDP-ML saliency_gaim240_cgvqd_finetuned → 0.6981 0.5421 0.2225 0.1332 0.6581 0.5017 0.5757 0.4981 0.7156 0.566 0.0008832 -0.03165 0.472 0.4744 0.5749 0.4269 0.7613 1.056 4.478 4.517 2.62 1.297 1.258 2.284

(1)Average RMSE is computed as an average of RMSEs of individual datasets so that each dataset has the same influence on the average RMSE regardless of the number of conditions it contains.

(2-3)Average correlation coeffcients are computed by averaging individual correlation coefficients (in the Fisher's transorm space).


ColorVideoVDP-ML saliency_gaim240_cgvqd

Test: SROCC = 0.6752 PLCC = 0.6112 RMSE = 1.387

ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_dataset_weighting SPEM

Test: SROCC = 0.6072 PLCC = 0.6023 RMSE = 1.455

ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd SPEM

Test: SROCC = 0.5885 PLCC = 0.5833 RMSE = 1.455

ColorVideoVDP-ML saliency_gaim240_cgvqd SPEM

Test: SROCC = 0.5825 PLCC = 0.5464 RMSE = 1.462

ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_dataset_weighting

Test: SROCC = 0.6612 PLCC = 0.6471 RMSE = 1.476

ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd

Test: SROCC = 0.6312 PLCC = 0.6218 RMSE = 1.484

ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_finetuned

Test: SROCC = 0.6356 PLCC = 0.62 RMSE = 1.631

ColorVideoVDP-ML saliency_gaim240_cgvqd_finetuned SPEM

Test: SROCC = 0.5439 PLCC = 0.5074 RMSE = 1.647

ColorVideoVDP-ML saliency_gaim240_xrdavid_nerf_cgvqd_finetuned SPEM

Test: SROCC = 0.6194 PLCC = 0.554 RMSE = 1.742

ColorVideoVDP ML-saliency

Test: SROCC = 0.5526 PLCC = 0.4504 RMSE = 2.236

ColorVideoVDP-ML saliency_gaim240_cgvqd_finetuned

Test: SROCC = 0.5715 PLCC = 0.5154 RMSE = 2.35