Repository logo
 

Psychometric scaling of TID2013 dataset

Accepted version
Peer-reviewed

Type

Conference Object

Change log

Authors

Mikhailiuk, Aliaksei  ORCID logo  https://orcid.org/0000-0001-9757-6644
Perez-Ortiz, M 

Abstract

TID2013 is a subjective image quality assessment dataset with a wide range of distortion types and over 3000 images. The dataset has proven to be a challenging test for objective quality metrics. The dataset mean opinion scores were obtained by collecting pairwise comparison judgments using the Swiss tournament system, and averaging votes of observers. However, this approach differs from the usual analysis of multiple pairwise comparisons, which involves psychometric scaling of the comparison data using either Thurstone or Bradley-Terry mod- els. In this paper we investigate how quality scores change when they are computed using such psychometric scaling instead of averaging vote counts. In order to properly scale TID2013 quality scores, we conduct four additional experiments of two different types, which we found necessary to produce a common quality scale: comparisons with reference images, and cross-content comparisons. We demonstrate on a fifth validation experiment that the two additional types of comparisons are necessary and in conjunction with psychometric scaling improve the consistency of quality scores, especially across images depicting different contents.

Description

Keywords

46 Information and Computing Sciences, 4603 Computer Vision and Multimedia Computation, Clinical Research

Journal Title

2018 10th International Conference on Quality of Multimedia Experience, QoMEX 2018

Conference Name

2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX)

Journal ISSN

Volume Title

Publisher

IEEE
Sponsorship
Engineering and Physical Sciences Research Council (EP/P007902/1)
European Research Council (725253)