Datasets

Dataset for the Comparison of Subjective Quality Assessment Methods for 3D Graphics

We provide a dataset of 80 3D meshes with vertex colors, generated from 5 source models and impaired with various color and geometry distortions. This dataset is associated with subjective quality scores collected from two psycho-visual experiments that compare the performance of three of the most prominent subjective quality assessment methodologies, with and without explicit references (ACR-HR, DSIS and SAMVIQ). The experiments were conducted in Virtual Reality (VR). This dataset allows to draw interesting conclusions about the importance of the presence of explicit references to assess the quality of 3D graphics, the most suitable methodology in terms of accuracy and time effort, and last but not least, the minimum number of participants required for such experiments.



This dataset was created at the LIRIS lab, Université de Lyon. It is associated with the following reference. Please cite it, if you use the dataset.

Yana Nehmé, Jean-Philippe Farrugia, Florent Dupont, Patrick Le Callet, Guillaume Lavoué, Comparison of Subjective Methods for Quality Assessment of 3D Graphics in Virtual Reality, ACM Transactions on Applied Perception (TAP), Vol. 18, No. 1, Article 2, 23 pages, 2021.

Acknowledgement: The Chameleon, Fish and Samurai models are derived from models produced by Rumpelstiltskin, Davita Kurniawan, and David Antalek, respectively.

3D Meshes with Vertex Colors Dataset

We provide a large dataset of 480 animated meshes with vertex color information, and associated with perceived quality judgments. The stimuli were generated from 5 source models subjected to geometry and color distortions. Each stimulus was associated with 6 hypothetical rendering trajectories (HRTs): combinations of 3 viewpoints and 2 animations. A total of 11520 quality judgments (24 per stimulus) were acquired in a subjective experiment conducted in virtual reality (VR) and based on the double stimulus impairment scale (DSIS) method.



This dataset was created at the LIRIS, Université de Lyon, and LS2N, Université de Nantes, labs. It is associated with the following reference. Please cite it, if you use the dataset.

Yana Nehmé, Florent Dupont, Jean-Philippe Farrugia, Patrick Le Callet, Guillaume Lavoué, Visual Quality of 3D Meshes with Diffuse Colors in Virtual Reality: Subjective and Objective Evaluation, IEEE Transactions on Visualization and Computer Graphics (TVCG), Vol. 27, No. 3, 2021.

Acknowledgement: The Chameleon, Fish and Samurai models are derived from models produced by Rumpelstiltskin, Davita Kurniawan, and David Antalek, respectively.

Textured 3D Meshes Dataset

We provide a large-scale dataset of textured meshes with over 343k stimuli generated from 55 source models quantitatively characterized in terms of geometric, color, and semantic complexity to ensure their diversity. The dataset covers a wide range of compression-based distortions applied on the geometry, texture mapping and texture image. The database can be used to train no-reference quality metrics and develop rate-distortion models for meshes.
From the established dataset, we carefully selected a challenging subset of 3000 stimuli that we annotated in a large-scale subjective experiment in crowdsourcing based on the double stimulus impairment scale (DSIS) method. Over 148k quality scores were collected from 4513 participants. To the best of our knowledge, it is the largest quality assessment dataset of textured meshes associated with subjective scores and Mean Opinion Scores (MOS) to date. This database is valuable for training and benchmarking quality metrics.
Quality scores of the remaining stimuli in the dataset (i.e. those not involved in the subjective experiment) were predicted (Pseudo-MOS) using a quality metric called Graphics-LPIPS, based on deep learning, trained and tested on the subset of annotated stimuli.



This dataset was created at the LIRIS lab, Université de Lyon. It is associated with the following reference. Please cite it, if you use the dataset.

Yana Nehmé, Johanna Delanoy, Florent Dupont, Jean-Philippe Farrugia, Patrick Le Callet, Guillaume Lavoué. Textured Mesh Quality Assessment: Large-Scale Dataset and Deep Learning-based Quality Metric, ACM Transactions on Graphics, 2023.