Objective Evaluation Criteria for Shooting Quality of Stereo Cameras over Short Distance

Stereo cameras are the basic tools used to obtain stereoscopic image pairs, which can lead to truly great image quality. However, some inappropriate shooting conditions may cause discomfort while viewing stereo images. It is therefore considerably necessary to establish the perceptual criteria that can be used to evaluate the shooting quality of stereo cameras. This article proposes objective quality evaluation criteria based on the characteristics of parallel and toed-in camera configurations. Considering the different internal structures and basic shooting principles, this paper focuses on short-distance shooting conditions and establishes assessment criteria for both parallel and toed-in camera configurations. Experimental results show that the proposed evaluation criteria can predict the visual perception of stereoscopic images and effectively evaluate stereoscopic image quality.


Introduction
Owing to the development of stereo technologies, such as three-dimensional (3D) display panels, stereoscopic systems and technologies have been gradually applied to various industries in recent years [1,2,3,4].Stereoscopic image pairs are captured using two cameras that function similar to the eyes of human beings.A major concern is setting the stereo cameras by choosing the appropriate focal length, shooting distance, inter-camera distance, and other shooting parameters.A slight change in these image acquisition systems often causes spatial distortions characteristic for stereoscopic image pairs, such as the puppet-theater effect [5] and the cardboard effect [6].When people observe stereoscopic images displayed on a 3DTV, they may experience visual discomfort and poor 3D effects [7,8,9].To solve these problems, many studies have been carried out [9,10,11,12], and the corresponding evaluation theories and systems have been established based on comparisons between the original stereo images and post processing images [13,14,15].Normally the captured 3D images are thought to be standard and ideal, while the improper parameter settings of the image sources during the capturing process may cause an uncomfortable stereo effect.
To address these issues, a number of researchers have focused on the analysis of stereo camera parameters and proposed some effective works, which can be classified into subjective and objective works.Ijsselsteijn et al., based on subjective experiment, investigated the effects of stereoscopic shooting parameters and display duration on observers' judgements of naturalness and quality of stereoscopic images [16].Later, Xu et al. conducted a subjective test to investigate the effect of the shooting distance and display sizes on perceived stereo images quality, and concluded that the strongest factor on stereo image quality is the distance between the camera setup and the closest photographed object [17].Since subjective method is timeconsuming and impractical for online applications, the objective model has been a fruitful area of work.A straightforward way is to study the objective perceptual shooting quality criteria by considering the factors of individual stereo cameras.Yamanoue et al. reported particularly on the setting of the optical axes of stereo camera and clarified the relationship between the shooting distance and the display position in stereo image space [18].Hasmanda and Riha proposed a method for calculating the best setting for a stereo camera pair used for capturing a scene based on selectable camera parameters (focal length, parallax, inter-camera distance) and object positions in the scene [19].Tsuchida et al. developed a one-shot six-band image-capturing and visualization system that combines multiband and stereo imaging techniques to evaluate the quality of resultant images [20].Furthermore, Ham [21], Yu [22], and others [23,24,25] have also conducted many related studies on camera parameters and shooting quality.What's more, some people focus on human visual perceptions, such as visual fatigue, puppettheater and cardboard effect and so on.Yamanoue et al. focus on puppet-theater and cardboard effects to analyze the setting rules of the optical axes and viewing conditions [26].Kim and Sohn presented a visual fatigue metric which can replace subjective experiments to evaluate the image quality based on studying the effect of shooting distance and intercamera distance on the quality of stereo images [7].
The above approaches studied the relationship between the effect of several parameters, e.g., shooting parameters and human perception.So far, however, there is no effective objective evaluation principle for stereo camera shooting quality.To address this problem, we proposed the objective evaluation criteria for shooting quality of stereo cameras.It is important to acknowledge that the shooting quality is strongly linked to the characteristics of 3D display, such as the size of the display screen, viewing conditions and so on.However, it would be big topic if we added all of the factors into this paper, so the effect of other parameters on shooting quality will be further studied in the future.Here, we take advantage of the individual characteristics of two stereo cameras over short shooting distance, and propose three effective evaluation factors.Besides that, inspired by the five point evaluation scale introduced in this paper, subjective experiments are conducted to give a more precise evaluation result.Through analysis of the mapping result between five point subjective evaluation value and evaluation factor value, the evaluation indexes of each factor are obtained.Finally, all evaluation indexes are linearly integrated into an overall score by considering the importance of each component, and the final evaluation criteria are built.
The remainder of this paper is organized as follows.Section 2 introduces the basic shooting principles; Section 3 presents the objective shooting quality evaluation criteria for stereo cameras; Section 4 describes the evaluation experiments and the establishment of the criteria; Section 5 describes the experimental results and analysis; Section 6 is the discussion and Section 7 concludes the paper.

Basic Shooting Principles
When people view stereoscopic images of natural scenes captured at different shooting distances (measured from the center of the stereo camera setup to the photographed objects), they are interested in different regions, and the information acquired by human eyes is different [17,27,28].Generally speaking, shooting distance can be classified into three types: macro, short distance and long distance.Because of the limited space, here we only focus on short distance.The evaluation criteria for shooting quality of stereo cameras over macro and long distances will be studied in the future.In addition, stereo camera configurations are generally divided into two types: parallel and toed-in [18,29].In [12], by analyzing the features of parallel and toed-in camera configurations and relating the stereo effects to convergence distances, researchers discovered that the shooting distance between the objects and the camera center has a significant effect on the quality of images captured by the camera.This paper aims to establish the evaluation criteria for parallel and toed-in camera configurations for short-distance shooting.Table 1 summarizes the semantic meaning of each parameter used in establishing the evaluation criteria.

Camera parameter Semantic meaning (Physical value)
h (m) The shooting distance d (mm) The inter-camera distance f (mm) The camera focal length p (degree( • )) The viewing angle l (mm) The scene depth L min (mm) Distance between stereo camera center and foreground L max (mm) Distance between stereo camera center and background Tab. 1. Meaning of stereo camera parameters.
In this paper, three short-distance shooting principles, which are the 1/30 rule [24,30,31], binocular overlap percentage [18,19,32,33], and parallax angle theory [23], were considered for identifying the shooting quality evaluation criteria, shown as follows: 1/30 rule: this rule is widely used in stereo photography [24,30,31], it stipulates that the inter-camera distance d should be no more than 1/30 of the shooting distance h from the camera to the first foreground object.
Binocular overlap percentage: this characteristic has a significant influence on the shooting quality of stereo cameras.The magnification of an image on the retina is BE CE [18,19,32,33], as shown in Fig. 1(a) (BE is the width of the captured stereo image, CE is the width of the composite image, which denotes the binocular overlap of stereo cameras).BE  CE plays an important role on the positive and the negative parallax which can affect stereoscopic images quality.For the sake of simplifying the calculation, CE BF is chosen as the evaluation index in this paper to analyze the effect of binocular overlap on shooting quality, where BF is the camera viewing region and p is the viewing angle of the camera.Based on the geometric relations in Fig. 1(a), we have Parallax angle theory: a previous study shows that a certain angular disparity should be maintained to generate comfortable images [33].When the angle disparity equals 70 , the best stereo effect can be achieved without visual discomfort [23].Furthermore, the foreground parallax angle α and the background parallax angle β should not exceed 1 • [27].In Fig. 1(b), we can conclude that: (2)

Objective Shooting Quality Evaluation Criteria
Based on the two types of stereo cameras configurations [18,29] and the above-mentioned basic shooting principles, a frameworks of the proposed criteria is shown in Fig. 2. For the parallel camera configuration, the evaluation of short-distance shooting quality was studied from two aspects: the Modified 1/30 rule and the binocular overlap percentage.Unlike the parallel camera configuration, the optical axes of toed-in camera configurations converge on a single point.The foreground objects have a significant effect on the quality of stereoscopic image pairs.Therefore, the evaluation criteria for the shooting quality of the toedin camera configuration was investigated using the parallax angle theory.

Evaluation Experiment and
Establishment of the Criteria

Observers
Fifty non-professional adult assessors, aged between 20 and 40 years, are invited to conduct the subjective experiment.They all had a normal stereo acuity with more than 0.8 binocular vision tested based on the Titmus Stereo Test in Tianjin EYE Hospital.During the experiments, all participants were asked to wear their usual optical corrections.Double viewpoint images were obtain from the stereo image library at the stereo vision laboratory of the School of Electronic Information Engineering, Tianjin University [34], and this database consists of 600 images generated from   Four types of displays were used in the subjective experiments: a Samsung 2233RZ 22-inch auto stereoscopic display, a Philips 423D6W0200 42-inch multi-view auto stereoscopic display, a Hyundai S465D 46-inch 3D stereoscopic LCD display, and an LG 47CM540-CA 47-inch 3D HDTV display.The subjective evaluation standard was divided into five levels, as shown in Tab. 2.  Before formal experiments, all participants needed to get a 5 minute training trail.They were suggested to watch the training stereo images with different camera parameters in random order for 8 s at a viewing distance which is equal to the height of the screen multiplied by factor 3 as suggested in the ITU-R BT.1438 for HDTV [35], and evaluate the quality of these stereo images by giving an evaluation value from 1 to 5.After the training trail and a short break, all observers started the formal experiment.Here, the stereo images used in the training trials are not the same as that in the formal subjective experiments, and the images used for the formal experiments are obtained based on the second and third column shooting scenes present in Fig. 3.Each stereo image was displayed for 5 s on four 3D displays respectively, followed by a 5 s interval of a 2D mid-gray image with the image index as a grading and relaxation period.The participants were asked to evaluate the stereo images at the viewing range suggested by the instructions for each display.Then the subjective evaluation score of each observer on each stereo image was calculated by averaging these four evaluation results.During the subjective experiment, the participants took a break (10 min in our experiment) after every 30 minutes of quality evaluation.
The mean opinion score (MOS) [36] of each image is computed by averaging fifty subjective scores, and the Stu- dent's t-test [37] is adopted to compute confidence intervals with the significant level being 95%.Then we calculate the range of each influenced factors, and summarize the mapping between each factor and MOS value.
d/h is specified as the evaluation index of the modified 1/30 rule.
First, we set f = 50 mm, h = 2 m, d rangeing from 10 mm to 140 mm with a 10 mm interval, and we used Autodesk 3ds Max and real stereo cameras to capture the corresponding stereoscopic image pairs.Then, changing the value of h with a 2 m interval, as shown in Tab. 3, similar processes were conducted under the same conditions.The range of d/h value was from 1/80 to 1/5, and the subjective experimental results are shown in Fig. 5(a), the red line.Finally, we changed the camera focal length f and carried out similar experiments.The subjective results are shown in Fig. 5(a), and marked by the blue line.
To enrich the experiments, several f were involved in our tests reaching from 40 to 200, each f corresponding to the same range of f * d/h from 1 to 4.5.The subjective results, shown in Fig. 5(b), indicated that the f * d/h was in inverse proportion to the evaluation value.So, here we take the index k p as the final evaluation index, shown in (3).
The mapping between k p and the MOS value was calculated, as shown in Tab. 4

Binocular overlap percentage.
All test images were obtained based on different shooting parameters, which were shown in Tab. 5. Firstly, with fixed value of f and h, we changed the value of d and obtained a number of stereo images.Then we changed the value of h with 2 mm intervals and the range of d to obtain the stereo images.Finally, we changed the value of f and conducted the above two steps.The range of CE BF values here is from 0.85 to 0.9985.Through the subjective experiments, we determined the mapping between CE BF and the MOS value, shown in Tab.6, which indicated that the CE BF value is at most 0.965, a good stereoscopic effect can be obtained.The binocular overlap evaluation criterion is then established, which is used to evaluate the influence of the binocular overlap ratio for short-distance shooting with the parallel camera configuration on the stereo images' quality.

Toed-In Camera Configuration
The principle of parallax angle is an important factor that should be considered in short-distance shooting with the toed-in camera configuration, shown in Fig. 1(b).Studies [27,38] have also concluded that the parallax angle ξ should not exceed 70 • .
To apply the parallax angle principles in the evaluation criteria in this paper, a five-level evaluation criterion was established through experiments as follows.With Autodesk 3ds Max and real camera shooting configurations, a series of stereoscopic image pairs were captured with different shooting parameters, shown in Tab. 7. The parallax angle ξ (shown in ( 2)) of each stereo image pair can be calculated; the range of ξ values in our experiments is from 30 to 250 .The mapping between ξ and the MOS value was studied based on the subjective experimental results and is listed in Tab. 8.This indicated that when the ξ value is at most 72 , a comfortable depth perception can be obtained.Tab. 8. Mapping between ξ and MOS value.

Comprehensive Objective Evaluation Criteria
Currently, the linear weighting method is a very common method used to integrate all the independent individual factors into a global index when evaluating the quality of images [39,40,41].In this paper, let a denote the output value of the modified 1/30 rule factor, b denote the output value of the binocular overlap percentage factor for the parallel camera configuration, and c denote the output value of the parallax angle factor for the toed-in camera configuration under short-distance shooting conditions.These individual factors are independent of each other, and therefore, we can integrate a, b, and c into a global quality score Q using a linear regression equation of the quality indices of each factors, which was defined as (4) where x, y, and z are the above weights of the three regions with respect to the entire image quality, and they are restricted by x + y + z = 1.Combined with the subjective experimental results, Table 9 shows the weights of each individual factor.

Camera type x y z
Parallel camera configuration 0.52 0.48 0 Toed-in camera configuration 0 0 1 Tab. 9. Weights of each individual factor in objective evaluation criteria.

Experimental Results and Analysis
To verify the proposed objective criteria, another thirty non-professional adult assessors, aged between 20 and 40 years, participated in the subjective assessment experiments.They were given the same stereo acuity test as the test mentioned in Section 4.1, and the test results were all higher than 0.8 and indicated that all the assessors have a normal stereo acuity.During the subjective experiment, they were unaware of the experimental hypotheses, and asked to watch the training images to evaluate the stereo images with different camera parameters.One hundred and forty test stereoscopic image pairs were used for the subjective test, which were obtained based on the left column of the shooting scenes shown in Fig. 3, including 2 real scenes and 2 synthetic scenes.In detail, 36 images for each real scene and 34 images for each synthetic scene, and the specific values of the shooting parameters are present in Tab.10.One of the selected scenes is shown in Fig. 6; stereoscopic image pairs with different parameters h, f , and d were captured by parallel and toed-in camera configurations (Fig. 6 Take the stereoscopic image pairs captured by the toed-in camera configuration as an example, as shown in Fig. 6(a)-(c).When d = 200 mm, h = 4 m, l = 500 mm, the quality score Q is 5.0.From the subjective experiments, MOS is 5.0, which indicated that the proposed model can be used to predict the subjective evaluation value.Besides, according to our evaluation formula, shown in (4), Q only determined by the factor of the parallax angle presented in (2), which indicated that the value of ξ is in proportion to the value of d.When d increased, ξ will increased and then lead to a lower evaluation value according to the proposed evaluation criterion for toed-in camera, present in Tab. 8. Here, when d = 600 mm, h = 4 m, l = 500 mm, the objective value calculated by the proposed model (4), Q is 4.And the subjective experimental results present that the MOS is 4.375, which reveals that our proposed criteria are in line with human perception.Similarly, when d = 200 mm, h = 4 m, l = 2500 mm, increased value of l will cause a higher ξ and then the evaluation will be decreased based on our proposed criteria.In fact, the objective Q is decreased to 1.0 and the subjective experimental result MOS is 1.17.The above results proved the validity of the proposed evaluation criteria for toed-in camera.
As for stereoscopic images captured by the parallel camera configuration, shown in Fig. 6(d)-(f).When d = 60 mm, h = 4 m, f = 50 mm, Q is 5.0.Then we set f as 85 mm, Q is 4.0 and the MOS is 3.83.Based on our proposed evaluation criteria shown in (4), the objective evaluation value will be determined by the two factors of "Modified 1/30 rule" and "Binocular overlap percentage".When f changes, the value of "Modified 1/30 rule" k p will be different because of the directly proportional relationship between f and k p .For the existed inversely proportional relationship k p and the evaluation value present in Tab. 4, the increased f will lead a decreased evaluation value which was in line with the results: Q  To further verify the effectiveness of the proposed objective criteria, another four group scenes were chosen to conduct the above experiments, which were obtained based on the right column of the shooting scenes shown in Fig.In this paper, we employed three performance indicators between the predicted objective scores after nonlinear regression and subjective scores to evaluate the proposed metrics, which are [42]: the Pearson Correlation Coefficients (PLCC), Spearman Rank Order Correlation Coefficient (SROCC), and Rooted Mean Squared Error (RMSE).Among these indicators, PLCC and RMSE, shown in ( 5) and ( 6), reflect the predicted accuracy of objective evaluation criteria, and SROCC, shown in (7), is used to assess prediction monotonicity.For a perfect match between the objective and subjective scores, the following should keep valid: PLCC = SROCC = 1 and RMSE = 0. Table 11 presents the overall performance of the proposed quality evaluation criteria.Higher PLCC and SROCC, lower RMSE indicated the good performance of the proposed evaluation criteria and can be applied to evaluate the shooting quality of stereo cameras over short distance.
where n is the number of stereo images, MOS i is the subjective score of the i-th image and Q i is the objective score of the i-th image, MOS i and Q i are the mean value of subjective and objective score, respectively.
where MOS i is the subjective score of the i-th image and Q i is the objective score of the i-th image, respectively.
where d i is the different between the i-th image's ranks in the subjective and objective evaluation.

Discussion
Although we could not compare our work with others as no previous studies have been conducted on shooting quality, however, from Fig. 8 and Tab.11, it can be concluded that the objective evaluation results for the proposed criteria in this paper are in accordance with those of subjective evaluation, which have good correlation with each other.With the combination of subjective experiments and theoretical analysis, the proposed criteria algorithm is applicable for evaluating the shooting quality of stereo cameras.
By the way, the criteria proposed in this paper also can be used to guide stereo photography.Take the factor of "Modified 1/30" as an example, based on the mapping result which was shown in Tab. 4, people can get the appropriate shooting parameters h they want through inputting focal length f and the inter-camera distance d, or get the relationship between shooting distance h and the inter-camera distance d based on the known focal length f .Same as above, some other shooting parameters or relationships can be calculated by the factor of "Binocular overlap percentage" and "Parallax angle theory".Finally, a comfortable stereoscopic image can be obtained based on appropriate shooting parameters.

Conclusion
This paper proposed the objective shooting quality evaluation criteria for stereo cameras short distance shooting.Two types of stereo cameras -parallel and toed-in camera configurations -were considered.Experimental results indicated that the proposed evaluation criteria were consistent with people's subjective perception and can be used effectively to assess stereo camera shooting quality.
The establishment of evaluation criteria in this paper also can be used to guide the stereo photography and be taken as a rational setting principle of shooting parameters for the amateur.Although most parameters are considered in establishing the evaluation criteria, this paper does not cover all the shooting parameters yet.Further researches and the analysis will focus on the effect of other parameters on the shooting quality and establish the more comprehensive evaluation criteria with all shooting parameters.About the Authors . . .

Fig. 2 .
Fig. 2. Schematic diagram about objective evaluation criteria for shooting quality of stereo cameras over short distances.z denotes the camera type, z = 1 means the parallel camera configuration, z = 0 means the toed-in camera configuration.

Fig. 4 .
Fig. 4. Schematic diagram of real stereo cameras.SONY ICX445 CCD, 1/3 , 3.75 µm; Global Shutter; 1288×964 at 30 FPS.(a) Stereo cameras: Inter-camera distance and shooting distance can be changed to get parallel and toed-in camera configurations.(b) Stereo cameras: Intercamera distance and shooting distance can be changed more widely, parallel and toed-in camera configurations can be obtained.(c) Matrix multi-camera configurations.

Fig. 5 .
Fig. 5. (a) Mapping between f * d/h and subjective evaluation when f = 50 mm and f = 85 mm.(b) Schematic diagrams of relation between f * d/h and f .

Tab. 7 .
Experimental parameter values of the parallax angle.
(a)-(c) were captured by the toed-in camera configuration, Fig. 6(d)-(f) were captured by the parallel camera configuration).
= 4.0 and MOS = 3.83.What's more, when d = 100 mm, h = 4 m, f = 85 mm, Q is 3.0, and the result of the subjective experiments showed that the MOS is 3.46.I can be explained based on the proposed evaluation theories, because the output value is inversely proportional to the value of d.The above results indicated the effectiveness of the proposed evaluation criteria for parallel camera.

Fig. 7 .
Fig. 7. Left view of stereoscopic image pairs from real stereo cameras and Autodesk 3ds Max shooting: (a)-(c) shortdistance shooting with toed-in camera configuration; (b)-(d) short-distance shooting with parallel camera configuration.

3
Scene type Image numbers with parallel camera Image numbers with toed-in camera parameters ( f , h, d) parameters ( f , h, d, l) Real camera 36 36 f : 50-85, h: 2-6, d: 10-120 f : 50-85, h: 2-6, d: 10-120, l: 100-3100 Autodesk 3ds Max 34 34 f : 50-85, h: 2-6, d: 40-260, l: 100-3100 f : 50-85, h: 2-6, d: 40-260, l: 100-3100 Tab. 10.Experimental parameter values.and the specific configurations are the same as that present in Tab.10.Two groups consisted of real stereo cameras shooting 3D scene pictures (shown in Fig.7(a)-(b)), and the rest were Autodesk 3ds Max scene pictures (shown in Fig.7(c)-(d)).By changing the values of the shooting parameters h, d, f , l, and L min , another one hundred and forty stereoscopic image pairs were chosen to test the validity of the proposed model.Fig.8shows the linear correlation between the objective evaluation result Q and the subjective evaluation MOS values.The consistency between the proposed criteria and the subjective evaluation is clearly identified in the figure.

1 Z
Meaning of other parameters except Tab.-The stereo camera type; d /h -The index of 1/30 rule; α -The foreground parallax angle; β -The background parallax angle; ξ -The parallax angle; γ -The convergence angle; CE BF -The index of binocular overlap ratio; k p -The ultimate evaluation index of parallel camera configurations; Q -The global quality score.
, by combining the subjective experimental results and the range of k p values.The values indicate that when the k p value is at most 1.88, a comfortable depth perception can be obtained.
Jiachen YANG was born in July 1979, Chinese Communist Party members, Liaoning province Anshan city.He received his Ph.D. degree in Communication and Information Engineering from the Tianjin University in 2009, Postdoctoral in signal and information processing, Institute of TV and Image Information, in TUTC, and a Postdoctoral in Tianjin Yaan Polytron Technologies Inc, Tianjin, China.He is an associate professor at the Tianjin University, School of Electronic and Information Engineering.From 2014 to 2015 to carry out exchange visits in British Loughborough University work.Research interests include image processing, wireless communication system, financial software development, hardware development, gas detection.Yun LIU was born in June 1987, the Anhui province Suzhou city.She Graduated from Changchun Institute of Technology in 2011, with bachelor's degree in Engineering.Now she is a Ph.D. student with School of Electronic Information Engineering, Tianjin University, Tianjin, China.Her research interests include stereo capture, displaying, coding, transmission, feature detection and description.Rongrong CHU was born in May 1988, Shandong province, Liaocheng city.She graduated from Qiingdao University in 2012 with bachelor's degree in Engineering.She is a M.S. student with School of Electronic Information Engineering, Tianjin University.Her research interests include quality evaluation, vision research, pattern recognition, stereo camera, stereo image capture, coding, transmission, feature detection and description.