University of Birmingham Depth perception on fundus images using a single-channel stereomicroscopy

Typical fundus photography produces a two-dimensional image. This makes it di±cult to observe the microvascular and neural abnormalities, because the depth of the image is missing. To provide depth appreciation, we develop a single-channel stereoscopic fundus video imaging system based on a rotating refractor. With respect to the pupil center, the rotating refractor laterally displaces the optical path and the illumination. This allows standard monocular fundus cameras to generate stereo-parallax and image disparity through sequential image acquisition. We opti-mize our imaging system, characterize the stereo-base, and image an eyeball model and a rabbit eye. When virtual realities are considered, our imaging system can be a simple yet e±cient technique to provide depth perception in a virtual space that allows users to perceive abnor-malities in the eye fundus.


Introduction
Binocular human vision a distance of 50-75 mm provides three-dimensional (3D) depth perception in object observation. The stereovision is based on the brain translation of two slightly di®erent views. 1 In two-dimensional (2D) imaging, both eyes see the same images that are captured by a camera, although the brain can still achieve a level of depth perception due to pattern recognition such as shadows and size. 2 Stereoscopic imaging includes two slightly di®erent views for each eye at the same time and provides stereovision via various stereo-display methods.
Stereoscopic imaging methods can be greatly bene¯cial and sometimes necessary in surgical microscopes, stereo-fundus imaging, and other extra-and intra-ocular purposes in ophthalmology. Stereoscopic imaging in ophthalmology can enhance the depth perception of a physician during visualization in diagnostic applications as well as in therapeutic operations. 3 The interior structure of the eye is relativity easy to access by optical imaging modalities that are applied to distinguish various interior eye diseases by providing biometric data such as length and shape. The topography of the blood vessels around the optic nerve head is one of the most important diagnostic parameters, particularly in glaucoma diagnosis. 4 In this case, the imaging target is not easily accessible, and 2D imaging may not be fully informative. Therefore, optical approaches such as stereoscopic imaging, confocal scanning laser tomography, and optical coherence tomography (OCT) can be used to obtain useful geometric and structural data. 4 Stereoscopic images of the fundus can be used to assess those blood vessels in glaucoma or other medical diagnostic procedures using popular fundus camera approaches owing to the wide availability of fundus cameras. 4 Recently, we have introduced a single-channel stereoscopic video imaging modality based on a rotating refractor (RR) that can be customized for various applications. 5,6 In the imaging modality, two slightly di®erent view angles are obtained through the rotating refractor, which is mounted on a step motor and placed inside an optical channel, mainly at the back focal plane (BFP) of objective lens. Therefore, the stereo-angle becomes a function of the refractive index and rotation angle of the rotating refractor. In this study, by optimizing the optical system and the controller unit to achieve a precise real-time control, we investigate the feasibility of a single-channel stereoscopic fundus video imaging modality based on the rotating refractor (SFIM-RR). During imaging of the eye interior and retina, the eye lens and cornea function as a part of the imaging optics. 7 Therefore, in contrast to the previous location of the rotating refractor, 5,6 which is placed behind the objective lens, the rotating refractor is placed in front of the objective lens at the Fourier plane within a working distance. In this new design, the rotating refractor is used to laterally displace the illumination and the optical paths relative to the pupil center to generate stereoparallax for image disparity.
Currently, two main approaches are available in stereo-image pairs of the fundus: (i) sequential image acquisition using a single fundus camera at di®erent positions and (ii) simultaneous image acquisition using two stereo-fundus cameras. 7 The simultaneous imaging approach captures two stereoparallax images at the same time. The sequential imaging approach captures stereo-parallax images sequentially that minimize equipment requirement. 8 We provide a rapid sequential automatic image acquisition that can be easily developed with a standard monocular fundus camera, namely the single-channel stereoscopic fundus video imaging modality based on rotating refractor. A ray-tracing simulation (WinLens3D Basic; 1.2.6; 2014-07) is performed to¯nd a preliminary con¯guration of the optical setup. Mathematical analysis is used to investigate the amount of stereo-base (SB), which is the minimum depth di®erence that can generate image disparity and cause 3D e®ect on the stereoscopic depth perception [ Fig. 1(a)], and the converging angle between the left and right views as a function of the rotation angle and size of the rotating refractor. Then, the simulation results are evaluated by experiment using an inverse optical pathway. The minimum required torque of the rotating refractor is calculated as a function of the imaging speed, motion pro¯le, and rotation angle. A simple human-eye surgery condition is simulated using an eyeball model to investigate the improvement in depth perception.  A beam splitter is employed to separate illumination and observation paths and illuminate the imaging target. The pulsed LED illumination is focused on the beam splitter, and L obj images the light beam into the pupil of the subject to illuminate the fundus. An aperture stop (Ø1-12 mm) is placed immediately behind the beam splitter in the observation beam path, followed by the relay lenses (L r1 , f ¼ 40 mm and L r2 , f ¼ 30 mm). The second real intermediate image is formed by L r1 in front of L r2 , which is incorporated with a camera lens (L c ) to form an image on the camera sensor. The aperture stop is imaged by L obj into the pupil plane of the eye model. It is used to control the exit pupil of the observation beam path and thus the optical resolution of the fundus. A combination of a linear polarizer in the illumination beam path, which is Depth perception on fundus images using a single-channel stereomicroscopy 2150012-3 used to polarize the illumination beam, and another linear polarizer as an analyzer, which is placed in the observation beam path, suppresses unwanted re°ection and light scattering.

Rotating refractor
The proposed rotating refractor is made from a high refractive index glass (SF10) with a refractive index of 1.73, a polished window of 8 Â 12 mm 2 , and a length of 20 mm. The rotating refractor is placed between L obj and the subject eyeball within the working distance, at the BFP of eyeball lens. The rotating refractor is capable of controlling the position of the illumination by changing its orientation  magni¯cation, and image and working distances) of the SFIM-RR system that is mathematically simulated and characterized using a WinLens3D Basic software (written for: QIOPTIQ, Germany; by: # Dr. Geo® Adamas, 1994-2016). In the simulation, we use the 20-mm length of rotating refractor. A combination of positive lenses with a focal length of 22 mm is considered as an eyeball lens [ Fig. 1(a)]. This eyeball structure is included in the simulation to calculate the¯eld of view and magni¯cation. Wē nd that the¯eld of view of the double-sized eyeball model is approximately 11 . The¯eld of view increases as the working distance and the focal length of the objective lens decrease. An increase in the size of the human eyeball results in shorter focal length of the eye lens.
The operation of imaging camera is synchronized with the rotating refractor and the light source that trigger the image acquisition. Figure 1(c) shows the digital signals to control the illumination and the motor (e.g., the direction and the step of rotating refractor). An appropriate step motor and a motor drive with su±cient holding torque are employed to  provide smooth motion pro¯le with high resolution for rotating the refractor. Through the digital I/O ports, the motor drive, LED, and CMOS camera are reconnected to a National Instruments (NI) PCI data acquisition (DAQ) system. The NI DAQ is programmed via LabVIEW for synchronization and real-time control of the motion pro¯le, pulsed illumination, and triggering of the camera. Figure 1(b) shows a photograph of the prototype SFIM-RR, which includes an eyeball model. The pulses generated by the controller, including the signals for camera triggering and illumination, motor-drive direction, and motor-drive steps, are evaluated using an oscilloscope to demonstrate their accuracy and precision.

Relationship between rotation angle, stereo-base, and converging angle
In the SFIM-RR, two stereo-parallax images are obtained by refracting the optical path rotating the rotating refractor, which induce di®erent perspectives. As shown in Figs. 1(a) and 2(a), the imaging axis is laterally shifted after passing through the rotating refractor due to the refraction. Capturing an image at each end of the rotation arc of the rotating refractor leads to the generation of two virtual optic channels with slightly di®erent imaging axes. The distance between the two virtual optic channels is a function of the refractive index, size, and rotation angle of the rotating refractor. In the fundus camera, the diameter of the aperture stop controls the diameter of the exit pupil of the observation beam path (d o ), which should be su±ciently small to¯t in the subject pupil and to maximize the resolution. Because the e®ective area of the rotating refractor window is larger than d o , the observation beam is not cropped with the edges of the rotating refractor. Thus, the distance between two virtual imaging axes (i.e., the SB) can be derived as follows: where L RR is the length, is the rotation angle, and n is the refractive index, respectively, of the rotating refractor (SF10, 1.73). To calculate the SB, the rotating refractor displacement should be imaged on the plane of the exit pupil, which is slightly lower than the rotating refractor physical displacement. To measure the distance between two virtual imaging axes, the illumination part is precisely replaced with a narrow laser beam such that the laser beam represents the position of the imaging axis. Then, the displacement of the center of the laser beam due to the refractor rotation is measured under di®erent RR rotation angles. The calculated stereo-base is compared with the measurements of the laser-beam displacement. In addition, the angular di®erence between the left and right views is theoretically analyzed to estimate the condition for depth perception of a human vision. To simplify the condition, the cornea and the eyeball lens are considered as positive lenses with a 24-mm focal length. The converging angle () of the left and right laser beams at the focal point is calculated as a function of SB for a given rotating refractor (L ¼ 20 mm).
can represent the converging angle of the imaging axes and thus may be used to suggest better display condition for the most comfortable observation. Figure 3(b) shows the measurements and calculations of stereo-base with di®erent lengths of rotating refractor (14, 20, and 26 mm). In the experiment, we employ a laser beam, rotate the RR according to the orientation , and measure the lateral displacement of laser beam. The measured stereo-base linearly increases as a function of the rotation angle of the rotating refractor. Figure 3(c) shows the converging angle of imaging axis that corresponds to di®erent stereo-bases.

Characterization of image disparity from stereo-base
The minimum stereoscopic depth perception (ÁL min ) that is perceived on the fundus can be expressed as follows 8,9 : where f 0 eye is the image-side focal length of the eyeball [ Fig. 3(a)] and is the wavelength of light. As expressed by Eq. (1), stereo-base is a function of the RR rotation angle (). ÁL min is calculated as a function of for the three di®erent d o values at a wavelength of 550 nm.
ÁL min is calculated as a function of the rotation angle of the rotating refractor at ¼ 550 nm for three di®erent exit pupil diameters d o , as shown in Fig. 4. The ability of SFIM-RR to resolve detail Depth perception on fundus images using a single-channel stereomicroscopy over the depth (i.e., depth resolution) improves as the rotation angle or the pupil diameter increases. This implies that the depth resolution can be improved up to where the rotation angle and pupil diameter reach their structural maxima.

Characterization of imaging speed and motor torque
The motor must be correctly selected to achieve smooth and stable RR rotation during image acquisition. The imaging speed, rotation quality, and even the image quality are a®ected by an appropriate torque generator for the rotating refractor based on the desired motion pro¯le. After designing the desired motion pro¯le and the rotating refractor, a two-phase hybrid step motor (Applied Motion; HT17-278) is chosen to provide su±cient holding torque and rotation speed. Because the torque is a function of the rotation speed, the required torque and minimum rotation time for each image are estimated for various rotation speeds considering the RR rotation angles that ranged from 6 to 30 [ Fig. 5(a)]. The torque can be calculated by where I is the moment of inertia of the rotating refractor and A is the angular acceleration, which is expressed as where t is the rotation time.
The torque for controlling the rotating refractor is calculated as a function of the rotation speed for various rotation angles of the rotating refractor [ Fig. 5(b)]. In this calculation, the e®ects from gravitation and air resistance are not considered. Figure 5(c) shows the time to take pair of images as a function of the rotation speed of the rotating refractor for di®erent rotation angles. The rotation time becomes the time delay between the left and right image measurements. The results show that the interval between the left and right images is less than 20 ms. This is su±ciently short for realizing a semi-simultaneous stereo-imaging.

Fundus imaging of eyeball model and rabbit eye
To investigate the feasibility of SFIM-RR, a table tennis ball with a diameter of 40 mm is used as an eyeball model to simulate a simple condition of a human eyeball. A section of the ball body with approximately 22-mm diameter is removed to create a window for the lens. A Ø1" achromatic doublets lens (f ¼ 45 mm) is used to simulate the cornea and eyeball lens to generate a clear image from an object at in¯nity on the internal posterior wall. A iris diaphragm is used to limit the entrance size of the eyeball model and to control the diameter and size of the pupil. Also, we use a rabbit eye to obtain in-vivo stereoscopic fundus image. This study was approved by the institutional review board at Wonju Severance Christian Hospital. We enhance the image quality and resolution of our imaging system by adding polarizer and by optimizing illumination: because the illumination and observation beams pass through the same optical pathway, there may exist some re°ection and back-scattering from the optical components that can a®ect the image quality. In conventional fundus imaging system, this problem is solved using a ring aperture in front of the light source and a pinhole mirror instead of a beam splitter. Thus, the annular light beam can be totally separate from the observation beam path.  Depth perception on fundus images using a single-channel stereomicroscopy

2150012-7
The stereovision is created as the rotating refractor modi¯es optical paths for two optical channels. As the refractor rotates, the observation beam path is laterally shifted to the left-and right-hand sides of the optical axis at the exit pupil. When the camera captures such shifted images, it makes image disparity between the left and right images. Then, the stereovision e®ect is generated. The depth di®erence greater than ÁL min causes image disparity and provides stereoscopic images in the optimal condition. For depth perception, ÁL min should be adjusted to match with the depth resolution of eye (i.e., the ability to resolve detail over the depth). The minimum stereoscopic depth perception is determined by the lens and sensor resolution. Also, the maximum values of SB, d o , and ÁL min are determined by the diameter of the patient pupil D pp . As shown in Fig. 1(a), D pp must be almost equal to SBþd i and larger than SBþd o . Therefore, for the stereoscopic imaging of the fundus, we dilate the subject pupil using mydriatic eye drop. Figure 6 show the stereo-parallax generated from our fundus imaging system. It clearly shows the image disparity. For such images, considering the pupil diameter and eyeball size, the refractor has the rotation angles of 14 and 32 for the rabbit-eye and eyeball model imaging, respectively. At these rotation angles, the converging angles of imaging axis () are almost 6 , which is highly recommended as the comfortable and optimized value for binocular human vision. Here in the eyeball model, considering d o ¼ 3 mm, ÁL min is about 0.32 mm. The table tennis eyeball phantom is approximately two times bigger than the standard actual human eyeball. Owing to the transparency of the table tennis ball, the eyeball phantom is capable of simple transscleral illumination, in addition to the common illumination beam paths. Although this type of illumination is mainly used for special cases, it is particularly useful in our experiment because of the lack of separation of the illumination and observation beam paths. In the case of in-vivo images, the unwanted re°ection and back-scattering from optical components and the rabbit eye and the lack of trans-scleral illumination signi¯cantly reduced the image quality. However, these limitations can easily be resolved in the commercial version using a sophisticated powerful light source and the separation of the illumination and observation beam paths. In addition, in our study, a smaller eyeball and the di±culty in stable positioning of the rabbit eye compared with the human or eyeball phantom lead to mediocre results.

Conclusion
We develop a stereoscopic fundus imaging system. The real-time control of the rotating re°ector allows stereoscopic fundus imaging with a single monocular fundus camera. For fundus imaging, we characterize the relation between the rotation angle, stereo-base, and converging angle of rotating re-°e ctor. The results show that when su±cient torque and power are applied, the SFIM-RR can provide sequential images for stereovision. One can construct such optical system by using standard monocular fundus camera and stereoscopic surgical video microscope. When augmented and virtual realities are considered, our imaging modality is a simple yet e±cient technique to make 3D depth perception of eye model in a virtual space that allows users to access every single point in one's retina using computer.

Con°ict of Interest
The authors declare no con°ict of interest.