PlenoptiSign: An optical design tool for plenoptic imaging

Plenoptic imaging enables a light-field to be captured by a single monocular objective lens and an array of micro lenses attached to an image sensor. Metric distances of the light-field's depth planes remain unapparent prior to acquisition. Recent research showed that sampled depth locations rely on the parameters of the system's optical components. This paper presents PlenoptiSign, which implements these findings as a Python software package to help assist in an experimental or prototyping stage of a plenoptic system.


Motivation and significance
Plenoptic cameras gain increasing attention from the scientific community and pave their way into experimental three-dimensional (3-D) medical imaging [1,2,3,4]. A limitation of light-field cameras is that the maximum distance between two viewpoint positions, the so-called baseline, is confined to the extent of the entrance pupil [5,6]. With triangulation, small baselines are mapped to depth planes close to the imaging device making them suitable for medical purposes such as in a microscope [1,2] or an otoscope [3].
When capturing depth with a plenoptic system, it is essential to place available light-field depth planes on the targets of interest. Hence, it is a key task in plenoptic data acquisition to choose suitable specifications for the micro lenses, the objective lens and the sensor early to save time and costs at the conceptual design stage of a prototype.
PlenoptiSign enables a priori depth plane localization in plenoptic cameras for stereo matching from sub-aperture image disparities [5,7,6] as well as computational refocusing via shift-and-integration [8]. PlenoptiSign can be used to pinpoint object distances in light-field images rendered by our complementary open-source software PlenoptiCam [9]. The underlying physical model of PlenoptiSign was devised and experimentally proven in studies by Hahne et al. [6,10] and applies to the Lytro-type setup [11] at this stage.
Given an experiment involving plenoptic image acquisition, a researcher may want to simulate the influence of the Micro Lens Array (MLA), the objective lens and its focus as well as optical zoom settings to investigate the depth resolution performance. With this software, the user is able to optimize an experimental setup as required. In its current state, the tool can be called from a Graphical User Interface (GUI), a web-server capable of handling the Common Gateway Interface (CGI) or from the Command Line Interface (CLI) where a user is asked for all input parameters.
It is the goal of this paper to raise awareness of the light-field model's capabilities when implemented as a software tool. In Section 2, we sketch the software architecture while turning the focus onto the ray function solver by means of linear algebra to complement previous publications where this has not been explained in much detail. This is followed by usage instructions and exemplary result presentations in Section 3 after which the potential influence on future applications is discussed in Section 4.

Architecture
Plenoptic camera parameters are passed to PlenoptiSign using either the CLI (cli script.py), a tkinter GUI (gui app.py) or a CGI web-server (cgi script.py). Based on Python's ddt package, a unit test was added to support potential future development. An overview of the code structure is depicted in Fig. 1. An object of the light-field geometry estimator is instantiated from a user interface by calling mainclass.py, which inherits mixin classes, namely refo.py, tria.py, plt refo.py and plt tria.py.

Functionalities
The core task of this software is to find intersections of light-field ray pairs. Since its implementation has not been entirely covered before [6,10], it is demonstrated hereafter. A concise form of finding an intersection of two ray functions is by solving a System of Linear Equations (SLE) given as where A and b make up ray functions and x contains unknowns representing locations of ray intersections in the y − z plane. Using two ray functions, we obtain a unique solution to the SLE by the algebraic inverse A −1 where A ∈ R 2×2 is invertible. To cover cases of overdetermined SLEs, a more generic solution is provided by the Moore-Penrose pseudo-inverse A + with ⊺ denoting the matrix transpose. Algebraic computations are implemented by means of the NumPy module in the solver.py file. For plenoptic triangulation, matrices A and b may be defined according to the notation provided in [6] with Eq. (18) as a ray function, which writes with U i, j as y-intercepts at the object-side principal plane and q i, j as chosen chief ray slopes in object space. Here, i is an arbitrary micro image pixel index and j represents a micro lens index in one direction. As seen in Fig. 2, we find the separation of two light-field viewpoints, so-called baseline B G , by two intersecting linear ray functions [6] B where our reference viewpoint depends on i and is separated by a scalar G of light-field viewpoints. After rearranging the two functions, we write which can be solved using the SLE given by where z here corresponds to the longitudinal entrance pupil position A ′′ H 1U [6].  Similarly, we can trace a pair of light-field rays to find an object plane to which a plenoptic image is focused using the shift and integration algorithm [8,10]. In accordance with the model presented in [10], an image-side ray function f c+i,j (z) is given by where m c+i,j denotes an image-side ray slope and s j the respective micro lens position. A ray is chosen by i = −c and j = a(M − 1)/2 with its counterpart ray having negative indices. Applying this to Eq. 10 and rearranging yields −m c+i,+j × z = s +j (11) −m c−i,−j × z = s −j (12) which can be represented in matrix form by where y is the vertical position and d a ′ is an elongation of the image distance which is mapped to respective refocusing distance d a using the thin lens equation while taking thick lenses and their principal planes into account [10].

Usage
After downloading PlenoptiSign from the online repository, installation is made possible with the setuptools module which can be run by \$ python plenoptisign/setup.py install in the download directory, provided that Python is available and granted necessary privileges. As indicated in Fig. 1, PlenoptiSign can be accessed from three different interfaces, namely web-based CGI, a GUI and bash command line. Executing PlenoptiSign from the command line is done by where −p sets the plot option with rays and depth planes being depicted as seen in Figs. 3 and 4. Starting the tkinter-based GUI application is either done by running a bundled executable file or by \$ plenoptisign −g For more information on available commands, use the help option −h.

Results
Light-field geometry results are provided as text values or graphical plots with two types of views. An exemplary cross-sectional plot is shown in Fig. 3.  In addition, the results can be displayed in 3-D, as depicted in Fig. 4, with triangulation planes Z (G,∆x) for disparities ∆x. For details on the design trends, scientific notations and deviations due to paraxial approximation, we may refer to our preceding publications [6,10] for further reading.

Impact
The presented framework provides answers and generic solutions to a question a researcher raised in a forum [12] and received recommendations from peers, indicating demands and potentials for future projects. Research directions that may benefit from this software include experimental photography [11], cinematography [13] or scientific fields ranging from volumetric fluid particle flow [14] to the many kinds of clinical studies [1,2,3,4].
It may be argued that depth plane localization can be done via extrinsic calibration as traditionally employed in the field of stereo vision. Despite its widespread use, this method cannot be performed prior to the camera manufacturing process, but solely after the fact. The same applies to a calibration of light-field metrics via least-squares fitting [15]. Due to the heuristic nature of this approach, zoom lens and optical focus settings as well as temperature fluctuations introduce degrees of freedom to the fitted curve that make calibration sophisticated. After all, it is essential to comprehend and exploit the underlying optical model in the development stage of a plenoptic camera for optimal performance. This part would turn out to be costly with the aforementioned alternatives.
A probably deviant attempt of the presented refocusing distance algorithm has been used in the auspicious Lytro cameras [11]. However, Lytro's method remains undisclosed up to the present day, which once more underlines the need and practical use of our open-source software.

Conclusions
Thanks to provided software, the light-field geometry of a plenoptic camera can be accurately predicted with ease of use. This is of crucial interest in the design stage of a prototype where distance range and depth plane density need to be determined and optimized in advance. This tool has been made available for web-servers, a graphical user-interface and the command line tool. It is the first open-source software to do so and may lay the groundwork for future research on plenoptic imaging in the medical field. Future development may lead towards the implementation of an automatic parameter optimization or an extension of a focused plenoptic camera model.