Theory and Implementation of Integral Illumination

Integral imaging, i.e., the use of lenticular optics to display stereoscopic/multiscopic images, is now being used in an array of products including glasses-free 3D displays. This paper describes integral illumination, an adaptation of integral imaging where fine-grained control of plenoptic light fields is used to realize new forms of programmable lighting. Relying on a combination of an imaging apparatus and custom lenticular optics, integral illumination devices can produce high-fidelity illusions of real and imagined light sources (e.g., spotlight, chandelier), replicating their illumination effects. Such devices have potential uses as ambient lighting fixtures, photography/videography equipment, components of artistic installations, etc. The paper will provide a general overview of integral illumination, describing its basic principles, hardware configuration, control mechanism, range of capabilities, and theoretical/practical limitations. We will also present a sample implementation of a working integral illumination device, describe its engineering details, report performance measurements, and discuss possibilities for future improvements and extensions.


I. INTRODUCTION
LED lighting has seen rapid adoption in recent years, quickly replacing incandescent bulbs and other conventional illumination technologies. In addition to offering sizable reductions in energy use, LEDs-owing to the ease by which they can be controlled through digital circuitry-have led to increased usage of programmable lighting systems, which enable lighting parameters such as color temperature and brightness to be dynamically adjusted through software. Such systems have a wide range of use cases, from ambient lighting in residential homes to decorative illuminations on building facades.
In this paper we describe integral illumination, a technical principle that realizes a new class of programmable lighting devices that are capable of simulating the illumination effects of various real and imagined light sources. Hypothetically, in a room fully furnished with an array of integral illumination devices, occupants will have the power to dynamically place, move, and delete virtual light sources within the space to suit their needs; a chandelier may be made to suddenly pop up in the middle of the room, only to be replaced later by a row of skylights through which an ''artificial sun'' can be seen.
The associate editor coordinating the review of this manuscript and approving it for publication was Eunil Park .
Integral illumination is a modern adaptation of the centuryold principle of integral imaging [1], i.e., the use of lenticular optics to control plenoptic light fields [2], [3] and create images that change appearance based on viewing direction. Although the two principles are similar, the difference in objective imposes separate sets of technical requirements, particularly with regards to optical design.
The paper offers a high-level overview of the foundations of integral illumination, covering topics such as hardware configuration, optical design, control mechanism, theoretical/ practical limitations, and potential applications. We will also present a prototype implementation of an integral illumination device, built using a combination of LED array, medical-grade LCD panel, and custom-designed lenticular sheet. We will discuss its design details and engineering challenges, describe a custom software for intuitive lighting design/control, and report performance measurements.
The main contributions of this paper are as follows: • Presents a comprehensive overview of integral illumination, which should allow readers to form clear understandings of the principle, its potential and limitations • Describes a sample hardware implementation of an integral illumination device, with sufficient detail to assist readers in replicating or building upon our work VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ • As part of the said implementation, describes a custom operating software which includes a script-based system that allows intuitive creation of new lighting effects • Discusses potential applications of integral illumination, illustrating the roles highly programmable lighting can play in real-world scenarios In light of the current trend in which digital technology is increasingly becoming entwined with the built environment, we view integral illumination as the next logical step in the evolution of artificial lighting. Ultimately, in a manner akin to how various paper-based media (posters, billboards, etc.) are rapidly being replaced by digital displays, we expect integral illumination devices to supersede a sizable share of artificial lighting hardware in the near-to mid-term future.

II. RELATED WORK
Integral imaging (Figure 1), first described by Lippmann [1] in 1908, refers to the use of 1D or 2D lenticular optics (e.g., a horizontal row of semi-cylindrical lenses or a grid of circular, convex lenses) to display plenoptic light fields. The lenticular sheet is placed on top of a 2D graphical pattern to form a two-layered panel; due to refraction by the lenses, different parts of the graphical pattern will be visible to viewers looking at the panel from different directions. By strategically designing the underlying graphical pattern, this principle can be used to produce stereoscopic (i.e., showing two different images simultaneously, intended to be seen by the left and right eye, respectively) or multiscopic (i.e., showing a large number of images simultaneously, allowing viewers to inspect displayed scenes from multiple angles) effects.
A classic real-world use of integral imaging can be found in lenticular postcards, often sold as toys or souvenirs. Beginning in the late 20th century, the same principle has been used to create glasses-free 3D displays (light field displays); this is achieved by replacing the static underlying graphical pattern with an electronic imaging apparatus, such as a digital display [4] or projector(s) tuned to focus on the rear surface of the lenticular sheet [5]. (Although tangential to our work described in this paper, integral imaging has also inspired the development of light field cameras [6], [7] that record, instead of display, plenoptic light fields.)

A. 3D DISPLAYS
Integral imaging is one of the most practical, and widely used methods for building glasses-free 3D displays. Such displays can either be stereoscopic or multiscopic as described earlier, and depending on lens geometry, images can be made variable along only one axis or along both x and y axes. As with electronic displays in general, such 3D displays have steadily improved over the years in aspects such as screen resolution, color accuracy, viewing angle, energy efficiency, etc.
Lenticular sheets are typically made of optical glass, or polymers with high refractive indices such as polycarbonate (PC), acrylic (PMMA), etc. Recently, the use of transparent 3D printable materials [8], [9] has become common especially in prototyping contexts, opening up new possibilities for non-planar integral imaging [10]. Efforts are also being made [11], [12] to create lenticular sheets using deformable, liquid-filled lenses that allow dynamic control of optical properties. Non-standard lens configurations have been investigated as well; Berkel [13] proposed the use of slanted lenticulars to reduce moiré-like visual artifacts, and Hirsch et al. [14] explored the addition of an extra optical sheet to expand viewing angles.
An assortment of techniques beyond integral imaging have been used to build 3D displays, such as parallax barriers [15], time-multiplexed LCD stacks [16], rotating mirrors [17], holography [18], laser-induced plasma emission [19], and photophoretic optical trapping [20]. HMDs (Head-Mounted Displays) [21] may also be considered a type of 3D display, albeit with a different, wearable form factor. The full list of proposed 3D display technologies is too long to include here; several comprehensive (albeit somewhat dated) surveys are available on the topic [22], [23].

B. PROGRAMMABLE LIGHTING
Electronically-controlled lighting has been in use long before LED lighting became viable, most often for entertainment or advertising purposes. One well-known example is the (now demolished) Westinghouse Sign in Pittsburgh, USA. Today programmable lighting systems are ubiquitous [24], [25], and used in various locales including homes, offices, film studios, and theaters. In urban areas, they are gaining traction as parts of media facades [26], i.e., building exteriors furnished with decorative lighting. Lately, concerns regarding light pollution and their effects on public health [27] and local ecosystems [28], [29] have drawn attention to the use of programmable lighting as public illumination, e.g., street lamps.
Most programmable lighting systems in actual use offer relatively simple controls of on/off patterns, hue, brightness, etc. In the early days of ubiquitous computing [30] research, the idea of using digital projectors as programmable lighting appliances had drawn attention [31]. While the idea failed to find much use outside of a few niche domains, the motivation to introduce finer-grained control to programmable lighting aligns closely with that of integral illumination.
There have been a modest number of efforts to appropriate integral imaging for programmable lighting [32]- [34]. Early efforts have focused on select use cases (e.g., smallscale photography, microscopy) where illumination-specific modifications are not required and standard integral imaging hardware can be directly repurposed as lighting equipment. In our past work [35]- [37], we have introduced the term integral illumination and explored the topic in more detail, identifying its unique constraints and technical challenges. To our knowledge, this paper stands as the first work to give a comprehensive overview of integral illumination, and present a viable design of a general-purpose integral illumination device backed by a functioning implementation.

C. INTERACTIVE ENVIRONMENTS
Under umbrella terms such as smart cites [38] and smart homes [39], numerous efforts are being made to integrate new interactive technologies as part of the built environment. Examples include mid-and large-scale digital displays, smart windows with tunable optical properties [40], shapeshifting furniture/buildings [41], [42], and spatial active noise control systems [43]. Programmable lighting (including integral illumination) can be considered a part of this trend.
Conceptually, such efforts have a long history in computer science. In as early as the 1960s, Sutherland [44] discussed the notion of the ultimate display, i.e., a room made out of programmable matter [45], [46] in which arbitrary objects can be made to appear/disappear at will. Although a convincing implementation of such technology still remains elusive, the vision has proved influential; over the years, researchers have submitted a series of new concepts regarding interactive, programmable environments [47]- [49], each updating Sutherland's original vision to reflect new technical advances. Figure 2 illustrates the standard hardware setup of an integral illumination device. Similar to a 3D display based on integral imaging, the setup comprises two main components: 1) a lenticular sheet, and 2) an imaging apparatus. As will be described later, in our prototype implementation we use an LCD panel / backlight combination as the imaging apparatus; this can be replaced by alternatives such as an LED matrix display or projector(s). (A range of devices capable of modulating light rays over a horizontal plane can be used here.) The general principle of light control is identical to that of integral imaging. Let us consider a 2D lenticular sheet made up of ideal lenslets (i.e., convex cylindrical lenses that serve as the sheet's constituent optical units) with no aberrations, and a Cartesian coordinate system local to each lenslet whose origin lies on the lenslet's axis and whose xy-plane is parallel to its bottom surface. A light ray that enters the lenslet's bottom surface at point p = (x, y, 0) will, independent of its direction of incidence α, emerge from the convex top surface in a direction given by the following unit vectord:

III. INTEGRAL ILLUMINATION
where: Here, h denotes the lenslet's height, n denotes the refractive index of lens material, and n air denotes the refractive index of air (1.00029 at standard temperature/pressure). VOLUME 10, 2022 By solving the inverse of the above, we can obtain the point of incidence p that will produce an outgoing light ray in directiond. This can be used to compute the pixel pattern (to be rendered by the imaging apparatus) that will result in rays being emitted from each lenslet in a desired set of directions. Here, if we assume the pixel pitch of the imaging apparatus to be sufficiently small relative to the lenslet diameter, and also the lenslet diameter to be sufficiently small relative to the size of the lenticular sheet, we have in effect a device capable of emitting rays in an arbitrary set of directionsD u,v (albeit subject to directional limitations imposed by the lenslet's finite viewing angle) from each point (u, v) on the lenticular sheet. Such a device can simulate the presence of arbitrary light source(s) placed in a virtual 3D space behind the lenticular sheet. Simulating a light source involves first determining the set of rays that will pass through the lenticular sheet if the light source actually existed, then displaying a pixel pattern on the imaging apparatus that will replicate this set of rays to the best of the device's capabilities.
Note that a light ray passing through a lenslet will experience some loss of intensity due to attenuation, and depending on p and α, may altogether fail to exit the lenslet due to total internal reflection, etc. The ratio between the respective intensities of incident and emergent rays can be described as follows, as a function of p, α, and a set of variables V that represent lens properties (e.g., geometry, material).
In practice, there are a number of additional factors that limit the performance of an integral illumination device. For example, pixels on the imaging apparatus typically have nonnegligible size relative to lenslet diameter, and thus lighting a pixel will cause rays to enter a lenslet not at a single point p but across a two-dimensional area. The size of the lenslets will also be non-negligible relative to that of the lenticular sheet, and their designs will never be ideal, i.e., there will be varying degrees of optical aberrations. We will discuss such limitations later in more detail.
In the above discussion, we assumed the use of a singleconvex, cylindrical lenslet as the basic unit of the lenticular sheet. This is not the only viable optical design and a range of alternatives exist; in some cases involving esoteric lenslet designs, obtaining pixel patterns may require more complex operations than what is shown above, e.g., deconvolution.

A. GHOSTS
Lenticular sheets are usually fabricated as a single, continuous slab of optically clear material, not as a grid of mutually isolated lenslets. This leads to crosstalk; a ray that enters one lenslet can freely travel to adjacent lenslets and subsequently emerge from their top surfaces. Due to this, lighting a pixel at point p results in rays emerging from multiple lenslets, at multiple angles of emergence, breaking the basic premise of integral illumination we have described earlier. As crosstalk stems from the way lenticular sheets are fabricated, the phenomenon is not unique to integral illumination and most integral imaging setups exhibit the same issue. For example, this is the reason why lenticular postcards appear to show repeating patterns as the viewer shifts viewpoints. With integral imaging, however, crosstalk has little actual impact on functionality and can be safely ignored.
With integral illumination, the effects of crosstalk manifest as a series of rogue illumination effects, or ghosts ( Figure 3). Here, let us consider a scenario where we program an integral illumination device to simulate a virtual flashlight, positioned just behind the lenticular sheet so that a narrow beam of light is emitted from the device. If no steps are taken to counteract ghosts, in addition to the desired, forward beam, a group of unwanted, extra beams will be emitted in multiple directions. (The pattern in which the extra beams will emerge mirrors the layout of lenslets on the lenticular sheet, e.g., rectilinear, hexagonal.) While the extra beams will be dimmer compared to the forward beam (as imaging apparatuses typically exhibit non-uniform radiation patterns, and also a sizable percentage of crosstalking rays will fail to exit lenslets), they must be suppressed for faithful replication of light sources.
All rogue, crosstalking rays will emerge from the lenticular sheet at angles greater than the lenslet's viewing angle. This explains why ghosts can be ignored with integral imaging; as long as the viewer is not looking at the image from extreme angles, rogue rays have no effect on the viewer's experience. By the same logic, ghosts can be ignored even with integral illumination, if targeting specific uses such as microscopy where the goal is to illuminate small (relative to the lenticular sheet) objects positioned within a predefined area. This is not the case for general-purpose integral illumination.
Ghosts can be countered by modifying the design of lenticular optics, modifying the imaging apparatus, or a combination of the two strategies. A simple countermeasure is to erect light-absorbing barriers inside the lenticular sheet, mutually isolating lenslets and physically preventing crosstalk ( Figure 4). While effective, the measure also comes with drawbacks; the composite structure complicates fabrication, and the barriers will have non-zero thickness resulting in some incoming rays being blocked from entering the lenticular sheet.
Another countermeasure is to alter the radiation pattern of the imaging apparatus, ensuring that the majority of rays will enter the lenticular sheet at small angles of incidence. Exactly how this can be achieved varies with the imaging apparatus. Assuming the use of an LCD panel / backlight combination, viable techniques include collimating backlight output using lenses/reflectors, and installing optical films such as louver films or prismatic films (such techniques unavoidably entail varying degrees of energy loss).
Adequate countermeasures can suppress ghosts to imperceptible levels. Note that ghosts can be exacerbated by flaws in device assembly, such as gaps between components.

B. CONTROL MECHANISM
Simulating light source(s) using an integral illumination device consists of several steps, as shown below: 1) Define light source(s) for the device to simulate 2) Determine light rays that the device must emit 3) Compute the pixel pattern that produces said rays 4) Render the pixel pattern Here, as a simple example, let us consider a scenario where we simulate a light source in the shape of a circular disc. We assume that only one side of the disc emits light, and that both the intensity and radiation pattern of outgoing light are uniform across this luminous surface. We place this disc at a fixed position, behind the rear surface of the lenticular sheet (hereafter referred to as the device window). If the luminous properties of the disc are well defined, we can calculate the 4D plenoptic light field L(u, v, θ, δ) across the entire device window using numerical techniques. (Analytical solutions may also be available for the simplest cases.) In our example depicted in Figure 5, L will be 0 for most values of (u, v), as rays from the disc-shaped light source will only pass through a small, circular subregion of the device window.
Integral illumination devices can simulate not only simple light sources positioned within a void, but complex 3D scenes populated with entities (luminous and otherwise) of various geometries and material properties, as long as the entire scene is contained in a volume behind the device window and there exist reasonable methods to calculate L. Physical phenomena such as reflection, refraction, absorption, and attenuation may be taken into account with varying levels of fidelity, again on the condition that L can be calculated. Once L has been computed across the device window, we can use Equations 1 and 2 to obtain the pixel pattern that best replicates L. If the computation was performed with enough accuracy, the device will illuminate the external environment with faithfully simulated lighting effects, and viewers will see a 3D scene extending behind the device window populated with light sources and (optionally) other objects.

C. LIMITATIONS
Although integral illumination devices are versatile, there are a number of factors which limit both the range and fidelity of their simulations. Several of such limitations are listed below:

1) VIEWING ANGLE
The angular range in which an integral illumination device can reliably emit light rays is dictated by the lenslet's viewing angle. However, designing lenslets with high dioptric power while keeping to the form factor of a lenticular sheet (while also minimizing aberrations) is not easy, and if we limit ourselves to simple designs, e.g., single-element convex lenslets, we quickly face theoretical limits. This poses problems for simulations that require rays to emerge at extreme angles.

2) PIXEL SIZE / ANGULAR RESOLUTION
Pixels on the imaging apparatus will have non-negligible size relative to the lenslet diameter, which hinders precise angular control of light rays. Since there are finite pixels beneath each lenslet, control will not be continuous but discrete and statebased. Also, lighting a pixel will result in rays being emitted as a pyramidical beam of progressively expanding width.

3) LENSLET DIAMETER / SPATIAL RESOLUTION
Lenslets will also have non-negligible size relative to that of the lenticular sheet, which results in loss of spatial resolution. This, combined with the aforementioned loss of angular resolution, places a limit on the device's capability to simulate complex 3D scenes. (There exists a tradeoff between the two resolutions; decreasing lenslet diameter will increase spatial resolution, at the expense of angular resolution.)

4) FOCUSING ABILITY
General-purpose integral illumination devices, owing to their lenslet design (as described in Equation 1) and limited angular/spatial resolutions, are ill-suited to form focused images on surfaces at finite distances, à la digital projectors. This introduces difficulties in select cases, such as when mimicking well-focused spotlights often seen in theater lighting.

5) OPTICAL ABERRATIONS
In preceding discussions we had assumed lenslets to have no optical aberrations, which is unattainable in practice. Though effects of chromatic aberrations are often reported as being minor with regards to integral imaging [50] (this is consistent with our experiences with integral illumination), other types of aberrations can visibly affect simulation quality, exacerbating issues of focusing ability, viewing angle, etc.

6) LIGHT PROPERTIES
Photometric and other properties of light rays (e.g., intensity, color, frequency, polarization) that can be produced by an integral illumination device largely depend on the modulation capabilities of the imaging apparatus. For example, emitting monofrequency, unpolarized, or coherent light is out of reach for our prototype described in this paper.

D. APPLICATIONS
Eventually, we expect integral illumination to replace and/or supplement conventional lighting in a wide range of domains. Below is a list of some of the most promising use cases:

1) PHOTOGRAPHY/VIDEOGRAPHY
A wide variety of lighting equipment is used in photography and filmmaking, a subset of which may be consolidated into a small number of integral illumination devices. Recently, film studios are rapidly embracing new digital equipment (such as LED display walls that show CGI backgrounds in real time), making film production one of the most promising near-term use cases for integral illumination.

2) THEATER AND ENTERTAINMENT
Indoor entertainment such as theater and concerts is another domain where a wide range of specialized lighting equipment is used. This presents another promising use case for integral illumination, although the routine use of high-intensity lighting may create additional engineering challenges.

3) AMBIENT LIGHTING
Programmable lighting systems are now commonly installed to provide ambient lighting in indoor and semi-outdoor environments (e.g., homes, offices, stores). Integral illumination devices may serve as enhancements to such systems, offering selective illumination based on real-time needs and facilitating multipurpose use of limited space.

4) STREET LIGHTING
Increasing concerns regarding the effects of light pollution on public health / local ecosystems have led to renewed interest in the roles of programmable (smart) lighting in urban areas. Such systems may be upgraded to use integral illumination, offering finer, city-wide control of urban illumination.

5) MEDIA FACADES
Decorative, digitally-controlled lighting on building exteriors has become a common sight in urban areas. Integral illumination devices, with their ability to mimic near infinite varieties of lighting hardware, may be a good match for such use cases where the primary concern is aesthetic. Figure 6 shows our working prototype of an integral illumination device, codenamed AnyLight. The prototype measures approximately 55 × 55×20 cm, and is connected to a 15-inch Apple Macbook Pro (2015 model) with a 2.8GHz Intel Core i7 CPU and an NVIDIA GeForce CT 750M graphics card.

IV. IMPLEMENTATION
A. HARDWARE 1) IMAGING APPARATUS As discussed earlier, an integral illumination device consists of two main components: a lenticular sheet and an imaging apparatus. For the latter, our prototype uses a combination of a monochrome LCD panel and an LED array backlight. The LCD panel is obtained by disassembling a JVC Kenwood MS25i2 medical-use display. The panel can display 8-bit grayscale images with a contrast ratio of 1400:1; it has a screen resolution of 1600 × 1200 px and dimensions of 432× 324 mm. The decision to use a monochrome panel is based on our desire to alleviate energy loss. Unlike their full-color counterparts, monochrome panels lack color filters which by themselves block roughly two-thirds of incoming light. (This, combined with other forms of losses such as absorption by rear polarizer and obstruction by inter-pixel circuitry, results in transmission rates of color LCD panels typically hovering below 10%.) An obvious drawback to using a monochrome panel is that light color (aside from brightness) is determined by the backlight, and cannot be adjusted on a per-pixel basis. Using an RGB backlight will enable light control at the perpanel level, which we deem sufficient for most use cases. Figure 7 shows the backlight. 195 white-color LEDs (Lumileds Luxeon M, correlated temperature 5700K) are laid out in a hexagonal grid, upon each of which is attached a reflector that focuses outgoing light to a beam angle (Full-Width Half-Maximum) of approximately 30 • . (This contributes to ghost suppression, as we have discussed earlier.) The backlight can only be turned on or off in its entirety; individual control of LEDs are not supported. With its maximum power consumption exceeding 2700W, the backlight is overkill for most use cases and designed to be capable of operating beyond normal usage conditions for testing purposes. Switching to a lower-powered backlight will enable slimmer hardware.
The backlight and LCD panel are assembled to create the imaging apparatus. Unlike typical LCD monitors, we do not install diffusers as they interfere with our ghost-suppression measures; consequently, the LCD panel must be placed at a distance (the exact value is calculated via simulation) from the backlight to achieve relatively uniform illumination of the panel. We also avoid the use of brightness-enhancement films (e.g., 3M DBEF), again as they interfere with ghost suppression. Cooling fans are attached to the side panels, although we found passive cooling to suffice in most situations. A number of alternatives exist regarding the imaging apparatus, each suiting different use cases. For example, projector arrays may be a more suitable option for scenarios in which maximum luminance takes priority.

2) LENTICULAR SHEET
As part of our ghost suppression measures, we use a customdesigned lenticular sheet where lenslets are optically isolated from one another with light-absorbing barriers. Such sheets can be prepared in various ways; while manufacturing details are outside the scope of this paper, to assist replication efforts we briefly outline our fabrication process below.
First, we designed individual lenslets as hexagonal prisms with aspheric top surfaces, which were then manufactured out of polycarbonate via injection molding. The lenslets were manually inserted into a 3D printed (using a Stratasys Fortus FDM printer) ABS frame measuring 432 × 324 mm. Finally, opaque, heat-resistant bond was poured into the frame, to fix the lenslets in place and also to construct the light-absorbing inter-lenslet barriers.
Our lenslets, with a minimal diameter of 7.56mm (equal to the width of 28 pixels on the LCD panel), are large compared to those typically found on lenticular sheets. This stems from manufacturing issues arising from the manual nature of our fabrication process, and is not indicative of any inherent limitations regarding integral illumination. The lenslets were designed to have a viewing angle of approximately 68 • .
In addition to the aforementioned sheets, we also utilized 3D printing to quickly fabricate a range of sheets for testing purposes. Figure 9 shows an example of such sheet, printed using a Stratasys Connex multimaterial 3D printer. (The clear sections are made of the material VeroClear.) Although 3D printing offers convenient, single-pass fabrication of  lenticular sheets, optical performance is suboptimal owing to the layered construction, and the relatively low HDTs (Heat Deflection Temperatures) of clear 3D printable materials make them ill-suited for uses requiring high luminous intensity.

B. SOFTWARE
The hardware is controlled by a custom software written in Objective-C, which also contains a scripting tool for creating new lighting effects ( Figure 10).

1) HARDWARE OPERATION
As explained earlier, simulating light sources on an integral illumination hardware is performed by controlling the pixel pattern displayed on the imaging apparatus ( Figure 11).
Here, for the sake of explanation, let us again consider the simple scenario we had introduced earlier, i.e., simulating a disc-shaped light source placed at a fixed position behind the hardware's device window. Again, we assume that only one side of the disc is luminous, and that both the intensity and radiation pattern of outgoing light are uniform. To simulate this source, we will need to compute the light field L across the entire device window. Our software performs this calculation numerically using the Monte Carlo method. First, a total of k 0 light rays emitted from the source are randomly sampled, where k 0 is defined to be proportional to the disc's luminous flux , as follows: Due to our assumption of uniform intensity and radiation pattern, if the samples were chosen randomly, the origins of the k 0 rays will be distributed more or less equally throughout the disc's luminous surface. The software then discards rays that do not pass through the device window, giving us a set of k rays (k ≤ k 0 ). We represent each of these k rays using a 4D vector (u, v, θ, δ), where coordinates u, v denote the ray's point of intersection with the device window, and angles θ, δ denote its orientation. Now, recall that integral illumination devices have limited spatial/angular resolutions; consequently, valid values of intersection u, v and orientation θ, δ are all finite. Taking this into account, for each ray the 4D vector (u, v, θ, δ) must be converted to a 2D vector (i, j), where i denotes the index of lenslet closest to u, v, and j denotes the index of valid (i.e., reproducible) ray orientation closest to θ, δ.
At this point, we have obtained a set of rays R containing k 2D vectors. Now, for any combination of i and j, we can calculate i,j which is the luminous flux of light that should be emitted from lenslet i in orientation j: where: From here, for all combinations of i and j where i,j > 0, we can use equation 1 to identify the 2D coordinates (s, t) of the pixel that must be turned on to emit the desired light. The grayscale pixel value v s,t can be determined using equation 2, as follows: The weight value w s,t needs to be computed for each pixel based on information regarding the hardware. In the case of our prototype (which uses an LCD panel / backlight combination as the imaging apparatus), this includes lenslet geometry Calculating v s,t for each pixel yields the pixel pattern that simulates the desired light source(s), which in this example is a single luminous disc. Our software simulates more complex light sources by compositing sources with simple geometries such as ellipses and squares. The software currently does not support any interactions of rays between light sources (e.g., obstruction), and thus compositing multiple sources is done through simple summation of pixel values.
The software uses GPGPU to speed up computation. Time required to simulate light sources varies based on parameters such as source geometry and desired fidelity. For simple cases like the aforementioned luminous disc, the process takes less than 0.03 seconds on our Apple Macbook Pro.

2) LIGHTING DESIGN
The software contains a basic scripting system through which new lighting effects can be designed, by writing code in a proprietary programming language (Twinkle).
Light sources are defined by creating instances of primitive sources (e.g., ellipse, rectangle), and setting their parameters such as size, position, intensity, and radiation pattern. Calling the function render() computes a pixel pattern that simulates the instantiated sources; the function can be called repeatedly to create animation effects such as moving or shapeshifting lights. The programming environment contains a library of basic mathematical functions, and a 3D visualizer to preview generated lighting effects. Figure 12 shows a collection of lighting effects designed using the scripting system. In addition to replicating existing light sources, integral illumination devices can render effects that would be difficult or impossible to realize using conventional lighting technologies. The two ''hidden cube'' effects in Figure 12 are examples of such effects-here, a glowing cube is made to appear only when the device is viewed from specific orientations.
The scripting system is easily learnable for those already familiar with tools such as Processing or Arduino. However, non-technical users may be better served by a more graphical interface similar to those of 3D modeling software. Another VOLUME 10, 2022 approach toward lighting design is to digitally scan physical light sources, automatically extracting the parameters needed to replicate them via integral illumination. Recent advances in photogrammetry and neural radiation fields [51] may be of potential use here.

V. PERFORMANCE
Below, we report the results of several performance measurements conducted on our prototype.

A. DIRECTIONAL CONTROL
First, we attempted to test the prototype's capacity to perform accurate directional control of light rays. We set up the device to simulate a single, disc-shaped light source, and measured luminance values at the center of the device window from 33 horizontal directions (−80 • to 80 • at intervals of 5 • ), as shown in Figure 13 (a). We took measurements in two rounds. In the first round the light source was placed parallel to the device window, and in the second it was placed at a horizontal angle of 20 • . All measurements were taken with a Topcon SR-UL2 luminance meter (spectroradiometer). Figure 13 (b) shows the results. Here, actual measurements are compared with ideal and theoretical values. Ideal values represent measurements that should result if the discshaped light source actually existed; theoretical values are predicted measurements computed using prior knowledge of the prototype hardware, including lens geometry/material, backlight radiation pattern, LCD panel properties, etc. (All values are standardized.) Comparing actual measurements to theoretical values, we can see that the values are roughly in agreement in the graph to the left (i.e., the first round of measurements). In the graph to the right (i.e., the second round), a small peak is visible at around -40 • which indicates imperfect suppression of ghosts; this is likely attributable to manufacturing defects, e.g., a gap between the lenticular sheet and the LCD panel. Comparing measurements to ideal values, we can see that in the graph to the right, actual values (and theoretical values as well) show an abrupt drop at around 40 • . This is most likely an effect of the lenslets' limited viewing angle.

B. GHOST SUPPRESSION
Next, we attempted to evaluate the effectiveness of our ghost suppression measures by comparing between two lenticular sheets, with and without the light-absorbing internal barriers. Here, instead of the composite-material sheet shown in Figure 8, we used two 3D printed sheets with identical designs aside from only one of them having the internal barriers. As with the directional control measurements, we programmed the device to simulate a single, disc-shaped light source, and measured luminance values from 33 horizontal directions in two rounds, using the same SR-UL2 luminance meter. Again, all values are standardized. (For this set of measurements, we used an earlier-version prototype that is slightly smaller than the hardware shown in Figure 6.) Figure 14 shows the results. We can see that the barriers do suppress rogue light rays, although not at 100% effectiveness. Again, we believe this is due mainly to manufacturing defects such as a gap existing between the LCD panel and lenticular sheet, or imperfect formation of the light-absorbing barriers. (Internal reflections within the LCD panel may be playing a role as well.) We can also see that ghosts are not as prominent when the light source is facing forward, which is in line with our understanding of the phenomenon.

C. RADIATION LOSS
To understand how much radiation produced by the backlight is lost to absorption, reflection, etc., we measured luminance values (using a Konica Minolta LS-150 luminance meter) at the center of the device window, for the following hardware configurations: • Backlight only: The lenticular sheet and LCD panel are both removed from the hardware shown in Figure 6 • Backlight + LCD panel The lenticular sheet is removed from the hardware shown in Figure 6 • Backlight + Color LCD panel A variation of the above condition, where a color LCD panel (ASUS MB168B-J) is installed in place of the monochrome LCD panel  Pixel pattern was set to all white for both the monochrome and color LCD panels. As our backlight is incapable of emitting uniform light, for the first three hardware configurations (i.e., conditions without the lenticular sheet) we installed an optical diffuser film inside the hardware; failing to do so was found to produce erroneously low luminance measurements. The backlight was driven at approximately 45 watts. Table 1 shows the results. We can see that the monochrome LCD panel has a significantly higher transmission rate compared to the color LCD panel, which agrees with our expectations. From the results for the fourth configuration, we can see that the radiation loss of our prototype, while nontrivial, is generally in line with those of typical digital signage widely used in commercial architecture.
Note that this is not intended to be a complete assessment of our prototype's efficiency, as that will require omnidirectional measurement which is beyond the capabilities of our testing environment. Since our lenslets are designed to have a relatively uniform radiation pattern (as opposed to that of the backlight), a complete assessment will likely reveal a higher transmission rate than the 3.8% reported in Table 1.

D. INTERNAL TEMPERATURE
Finally, we measured how the prototype's internal temperature (defined here as the temperature of the LCD panel's rear surface, taken using an A&D AD-5611A infrared thermometer) rises over time under different operating conditions. Figure 15 shows the results. Here, illuminance values were measured using a Sanwa LX2 lux meter at a distance of 200 cm from the device window, with the pixel pattern set to all white. Room temperature during measurements was roughly 25 • C. (Cooling fans were not activated.) We can see that our prototype is able to produce output at levels similar to those of ambient lighting fixtures, while keeping temperatures well below the maximum operating temperatures of LCD panels, which are typically around 80 to 100 • C.  down the range of light rays that the device must be able to reproduce, we can employ optimizations to make the best use of the finite dioptric power. For example, in the case of our prototype, if we know in advance that rays never need to be emitted rightwards, we may theoretically employ asymmetric lenslets whose viewing angle is sizably larger in the leftward direction than in the rightward direction.

B. NON-PLANAR INTEGRAL ILLUMINATION
Though our prototype is designed to have a flat surface, this is not a hard requirement for integral illumination as neither lenticular sheets nor imaging apparatuses are required to have planar geometries. For example, it should not be overly difficult to create convex, concave, or even spherical devices, by using readily available components such as projectors or curved LCD panels. Exploring such alternative form factors may broaden the range of lighting effects reproducible using integral illumination.

VII. CONCLUSION
This paper described the foundations of integral illumination, an adaptation of the classical principle of integral imaging to artificial lighting. We presented a functional prototype of an integral illumination device, discussed its engineering details and reported performance measurements. We believe that in the coming years, integral illumination will replace a sizable share of conventional artificial lighting, turning lighting into a form of digital media that can be edited at will by end users and freely distributed over networks.
We plan to further continue our work on integral illumination; future work will focus on producing practical hardware designs optimized for specific applications and use cases.
YUICHIRO TAKEUCHI was born in Toronto, ON, Canada, in 1980. He received the Ph.D. degree in informatics from The University of Tokyo, Japan, in 2008, and the master's degree from the Harvard University Graduate School of Design, USA, in 2012. He is currently a Researcher at Sony Computer Science Laboratories Inc., Kyoto, Japan. His research explores the intersection between digital technology and architecture/urban design. He is a member of ACM. He is an Active Member of the Academic Community. He has been recognized with multiple domestic and international awards, including the ACM CHI Best Paper Award. He has served in various leadership roles, including the General Chair for ACM ISS 2018.
KUNIHIKO NAGAMINE received the B.S. degree from Chiba University, in 1993. He is currently an Engineer at Sony Corporation, Tokyo, Japan. His research interests include optics and display technologies, with particular emphasis on developing tools for their quality assessment. His work has been the recipient of multiple awards, including an IDW 2009 Outstanding Poster Award.