Detection and Mapping of Specular Surfaces Using Multibounce Lidar Returns

We propose methods that use specular, multibounce lidar returns to detect and map specular surfaces that might be invisible to conventional lidar systems that rely on direct, single-scatter returns. We derive expressions that relate the time- and angle-of-arrival of these multibounce returns to scattering points on the specular surface, and then use these expressions to formulate techniques for retrieving specular surface geometry when the scene is scanned by a single beam or illuminated with a multi-beam flash. We also consider the special case of transparent specular surfaces, for which surface reflections can be mixed together with light that scatters off of objects lying behind the surface.


Introduction
Although lidar is widely used for mapping the 3D geometry of surfaces, the technology has historically been challenged by specular, or mirror-like, surfaces that typically scatter very little light directly back to the receiver.This inability to detect and localize specular surfaces can result in the failure to detect navigational obstacles like mirrors and windows, or hazards such as wet or icy patches on the ground.It can also result in incomplete scans of cityscapes or man-made interior environments in which glass and metal surfaces are relatively common, and in the complete inability to digitize artifacts that are made of glass or that present a polished metal or chrome finish.
The range to a surface is typically computed using the lidar range equation, which is only valid for single-scatter time-of-flight measurements.Thus, when the direct, single-scatter return from a specular surface is too faint to detect, it becomes impossible to compute the range to that surface via conventional means.Nevertheless, the presence of specular surfaces is often revealed by intense multibounce returns.For instance, when one directly illuminates a diffusely reflecting surface, nearby specular surfaces may produce mirror images of the true laser spot (also referred to as highlights) that appear just as bright as the original.(a) We acquire the shape of a glass window (green and red points) and the scene surrounding it (blue points), which includes objects behind the window (white points, photos on left).When single-scatter paths are assumed (right), the window surface cannot be retrieved and numerous false points are generated.(b) We scan the shape of a copper pitcher (top) by directly illuminating a sequence of 60 laser spots on nearby surfaces and measuring the time-of-flight of two-bounce returns that reflect off of the pitcher (results bottom left).We repeat the scan four times, rotating the pitcher 90 • between scans, to acquire the full shape (bottom right).
Alternatively, when a specular surface is illuminated directly, the beam is often deflected such that it lands on a diffusely reflecting surface nearby.
In this work we demonstrate that multibounce returns are both an important cue that reveals the presence of specular surfaces, as well as an information source that can be used to estimate a specular reflector's shape.We review the geometry of multibounce returns that are encountered in scenes that contain both diffuse and specular reflectors.Using our knowledge of this geometry, we propose criteria for unambiguously detecting the presence of specular multibounce signals, as well as a set of equations that relates the time-and angle-of-arrival of multibounce impulses to the position and orientation of points on the specular surface.
We apply our findings in a series of experiments in which we use multibounce lidar measurements to acquire the 3D shape of various specular objects.These objects include planar reflectors such as mirrors and windows, as well as a polished metal object with a freeform shape.For the special case of transparent surfaces, we propose criteria that can be used to distinguish between multibounce reflections off of the surface, and single-scatter returns from objects that lie behind the surface.
Our methods can be implemented using any lidar system for which the transmit and receive axes can be steered independently, or that uses a wide FOV receiver such as a single-photon avalanche diode (SPAD) array.For most of our experiments we use a pencil-beam illumination source that must be scanned to acquire a point cloud of the full scene.However we also propose an algorithm that can be used to map the surface of planar reflectors when the scene is illuminated by many beams simultaneously.Multi-beam illumination enables faster scene acquisition, and is employed by several commercial lidar scanners [1].

Related Work
Detecting Mirrors and Windows Using Lidar Specular surfaces typically reflect most light away from the lidar receiver, resulting in very faint direct returns.When specular reflections are directed towards the receiver, the signal can be so intense that it saturates one's detectors (this event is sometimes called "glare").The presence of mirrors in a scene can also produce false detections of the mirror images of real objects, which typically appear behind the mirror.
To overcome these challenges, Diosi and Kleeman [4] use an ultrasonic scanner to detect specular surfaces that lidar scanners can't see.Yang et al. [15] infer that framed gaps in detected surfaces likely contain mirrors or windows.After classifying mirrors in this way, they identify mirror image detections and reflect them across the mirror plane to the position of the true object.Foster et al. use sparse glare events to detect specular surfaces, but accumulate glare information over time in an occupancy grid as their laser scanner moves through a space [6].Like Yang et al., Tibebu et al. [12] search for frames that are indicative of windows.However, to avoid detecting windowless holes, they use measurable pulse broadening caused by transmission through glass as a second heuristic to make their window detector more robust.
Unlike this previous work, our method does not rely on contextual cues like frame or mirror image detection that might be produced by non-specular scenes, we do not rely on rare glare events, and we do not require additional sensing modalities like ultrasound.The work that comes closest to ours was a speculative report by Raskar and Davis [9], who suggested that the time-of-flight of two-bounce returns could be used to localize specular surfaces, but did not demonstrate their method, and did not (as we do) consider the scenario where the specular surface is illuminated directly.Henley et al. [7] and Tsai et al. [13] estimate scene shape from two-bounce time-of-flight measurements, but neither consider the special conditions imposed by specular reflectors.

Specular Surface Estimation Using Cameras
There is an extensive body of literature that investigates methods for estimating the geometry of specular surfaces using conventional cameras that cannot capture time-of-flight information.A relatively recent review of this research was provided by Ihrke et al. [8].A camera that observes a specular surface will see a distorted (if the mirror is curved) reflection of the scene that surrounds the surface.In most camera-based methods for specular geometry estimation, features in the distorted, reflected image are matched to features in the true scene.If the positions of the camera and true feature are known, then the depth and surface normal of the surface point that reflects the distorted feature can be determined up to a one-dimensional ambiguity.This ambiguity can be resolved in a variety of ways.In shape from specularity methods, the reflected scene features are point sources that produce specular highlights in the camera image [16] [2].In shape from distortion methods, a camera observes how specular reflection distorts the reflected image of a reference pattern [11][14] [3].In specular flow methods a camera observes how the distorted reflection of an uncalibrated scene appears to move across the surface of a specular object as the camera, object, or scene are moved [10].
Because we observe the specular reflection of laser spots, our method can be interpreted as a shape from specularity technique that uses time-of-flight constraints to resolve the depth-normal ambiguity.In future work, it would be interesting to explore the combination of time-of-flight constraints with other imaging strategies such as shape from distortion or specular flow.

Geometry of Specular Multibounce Signals
Consider the scenario illustrated in Fig. 2, where a mirror has been placed in an environment that is otherwise composed of diffusely reflecting surfaces.The mirror is a perfect specular reflector, which means that it will reflect all light from an incident beam in a single direction that is determined by the light's direction of incidence and the mirror's surface normal orientation.Our lidar system consists of a transmitter at position L that emits a focused, pulsed beam in a single direction.A receiver array is placed at position C, which may be displaced from L by a baseline distance s.
In this scenario the transmitter can illuminate either a diffuse reflector or the mirror directly.These two cases introduce distinct multibounce geometries that must be treated separately.In either case, our goal is to compute the position of all directly and indirectly illuminated points in the scene using the time-of-flight and angle-of-arrival of all observable direct and multibounce returns.
First surface is a diffuse reflector.In the first case, shown in Fig. 2(a), the transmitter directly illuminates a point D on the diffusely reflecting surface.The receiver subsequently observes two laser spots: the true spot, which (correctly) appears to be located at D, and a mirror image of that spot.Light from the true spot has propagated along the single-bounce path LDC, whereas light that appears to originate from the spot's mirror image has propagated along the twobounce path LDSC.Here S is the point at which light reflects off of the mirror.We use a method that was originally introduced in [7] to estimate the positions of D and S from the times-and angles-of-arrival of the one-and two-bounce returns.Angular positions are described by a clockwise rotation by θ about ŷ followed by a clockwise rotation by ϕ about x.
Light incident from the direction of the true spot D will arrive at the angle (θ DC , ϕ DC ) at time t 1 = 1 c (r DL + r DC ), where r DL and r DC are the distances to D from L and C, respectively.The distance to r DC can be computed using the following bi-static range equation: Light incident from the direction of S will arrive at the angle (θ SC , ϕ SC ) at time t 2 = 1 c (r DL + r DS + r SC ).Here r DS is the distance from S to D and r SC is the distance from C to S. From simple arithmetic we see that r DS = c(t 2 − t 1 ) + r DC − r SC .We plug this expression into the law of cosines for the triangle DCS to obtain Here ∆t 12 = t 2 − t 1 and δ is the apparent angular separation of the true spot and its mirror image, with cos δ = cos θ DC cos θ SC +sin θ DC sin θ SC cos (ϕ DC − ϕ SC ).
We have now completely determined the positions of S and D. We can additionally compute the surface normal at S which, by the law of reflection, must be the normalized bisector of the angle formed by line segments DS and CS.
First surface is a specular reflector.In the second case, shown in Fig. 2(b), the mirror is illuminated directly at S 1 .Because the mirror is a perfect specular reflector, no light that scatters at S 1 will travel directly back to the receiver, and so we do not observe a one-bounce return.Instead, all light in the beam is deflected such that it illuminates a spot D on a diffusely reflecting surface.This laser spot is visible to the receiver.Once again, however, the receiver also sees a mirror image of the true laser spot.The mirror image of D appears to lie behind3 the mirror at point D ′ .This time, light that arrives from the true spot has traveled along the two-bounce path LS 1 DC, and light that arrives from the spot's mirror image follows the three-bounce path LS 1 DS 2 C.
Here we show that it is possible to compute the positions of D, S 1 and S 2 if S 1 and S 2 lie on the same plane 4 .This condition is automatically satisfied if the specular surface is itself a plane, or if the baseline s = 0 (in which case S 1 = S 2 ).
Our derivation hinges upon the observation that light that has actually followed the three-bounce trajectory LS 1 DS 2 C will appear, from the perspective of the receiver, to have followed the one-bounce trajectory LD ′ C, which includes a single scattering event at D ′ .We can thus compute the apparent range to D ′ from the three-bounce travel time ).Using the range equation from Eq. 1, we obtain The range to the illusory point D ′ is useful because it can be directly related to the range of the true point D using the following expression: where is the two-bounce travel time.Once we've obtained r DC we compute the range to S 2 by substituting r DS 2 = c(t 3 − t 2 ) + r DC − r S 2 C into the law of cosines for the triangle DCS 2 to obtain where ∆t 23 = t 3 − t 2 and δ once again refers to the apparent angular separation of the true spot and its mirror image.We note that if there are multiple specular surfaces in the scene then there may be multiple mirror image spots that are visible to the receiver.Only light from one of those mirror images can be used to compute r DC via Eq. 4. This will be the mirror image that appears to lie along the transmitted beam (in the direction of S 1 L).However, the range to the reflection points on other specular surfaces can be computed once r DC is known.This is accomplished by plugging the angle-of-arrival and three-bounce time-of-flight associated with these other mirror image spots into Eq. 5.
If the direction of the transmitted beam is known then we can also compute the position of the directly illuminated point S 1 .We begin by computing the apparent distance r D ′ L from the laser to D ′ : The distance from L to S 1 can then be computed using the identity r DS 1 = r D ′ L − r S 1 L and the law of cosines from the triangle S 1 LD.This distance is where Here dS 1 L is the direction of the transmitted beam.Finally, from the law of reflection, we can compute the surface normal vectors at S 1 and S 2 once D, S 1 and S 2 are known.
Identifying the true spot and the first surface.Although we have derived expressions that could in theory be used to compute the positions of all diffuse and specular scattering points within the scene, there are two ambiguities that need to be resolved before these expressions can be applied.First, we must determine whether the directly illuminated surface is a diffuse or a specular reflector.Doing so from raw measurements is not as straightforward as it might seem because in either case our receiver will see at least two spots, and one of these spots will appear to lie along the transmitter's boresight.Second, regardless of which surface was illuminated first, we must also determine which observed spot is the true laser spot and which are mirror images.
Resolving the second ambiguity is always straightforward if we have timeof-flight information.Light from the true spot always arrives before the light from the mirror images.This can be confirmed by inspecting Fig. 2. In this figure all propagation paths travel through D. However, light from the true spot travels directly from D to C, whereas light from the mirror images must follow a longer, indirect path that includes an additional reflection.The firstsurface ambiguity can be resolved once the true spot has been identified.If the true spot lies along the transmitter boresight, then a diffuse reflector was illuminated first.If it doesn't, then the specular surface was illuminated first.Once both ambiguities have been resolved, the appropriate range equations can be applied to determine the scattering points.
It is worth noting that we are only able to disambiguate these two cases because we have time-of-flight information.If we had instead been viewing the scene with a regular camera that only measured angles and intensities, then there would be no direct way to determine which spot was the true spot and which surface was illuminated first.Instead, we would have to rely on contextual cues to determine, for instance, which pixels seemed to have mirrors in them.

Transparent Specular Surfaces
Many of the most common specular surfaces-glass windows, for example-are partially transparent.Transparency introduces an additional source of ambiguity into our measurements, which is highlighted in Fig. 3(a).Here we see that, as was true for the mirror surface, three-bounce returns will typically resemble spots that lie behind the specular surface.When that surface is also transparent, however, these three-bounce returns may be mixed together with one-bounce returns that have scattered diffusely off of something that lies behind (or on) the specular surface.In this section we propose several criteria that can be used to detect and disambiguate these two kinds of signals.
Multiple Detections Along Beam.Our disambiguation logic is triggered when we detect multiple spots lying along the transmitted beam vector.When this occurs, it suggests that we may have detected the mirror image of a true spot that lies in the scene in front of the window, in addition to a one-bounce return from a true spot that lies behind the window or on its surface.
Comparison to Two-bounce Travel Time.When the window is illuminated directly, light that arrives from the true spot in front of the window follows a two-bounce path, and the spot will not lie on the beam vector.As was explained in Sec.3.1, the mirror image in this scenario must correspond to a three-bounce return, and the associated three-bounce time-of-flight must be greater than the two-bounce time-of-flight from the true spot.Thus, if the time-of-flight associated with any of the spots that lie along the beam vector is less than the two-bounce time-of-flight, we deduce that they cannot be mirror image returns and, thus, must be single-scatter returns from on or behind the window surface.
Multiple Reflections Diminishes Intensity.If the time-of-flight associated with more than one in-beam spot is greater than the two-bounce time-of-flight, we infer that the dimmest spot corresponds to the mirror image.The reasoning behind this is that three-bounce signals are reflected twice by the window, whereas one-bounce signals are transmitted twice.Common transparent, specular materials typically transmit more light than they reflect, although this assumption can break down at glancing incidence angles.To choose the dimmest spot, we rank in-beam detections by their range-adjusted intensity (∼ r 2 I).For this computation, we use the apparent range of each spot, computed using Eq. 1.

Illumination with Multiple Beams
It is straightforward to employ the concepts introduced in Secs.3.1 and 3.2 to acquire specular-surface geometry if the scene can be scanned by transmitting a single beam at a time.Unfortunately, the time required to do so may be unacceptably long for some applications.In this section we propose an algorithm that permits the mapping of specular surface geometry without any knowledge of spot-to-beam associations, and thus can be implemented when the scene is illuminated by a multi-beam flash.
Mirrored Source Geometry If a transmitter emits a pulse at time t in a scene that contains a flat mirror, then a receiver that is pointed at the mirror will observe a mirror image of the transmitter that also appears to emit a pulse at t.The direction that the mirrored pulse propagates will be flipped across the mirror plane.This geometry is visualized in Fig. 3(b), where the true source is at L, and its mirror image is at L ′ .The position of L ′ is significant because the plane of the mirror perfectly bisects the line segment LL ′ , and the plane's surface normal is parallel to LL ′ .This means that estimating the position of L ′ (relative to L) is equivalent to estimating the mirror plane.This is the key principle that underlies our multi-beam surface estimation algorithm.
The one-, two-, and three-bounce signals described in previous sections can also be interpreted under a mirrored space model.From the perspective of the receiver, one-and three-bounce returns appear to originate from the true source L. Two-bounce returns appear to originate from the mirrored source, at L ′ .From Fig. 3(b), we see that the order of the bounces determines where the scattering event will appear to occur.If light bounces off of the mirror first, then the scattering event appears to occur in the true space at D. If it bounces off a diffuse reflector first, then light appears to scatter in the mirrored space at D ′ .

Source Localization Using Multilateration
Because spots produced by oneor three-bounce light paths appear to originate from the true source, they must also lie along the axis of one of the true, transmitted beams.Spots that result from two-bounce light paths, on the other hand, will not in general lie along a transmitted beam vector.Because of this it is straightforward to determine which detected spots correspond to two-bounce light paths.An image of spot detections classified in a multi-beam data collection is shown in Fig. 6(b).
If we could compute the apparent positions of at least three two-bounce spots, then we could compute the ranges r DL ′ = ct 2 − r DC or r D ′ L ′ = ct 2 − r D ′ C , and determine L ′ by solving a multilateration problem.Although we cannot compute these positions directly without knowledge of spot-to-beam associations, we can approximate them.From one-bounce returns, we compute the positions of several points in the true space using Eq. 1.We also compute the positions of a number of points in the mirrored space from three-bounce returns, using Eq. 3. We then approximate the positions of the two-bounce spots by interpolating from the positions of the nearest (in apparent angle) one-and three-bounce spots.
Once we've computed these approximate positions we can estimate the mirrored source position L ′ = [x 0 , y 0 , z 0 ] T by solving the following optimization problem {x 0 , y 0 , z 0 } = arg min (8) Here (x i , y i , z i ) is the approximate position (in C-centered coordinates) of the i th detected two-bounce spot, r i = x 2 i + y 2 i + z 2 i and t i is the two-bounce travel-time associated with the i th spot.The objective function is twice-differentiable, so we can solve the problem using Newton's method.In practice, misclassifications of two-bounce spots as one-or three-bounce spots, or vice-versa, produce large errors in the source localization result.To make our algorithm more robust to misclassification errors, we solve Eq. 8 using a RANSAC [5] approach that is robust to outliers.

Determining Scattering Points
As can be seen in Figure 3(b), rays drawn from the camera and from the true source to two-bounce points lying behind the mirror plane will intersect the mirror plane at reflection points, as will rays drawn from the mirrored source to two-bounce spots in front of the mirror plane.Thus, once the mirror plane has been computed from our estimate of L ′ , we can find all points of specular reflection.From these reflection points we can also estimate the mirror's boundary.This allows us to determine which spots correspond to three-bounce paths.The apparent positions of three-bounce spots can then be reflected across the mirror plane to retrieve the true diffuse scattering positions.This can be achieved even when the true scattering points lie outside of the receiver's field-of-view (e.g. if they are hidden around a corner).
It is important to note that Eq. 8 only applies to planar specular surfaces, and so this algorithm will not produce accurate reconstructions of non-planar specular surfaces.

Implementation
Lidar System.A photo of our lidar system is provided in Fig. 4(a).The transmitter consists of a focused, pulsed laser source (640nm wavelength) that is scanned using a two-axis mirror galvanometer.The receiver is a single-pixel SPAD detector with a focused FOV that can be scanned independently from the laser using a second set of galvo mirrors.The overall instrument response function (IRF) of our system was measured to be 128 ps (FWHM).Details concerning the specific equipment used can be found in the supplemental material.
For each beam pointing direction, we reproduce the angular sampling pattern of a SPAD array camera by scanning the FOV of the detector across a dense, uniform grid.For experiments reported in this paper, the per-pixel dwell time was either 5 or 10 ms.The laser was operated at a pulse repetition frequency of 20 MHz and an average transmitted power of 5 µW .Spot Extraction.Raw photon count measurements are sorted into a data cube that is binned by photon time-of-arrival and detector scan angle.From these raw measurements we detect spot-like returns and then extract the time-offlight, angle-of-arrival, and returned energy of each spot.An image of per-pixel detected photon counts collected from a single detector scan is shown in Fig. 4(b).Detected spots are circled in red.A more detailed description of our spot detection and low-level signal processing pipelines is provided in the Supplement.

Planar Surface Scans
Planar Mirror We scanned a scene (shown in Figure 5(a)) that contained a tall, flat mirror, a wooden floor, and a matte white wall.The mirror was placed approximately 2m away from the lidar scanner.We scanned the laser beam along a 10 × 10 grid of pointing directions.14 beams illuminated the mirror directly.The remainder illuminated the floor, wall, or the mirror's wooden frame.For each beam direction the detector FOV was scanned across a 200 × 200 grid that subtended a ±30 • angle of view in both vertical and horizontal directions.The per-pixel dwell time was 5 ms.Total acquisition time was 5 hours and 33 minutes, although we note that an equivalent data collection made using a 200 × 200 pixel SPAD array would have been captured in 0.5 seconds.
Our results are shown in Fig. 5(a).Here, blue points represent points on diffuse surfaces, red points are points on the mirror as seen from the receiver (S or S 2 in Fig. 2), and green points are points on the mirror that were illuminated directly (S 1 in Fig. 2).Points computed using diffuse-first equations are marked by circles, whereas points computed using specular-first equations are marked with asterisks.Surface normals are also plotted for specular surface points.
It is evident from Fig. 5(a) that the point cloud accurately captures the dimensions of the scene.From a comparison to a ground truth scan collected on the mirror's frame, we determined that the points on the mirror surface had an RMS displacement of 9.5mm with respect to the ground truth plane, and the surface normals had an RMS tilt of 0.65 • with respect to the true surface normal.A complete evaluation of the scan's accuracy is provided in the Supplement.

Planar Glass Window
We also scanned the shape of a single-pane glass window, which is shown in Fig. 5(b).The window was placed in the same position and orientation as the previously scanned mirror.For this collection we did not place any objects behind the window, and so the primary challenge was that the lower reflectance of the glass resulted in fainter multibounce returns, particularly for three-bounce light.We found that each reflection off the window reduced the range-adjusted intensity of a spot by approximately a factor of 10.
We compensated for the lower reflectance by doubling the per-pixel dwell time, to 10 ms.The laser scan pattern was changed to an 11 × 9 grid that more densely sampled the portion of the scene that contained the window.The detector scan pattern was identical to that used in the mirror experiment.The total acquisition time was 11 hours, although we note that an equivalent data collection could have been captured by a SPAD array in just under one second.
Our results are shown in Fig. 5.The window pane can be seen clearly, and appears to have been mapped accurately.Despite the lower reflectance of the glass, we are able to detect all two-bounce and three-bounce returns with no misses or false alarms However, when the point cloud is viewed from the topdown it is clear that the points computed from three-bounce returns (marked by red and green asterisks) are noisier than two-bounce points.This was likely a consequence of the lower relative intensity of three-bounce returns.

Detecting Objects Behind a Window
We placed two objects behind the window and the repeated the window scan described previously.Our results are shown in Fig. 1(a).Behind-window detections are marked by white dots circled in blue.From these results we see that our disambiguation logic accurately classified four spots that scattered off of the objects behind the window.We note that all other points that appear to lie behind the window in the top-down view of the point cloud in fact lie above or below the window aperture.
In addition to the results shown here, we performed a second experiment that was designed to rigorously test the disambiguation criteria described in Section 3.2.These additional results can be found in the supplemental material in Section S.4.
Comparison to Naïve Single-bounce Processing.On the right side of Fig. 1(a) we show the point cloud that would have been generated if the positions of all spots had been naïvely computed using the conventional one-bounce range equation.Notably, this naïve procesing detects no points on the window's surface.Detections that might have been mapped to the window surface are instead mapped to erroneous points.Only true one-bounce returns are mapped correctly.

Non-planar Object Scans
Our method can also be used to acquire the shape of non-planar specular surfaces.We demonstrated this by scanning the shape of a copper pitcher.A photo of this pitcher and the results of our scan are shown in Fig. 1(b).The pitcher was placed on a wooden floor and in between two white side walls.We illuminated a sequence of 60 laser spots on these surfaces, and observed two-bounce returns that reflected off of the pitcher.
Laser spots and two-bounce highlights were acquired using a 200 × 100 pixel scan with a 5 ms per-pixel dwell time.We repeated the scan four times, rotating the pitcher 90 • between each scan so that we could acquire it's front, back, and side-facing surfaces.The four point clouds were aligned to a pitcher-centered coordinate system and then combined.From the combined point cloud in Fig. 1(b), we see that were are able to recover the general shape of the pitcher, although there are several gaps that correspond to regions that did not produce any detectable highlights due to their surface orientation.
We point out two distinctive features of the non-planar surface scanning process.First, because the pitcher's surface alternated between convex, concave, and hyperbolic curvature, one laser spot would produce multiple highlights, each of which could be used to locate a point on the pitcher's surface.This can be seen in the photograph on the top-right of Fig. 1.Second, Eqs. 4 and 5 are only correct when the tangent planes of surface points S 1 and S 2 are coincident.This is automatically satisfied for monostatic (s = 0) lidar systems, for which S 1 = S 2 .However, because our scanner was bi-static, these equations could not be applied for non-planar surface mapping.Consequently, all points in Fig. 1 had to be acquired by directly illuminating spots on diffuse reflectors, and observing two-bounce returns reflected by the pitcher.Specular-first returns could not be used.However, using the criteria outlined in Sec.3.1, specular-first returns could be detected automatically and discarded.

Multiple-Beam Illumination
To test our multi-beam surface mapping algorithm, we summed the single-beam photon count histograms acquired during the first planar window scan described in Sec.4.2.This produced the equivalent of a single multi-beam flash exposure.Detected per-pixel energy values from this multi-beam data are shown in Fig. 6(b).One-and three-bounce spots are circled in green, and two-bounce spots are circled in red.
Our results are shown in Fig. 6(a).By visual inspection we see that the window scattering points closely match the single-beam results in Fig. 5(b).Points in the multi-beam results in fact appear less noisy, although this is because they are intersections with a single, fitted plane.Many reflected three-bounce points appear in erroneous positions.Most of these points correspond to twobounce spots that were incorrectly classified as three-bounce spots.

Discussion
We have demonstrated methods that use multibounce specular lidar returns to detect and map specular surfaces that might otherwise be invisible to lidar systems that rely on single-scatter measurements.We considered the cases of single-beam and multiple-beam illumination, and demonstrated our methods by scanning planar, non-planar, and transparent surfaces.
Because specular surfaces are relatively common, our work could be used in most domains that lidar scanning is applied to, including autonomous navigation, mapping of indoor or outdoor spaces, and object scanning.Future work might address failure cases, such as propagation paths with consecutive specular reflections, or geometries for which the true or mirror image spot lies outside the receiver FOV.We would also be interested in extending our multi-beam algorithm to the mapping of curved surfaces, and to allow dense flash illumination.robust to outliers.In our implementation, the parameters of the fitted model are the coordinates {x 0 , y 0 , z 0 } of the mirrored source position.At each iteration we produce tentative source position fits to sets of n points, and find inliers to this fit among the set of all other points.If there are greater than d additional inliers, we incorporate these inliers into an improved fit.We run the RANSAC procedure for k iterations, and the best model is selected to be the model with the lowest mean squared error among it's inlier set.For results reported in this paper, we used values of n = 4, d = 10, and k = 1000.

S.2 Implementation: Additional Details
Lidar System A photo of our lidar system is provided in Fig. 4(a) of the main text.The transmitter consists of a focused, pulsed laser source (Picoquant LDH-D Series) with a wavelength of 640 nm that is scanned using a two-axis mirror galvanometer (Thorlabs GVS012).The receiver is a single-pixel SPAD detector (MPD PDM series) with a focused FOV that can be scanned independently from the laser using a second set of galvo mirrors (Thorlabs GVS412).A 50 mm lens was used to focus the SPAD FOV at infinity, and a bandpass filter centered at 640 nm was placed in front of the lens to minimize ambient background interference.A Picoquant Hydraharp 400 was used to synchrnously record photon arrival times.The timing jitter of our SPAD was 50 ps (FWHM) and the pulsewidth of our laser was 90 ps (FWHM).The overall instrument response function (IRF) of our system was measured to be 128 ps (FWHM).
Low-level Signal Processing.Raw photon count measurements are sorted into a data cube that contains a time-of-flight histogram for each angle, or pixel, of the detector scan.Each histogram is convolved with a matched filter that resembles the system IRF.A detection is declared if the matched peak is greater than a noise-dependent false-alarm threshold and a hand-tuned absolute counts threshold.A fitting procedure (details are provided in [?]) estimates the return pulse's time-of-arrival in each pixel to sub-timing-bin resolution using counts within a small fitting window surrounding the matched peak.The estimated uncertainty in each time-of-flight estimate is also computed.The return energy in each pixel is reported as the sum of the photon counts within this fitting window minus the expected number of background counts.Counts in a designated set of noise bins are used to estimate the background level in each histogram.
Spot Extraction Per-pixel energy values, time-of-flight values, time-of-flight uncertainties, and boolean detection flags are passed to a spot extraction pipeline.The energy image is then filtered by a matched "spot" filter (which is a laplace filter when a 3 × 3 filter size is used) and a box filter of the same size.The ratio of these filter outputs is computed at each pixel as a measure of "spottiness".A "spottiness" threshold is employed to filter out intense but un-spot-like returns, such as the diffuse inter-reflections seen near edges and corners.We find contiguous regions of pixels that have passed both the spottiness test and the low-level detection criteria described previously.In each of these regions we find the pixel with the maximum energy value and center a small (3 × 3 or 5 × 5) window around that pixel.The spot center is set to be the centroid of the energy values within the window.This spot center is first computed in pixel space, and then converted to angle-space to retrieve the angle-of-arrival of the spot.The spot's energy is set to the sum of the energy values within the window, and the spot's time-of-flight is set to be the uncertainty-weighted average of the time-of-flight values within the window.If two spots are too close together (less than half the window size, rounded up), we throw away the spot that has the lower energy.

S.3 Analysis of Errors in Mirror Scan Experiment
To validate our method, we investigated the accuracy of the points computed for our scan of the large mirror that was reported in Sec.4.2 of the main text.We fit a plane to a set of 22 ground truth points on the mirror's frame.These ground truth points were acquired during a separate data collection which utilized a dwell time of 25ms per pixel-which was five times higher than the dwell time of the original scan.This longer dwell time allowed us to collect more photons per point and thus make more precise position estimates.All points on the mirror frame reflected light diffusely, so their positions could be computed from the conventional single-scatter range equation.The ground truth plane fit was then adjusted to account for the fact that the mirror surface was recessed 1.6cm behind the plane of the frame.
We then compared the position and surface normal of mirror surface points computed from the original scan data to this ground truth plane.In Figure S.1(a) we plot the orthogonal displacement of all estimated mirror plane points as a function of their projected position on the ground truth plane.The root- mean-square (RMS) displacement of these points was 9.4mm and the mean displacement was −7.1mm.In Figure S.1(b) we show the angular displacement of the estimated normal vectors from the ground truth normal vector.The RMS angular displacement was 0.63 • and the mean displacement was 0.54 • .The presence of small but significant mean errors suggest a systematic bias in our results.This bias could have resulted from many factors.First, the angular calibration of our system was not perfect, and errors in angle-of-arrival estimates of diffuse surface returns may have been coupled into specular surface positions.Second, the ground truth scan was taken 24 hrs after the initial scan, and it is possible that the mirror moved or tilted slightly in between scans.Finally, the presence of a glass first-surface may have also contributed to the bias.The mirror's protective glass was 6.35mm thick, and we did not account for the reduced speed of light within the glass while estimating the positions of points on the mirror surface.To do so would have been non-trivial because the increase in travel time would have been dependent on angles of incidence and exittance with respect to the glass surface.Another prominent artifact is the larger normal vector errors associated points computed from specular first-surface measurements as opposed to diffuse first-surface measurements.These points form a vertical line down the center of the plane which can be seen plainly in To assess the precision of our estimates, we fit a plane to the collection of mirror surface points and then compared individual points to the plane fit.Fitting a plane to oriented points is straightforward.Because each oriented point includes a position and a surface normal vector, each oriented point defines a plane.Thus, to "fit" a plane to these oriented points, we simply need to compute the plane parameters for each individual point and then take the average of these parameters.We define a plane by the identity nx = d.The fit plane had the parameters nfit = [−.8797− .0048− .4754]T and d f it = −1.406m.

S.4 Additional Experiment: Transparent Specular Surface
We carried out an additional experiment to more exhaustively test the transparent specular surface processing logic that was proposed in Sec.3.2 of the main text.In this experiment we scanned a scene that contained two large, diffusely reflecting walls that were respectively placed in front of and behind a single-pane glass window.A photograph of this scene is shown in Fig. S.3(a).Dark pieces of construction paper were pasted onto each wall to produce surfaces that varied between high and low reflectance.At some view angles the mirror image of the front wall appeared to be closer to the camera than the true plane of the back wall, whereas at other view angles it appeared to be more distant.This assured that both the time-of-flight and reflectance criteria proposed in Sec.3.2 were put to the test.For added scene complexity, we also placed a small rabbit figurine immediately behind the window.
We scanned the scene using a pattern of 14×10 beams scanned in a uniformly spaced rectangular grid.The detector scan was 190 × 140 pixels that spanned 50.3 • ×37.3 • with a per-pixel dwell time of 10ms.Total scan time was 11 hours, 7 minutes, and 20 seconds, although a SPAD array could have made an equivalent measurement in 1.4 seconds.
Undetected Three-bounce Highlights In contrast to the experiments described in the main text, in this scene there were many occasions where the three-bounce reflection off of the window surface was too faint to detect.This frequently occurred when the laser beam was deflected onto a dark surface on the front wall.Losses from absorption at the diffuse surface compounded with losses from two glass reflections, resulting in extremely faint three-bounce returns that could not be detected.Our original processing logic could not be used in this scenario, because it relied on the presence of multiple detections along the beam vector to detect the presence of a window.
We addressed this scenario by implementing an additional test.As in our regular pipeline, if only two spots were detected, we assumed that the spot that was associated with the lower time-of-flight measurement corresponded to a true spot on a diffuse surface lying in front of the window (D 1 in Fig. 3(a)).If the range-adjusted reflectance (computed using intensity and naïve single-scatter range) of this first spot was lower than the range-adjusted reflectance of the second spot, we infered that the second spot was in fact a diffuse return from a surface behind the window (D 2 in Fig. 3(a)).The reasoning behind this is that a true three-bounce return should always have a lower apparent reflectance than the two-bounce return that it is associated with, due to the additional reflection off of the glass surface.
This line of reasoning fails in two important cases.First, if the time-of-flight associated with the behind-window detection is smaller than the time-of-flight associated with the spot in front of the window, then the true spot in front of the window will be interpreted as a return from a specular surface (S in Fig. 2(a)).Second, if the illuminated spot behind the window has very low reflectance, it might be interpreted as a three-bounce highlight.However, there is no way to directly disambiguate these scenarios using only the relative times-of-flight and reflectance measurements.
Results.Our results are shown in Fig. S.3(b).We are able to retrieve the surface of the window as well as the surface of the two large walls and the floor.A single detection off of the rabbit figurine was also registered.However there are a few artifacts in our point cloud that are worth pointing out.
First, three (of 140) beams produced detections in false positions.These detections can be seen floating in the space between the lidar system and the window, and floating behind the front wall.In all three cases, the true threebounce returns were not detected.Two of these beams illuminated a spot on a dark surface lying behind the window.The behind-window spot was misinterpreted as a three-bounce return, and the wrong set of range equations was applied, resulting in three erroneously mapped detections in each case.For the third beam, the time-of-flight of the behind-window spot was smaller than the time-of-flight arriving from the spot in front of the window.This meant that the spot on the front wall was interpreted as a specular reflection point, and was erroneously mapped to a point directly in front of the receiver.
Second, many points on the back wall are missing or misclassified as lying in front of the window.Most of the missing points correspond to portions of the wall that lay in the shadow of the rabbit figurine.The remaining missing points correspond to the misinterpreted behind-window spots that were described in the previous paragraph.Four spots lying behind the window were classified as lying in front of the window.This occurred during exposures for which the spot behind the window was the only spot detected, and was thus treated as a regular spot on a diffusely reflecting surface.

S.5 Multi-beam Scan of Large Mirror Target
We also tested our multi-beam surface scanning algorithm on the data collected for the large mirror scan reported in Sec.4.2 of the main text.As with our multibeam scan of the window, we summed together the photon-count histograms acquired from the single-beam scan of the mirror to generate the equivalent of one multi-beam flash exposure.Detected per-pixel energy values of this summed dataset are shown in ).If multiple spot positions were deemed to intersect the same beam vector, the spot closest to the beam vector was chosen as the candidate one-or three-bounce spot, and the others were classified as two-bounce spots.This classification process was not perfect, and misclassification errors justified our use of a robust RANSAC [?] source fitting algorithm that was described in Sec.S.1.1.
Our results are shown in Fig S .4(a).As with the window scan, the results of the multi-beam surface scanning algorithm produced for the large mirror target match the single-beam results very closely.Interestingly, because our multi-beam algorithm did not require spot-to-beam associations, our multi-beam algorithm was able to reclaim certain points that were not computed during the original single-beam scan.These additional points corresponded to points directly illuminated near the edge of the mirror's surface (S 1 in Fig. 2(b)).In the original scan, no highlight (S 2 ) was observed for these beams because the point of reflection would have been beyond the mirror's edge, and thus S 1 was not computed.However, in the multi-beam algorithm, the intersections of all beam vectors with the fitted mirror plane are computed, including the points that had no associated highlight measurement.

S.6 Scan of a Thin Transparent Object
In addition to scanning a metal pitcher, we also scanned a glass vase.We used the same detector and laser scan patterns that were used to acquire the shape of the copper pitcher in Sec.4.3 of the main text, but increased the per-pixel dwell time of our detector to 10ms to account for the lower reflectivity of the glass.
Our result is shown in Fig. S.5.Due to the rotational symmetry of the vase, we did not rotate it to measure the shape of all sides.Interestingly, after scanning the vase from just a single vantage point, we noticed that we were able to measure points on both the front and the back surface of the vase, due to the fact that the vase was transparent.Furthermore, because the vase is a thin glass object, we expect that the effect of distortion of the returned signals due to refraction should be small.Thus, the position of these back surface points should be relatively accurate.Surface normals for back-surface points are identified as inward-facing, whereas surface normals on the front surface are outward facing.This is technically correct, and in each case corresponds to the true direction of the surface normal vector at the point of specular reflection.

S.7 Common Failure Cases
Our methods can be used to detect and localize specular surfaces that might be invisible to a conventional lidar system that relies on measurements of direct, single-scatter returns.Nevertheless there are specific circumstances that cause our method to fail.We depict some of these cases in Figure S.6.
Only one spot is detected.When the scene is illuminated with a single beam, the position of the specular surface can only be determined if both the  true laser spot and its mirror image are detected.There are many scenarios in which this condition is not met.The most obvious is when the laser beam reflects off of a specular surface but never lands on a diffusely reflecting surface.This scenario is more common in wide-open outdoor environments in which the beam might get reflected towards the sky.A scenario that is more common in enclosed environments is when the beam is deflected onto a surface that lies outside the receiver's field-of-view (FOV).We illustrate this scenario in Fig. S.6(a).Such a scenario often occurs when a specular surface is illuminated at near-normal incidence, such that the beam lands on a diffuse reflector that is behind the lidar.One remedy in this context is to use a receiver with a very wide FOV.Although such receivers may not be typical, many lidar systems used by autonomous vehicles have a 360 • horizontal FOV.Additionally, while near-normal incidence angles challenge methods that rely on specular multibounce returns, they are also more conducive to observations of the direct, single-scatter returns that can be interpreted using conventional processing.Lastly, it is also possible that an occluding surface will block the receiver's line-of-sight to the true laser spot.
The consequences of detecting only one of two spots depend upon which spot is detected and which surface was illuminated first.If a diffuse reflector is illuminated first and only one-bounce returns are visible, then the diffuse reflection point can be determined accurately using Eq. 1 but we will learn nothing about the specular reflector.If the specular reflector is illuminated first and only the three-bounce return is observed, then the three-bounce return may be interpreted as a one-bounce return, causing a false point to be mapped behind the specular surface.Regardless of which surface was illuminated first, if only the two-bounce return is detected then it can be identified and discarded without producing false points.This is because the direction that two-bounce light returns from will not, in general, match the direction of the transmitted beam.
Multiple specular reflections.Our method does not consider propagation paths that include two or more specular reflections in a row.We've depicted one of these propagation paths in Figure S. 6(b).This scenario can occur when scanning a free-form specular object with concavities, or when a scene has multiple planar reflectors that face each other.Although we do not treat such cases in this work, it is likely that the additional specular points can be mapped in these circumstances if the additional highlights that they produce are observed.Alternatively, certain cues such as chirality flips of an asymmetric beam pattern might be used to identify and discard multi-specular returns.
Reflection point is off surface.If there is a non-zero baseline separation between the transmitter and the receiver (that is, if the lidar system is bi-static), then there will be a disparity between the point that the transmitter directly illuminates on a specular surface (S 1 ) and the second point of reflection (S 2 ), which corresponds to the specular highlight that is visible to the receiver.This sometimes leads to the scenario depicted in Fig. S.6(c) when a specular surface is illuminated near its edge.Here, the point that would have reflected light back towards the receiver is just empty space-it's beyond the edge of the specular surface.In this case no three-bounce return is observed, but a two-bounce return is still produced.In such a scenario, S 1 can be recovered by finding the point at which the beam vector intersects a plane (or other shape) fit to other specular surface points that were computed using measurements corresponding to different beam vectors.Once S 1 is retrieved, the position of the diffuse (two-bounce) spot can also be determined using angle and time-of-flight constraints.
We encountered this issue when scanning the shape of the large mirror during the experiment reported in Sec.4.2 of the main text.The mirror was much taller than it was wide, and was also oriented at an angle such that its surface subtended a relatively small horizontal angle from the perspective of our lidar system.As a result, only 9 of the 14 beams which directly illuminated the mirror produced three-bounce returns.We did not compute the additional S 1 positions, and so this effectively reduced the number of points that we acquired on the mirror's surface.One of the reasons that we encountered this phenomenon was that the transmitter-receiver baseline separation of our lidar system was very large (25.7cm) compared to typical laser scanning systems.The effect would have been less noticeable if a system with a smaller baseline had been used.

Fig. 1 .
Fig. 1.(a) We acquire the shape of a glass window (green and red points) and the scene surrounding it (blue points), which includes objects behind the window (white points, photos on left).When single-scatter paths are assumed (right), the window surface cannot be retrieved and numerous false points are generated.(b) We scan the shape of a copper pitcher (top) by directly illuminating a sequence of 60 laser spots on nearby surfaces and measuring the time-of-flight of two-bounce returns that reflect off of the pitcher (results bottom left).We repeat the scan four times, rotating the pitcher 90 • between scans, to acquire the full shape (bottom right).

Fig. 2 .
Fig. 2. (a) A diffusely reflecting surface is illuminated directly at D. The camera observes one-bounce returns incident from D and two-bounce returns incident from point S on the mirror surface.(b) The specular surface is illuminated directly (at S 1 ) and the single-bounce return is too faint to observe.The camera observes two-bounce returns incident from D and three-bounce returns incident from S 2 .

Fig. 3 .
Fig. 3. (a) When a transparent specular surface is illuminated directly, multiple laser spots may be detected along the beam vector-a true spot D 2 that lies behind the surface, and a mirror image (D ′ 1 ) of a spot D 1 that lies in front of the surface.(b) Two-bounce paths (right) appear to originate at L ′ , a mirror image of the true source.One-and three-bounce paths (left) appear to originate at the true source L.

Fig. 4 .
Fig. 4. Implementation.(a) Our lidar system consists of a focused, pulsed galvoscanned laser and a single-pixel SPAD that can be scanned independently of the laser.In our experiments the SPAD FOV is scanned across a dense, uniform grid to mimic the angular sampling pattern of a SPAD array camera.(b) Per-pixel map of detected photon counts.Detected spots are circled in red.

Fig. 5 .
Fig. 5. Scans of (a) a large flat mirror and (b) a single-pane glass window.Red and green points indicate points on specular surfaces, blue points indicate points on diffusely reflecting surfaces.Circles indicate diffuse-first illumination geometry and asterisks indicate specular-first geometry.Surface normals are also plotted at specular points.

Fig. 6 .
Fig. 6.(a) Point-cloud generated using multi-beam surface mapping algorithm on glass window dataset (Fig. 5b).(b) Detected per-pixel energy values of multi-beam data collection, overlaid with spot positions.Red circles mark two-bounce spots, and green circles mark one-or three-bounce spots.

Connor
Henley was supported by a Draper Scholarship.This material is based upon work supported by the Office of Naval Research under Contract No. N00014-21-C-1040.Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Office of Naval Research.

Fig. S. 1 .
Fig. S.1.Mirror scan error analysis.Plots of (a) distance of points from ground truth mirror plane and (b) angular displacement of surface normal vectors from the ground truth plane.Errors are plotted as a function of the projected position of each point onto the ground truth plane.

Fig. S. 2 .
Fig. S.2.Mirror scan residuals analysis.Plots of (a) distance of points from the fitted mirror plane and (b) angular displacement of surface normal vectors from the fitted plane.Residuals are plotted as a function of the projected position of each point onto the fitted plane.

Fig. S. 3 .
Fig. S.3.(a) Photograph of scanned scene which included a glass window as well as surfaces in front of and behind the windw that had varying reflectance.(b) Point cloud acquired for the scene.Marker key is shown on the bottom left.
For comparison, the ground truth plane had parameters ntruth = [−.8825− .0010− .4704]T and d truth = −1.389m.We plot the orthogonal displacement of all estimated mirror plane points with respect to the fitted plane in Figure S.2(a) .We show the angular displacement of the estimated normal vectors from the mean normal vector in Figure S.2(b).The RMS orthogonal distance was 4.7mm and the RMS angular displacement was 0.70 • .

Fig. S. 4 .
Fig. S.4.(a) Point-cloud generated using multi-beam surface mapping algorithm on large mirror dataset (Fig. 4a).(b) Detected per-pixel energy values of multi-beam data collection, overlaid with spot positions.Red circles mark spots classified as two-bounce, and green circles mark spots classified as one-or three-bounce.
Figure S.4(b).Spots were classified by intersecting naive position estimates of all spots with the set of all transmitted beam vectors.Spot positions that lay within some angular distance of a beam vectors were classified as potential one-or three-bounce spots (circled in green in Fig. S.4(b)).The remainder were classified as two-bounce spots (circled in red in Fig. S.4(b)

Fig. S. 5 .
Fig. S.5.We scanned the shape of a glass vase (top left) by directly illuminating a sequence of 60 laser spots on nearby surfaces and measuring the time-of-flight of twobounce returns that reflect off of the pitcher.We show the estimated point cloud from multiple views (top right and bottom row).Blue dots are points on the floor and sidewalls, red dots are points on the vase, and red arrows are the surface normal vectors associated with those points.Because the vase was transparent, we were able to map points on the front and back surfaces of the vase without moving our lidar, or rotating the vase.