Feedback-driven adaptive multi-robot timber construction

.


Introduction
The construction industry has been facing many challenges, including low productivity and stagnant productivity gains over time compared to other industries such as manufacturing [1][2][3], and a significant construction workforce shortage (e.g., more than half a million workers in the United States in 2023 [4]).Moreover, construction jobs frequently require physically demanding tasks such as lifting heavy objects, which have been associated with severe negative impacts on construction workers' health and safety, including work-related injuries and musculoskeletal diseases, among others [5][6][7][8][9].Construction automation has been proposed as a viable solution to address these challenges [2,[9][10][11].Within this context, the introduction of industrial robotic arms for construction with discrete elements has been investigated in research, such as robotic brickwork [12][13][14][15], robotic timber construction [16][17][18][19], and robotic fabrication of spatial metal structures [20,21].
For decades, robotic arms have been employed in manufacturing industries, such as automotive.However, it can be argued that a critical difference between construction and manufacturing industries is the inherent flexibility required to perform non-or quasi-repetitive [9] routines.For instance, in the automotive industry, multiple robotic arms are usually integrated into manufacturing workcells to perform repetitive tasks such as picking and placing metal components and performing spot welding repeatedly [22].Therefore, to achieve consistent and reproducible quality, the robotic processes are well-calibrated (usually taught), optimized, and pre-programmed to perform repetitive tasks with consistent quality and avoid interruptions such as those due to parts collision [22].However, in construction, most buildings and structures are one-offs, which could make repeatability and consistent quality (e.g., tight tolerances) challenging.
There are also other aspects of construction work, such as the relatively large dimensions of building components and structures compared to manufacturing products, as well as the complexity of building assembly due to its inherently spatial and evolving nature, which could make the implementation of single-robot systems challenging.Generally, multi-robot systems have several advantages over single-robot systems that could potentially address these construction work challenges, which include [23][24][25][26]: • The cooperation to complete complex tasks that would not be feasible by a single robot.
• More effective scaling compared to single-robot systems; the robots in multi-robot systems can collectively cover a larger working envelope and handle larger workpieces.• Greater flexibility and adaptability to a broader range of tasks.
• More efficient task completion through parallel processing and simultaneous task space coverage.• Cost-effectiveness of implementing multiple modular robots rather than developing a single-robot system capable of handling complex tasks.• Greater robustness and reliability by implementing redundancy.
However, multi-robot calibration, especially in unstructured environments such as construction sites, could present a significant challenge to achieving process robustness and consistent quality due to any slight discrepancies in their pose estimation.More specifically, adaptive pose correction is typically performed in reference to a fixed world frame, and in the case of a stationary single robot, this world frame is conventionally placed at the robot root, making localization trivial.In the case of multi-robot systems, each robot has its own localization of the world frame.If that localization is not precisely calibrated or coordinated through an extensive sensor system, the robots will fail to precisely match each other's task space.Precise calibration and widecoverage sensor systems are not easily feasible for construction, making the common world frame challenging to define, especially in a dynamic environment prone to sensor perturbations.This challenge is compounded when dealing with uncertainties inherent in construction work, such as material imperfections and fabrication inaccuracies, discussed in the following paragraph.
This research specifically considers construction with discrete elements, reflecting the nature of numerous construction processes that involve the assembly of distinct components, such as brickwork, masonry, timber framing, and steel framing.Within this context, we focus on timber framing with dimensional lumber as the case study construction system due to its widespread use in single-family housing in the United States [27] and its potential for developing low-carbon construction practices [28].A key characteristic of lumber is the uncertainties inherent in this natural material [18,29].First, the crosssections of lumber pieces can deviate from nominal values by several millimeters depending on the manufacturing quality, wood type, and grade of the lumber.Second, lumber elements can shrink and expand due to the variability of their moisture content and the different environmental conditions and temperatures they are exposed to.Third, timber framing typically utilizes members with a minimum length of approximately 2.4 m (8 ft), which are often not straight and usually demonstrate considerable twists and bows.We refer to these characteristics as material imperfections.
Besides material imperfections, there are also other uncertainties inherent in robotic construction, such as cut length and angle deviations after lumber elements are processed using computer numerical control (CNC) machines (e.g., a CNC saw) and their pose inaccuracies (i.e., position and orientation) within a subassembly1 due to robotic system calibrations [18,29].We refer to these uncertainties as fabrication inaccuracies.If not accounted for systematically, material imperfections and fabrication inaccuracies accumulate during the robotic construction of building subassemblies and could result in a considerable deviation between the as-built structure and the as-planned model, leading to parts collision and robot failure when robotic trajectories are derived from the as-planned digital model without considering these deviations [10,11,29].
To address the discussed challenges, we present a feedback-driven framework consisting of two complementary adaptive methods for cooperative multi-robot timber construction to accommodate for perceived uncertainties due to material imperfections and fabrication inaccuracies.Our proposed adaptive methods incorporate robotic perception, reasoning, and acting to handle uncertainties inherent in multi-robot construction work.We evaluate our framework through building-scale construction experiments, quantifying their deviations from their as-planned digital models.While the case-study construction system for the research presented in this paper is timber framing, our methods could be adapted and applied to multi-robot construction processes with other discrete elements, such as bricks and light steel frames, among others.

Related work
This section reviews related work, including cooperative multi-robot fabrication, adaptive robotic fabrication, and adaptive robotic fabrication for timber construction.
Multi-robot fabrication methods for construction work have been actively researched in recent years.Prado et al. developed a cooperative process employing two six-axis industrial robotic arms, enabling the fabrication of geometrically complex modular fiber composite building components with differentiated fiber layouts [31].This project showcased how employing multi-robot systems can enhance the filament winding process by eliminating the requirement for cores or mandrels, which would not have been feasible employing only one robotic arm due to the intricacies involved in the core-less winding process [31].Building on this work, another research project utilized two six-axis industrial robotic arms and a custom-made unmanned aerial vehicle (UAV) to scale up the fabrication setup and process, enabling the manufacturing of a 12-m long fiber composite structure [32,33].This project demonstrated the potential of a multi-robot system for constructing a continuous long-span fiber composite structure without the necessity of dividing the structure into modular components due to the working envelope constraint of individual robotic arms [32,33].
Multi-robot systems have also been investigated for the scaffold-free cooperative assembly of building-scale structures made of discrete elements, such as spatial metal structures [20,21], timber frame modules [18,34], and masonry vaults [35,36].These projects demonstrated the potential of cooperative multi-robot fabrication techniques to perform assembly tasks that cannot be achieved by a single robotic arm, such as addressing the critical necessity to stabilize the in-progress structure during construction with robotic supports and eliminating the need for temporary supports or scaffolding.Conventionally, temporary supports or scaffolding are used to hold the structure during construction, which are removed once the structure becomes self-supporting.This routine extends the construction timeline and increases costs by adding extra steps, including installing the supports, securing the structure to these supports, and their subsequent removal after completion of the structure [18].In a multi-robot system, one or more robots could be used to replace scaffolding by supporting the in-progress structure in strategic locations and stabilizing it during construction while another robot assembles the subsequent element.
While several recent projects [37][38][39][40][41] have investigated nonstandard robotic timber construction, open-loop multi-robot automated timber framing with dimensional lumber has proved challenging due to the material imperfections and fabrication inaccuracies discussed in Section 1, coupled with the tight assembly tolerances required for structural integrity [18,29].Several research projects explored design and fabrication methods to address this challenge, which we review in the following paragraphs.
As discussed, one of the uncertainties is the pose inaccuracy of the robotic arms' end effectors, especially when working on building-scale subassemblies, which could result in a significant deviation between the as-built structure and the as-planned model, potentially causing part incorporated with other units in a finished product" [30].Building on this definition, we define a subassembly as a pre-assembled component made of discrete parts, which will be incorporated with other subassemblies on site to form a building or a structure.imprecision or unforeseen collisions.To minimize the positioning error of end effectors in large-scale multi-robot setups for digital construction, researchers developed a system providing static and dynamic end effector pose correction through external pose tracking [42].Their developed system fuses measurements from an Indoor Global Positioning System (iGPS) with an Inertial Measurement Unit (IMU), which can reduce average static positioning error down to 0.10 mm [42].However, this system has limited application outside of a defined workspace and might not scale well to building-scale construction.Such a system would require significant capital investment for the numerous sensors necessary to cover a large area, as well as extensive calibration, which may be infeasible in a dynamic construction site.Moreover, this system alone is insufficient for addressing material imperfections and fabrication inaccuracies inherent in construction.For instance, this system was used in the cooperative multi-robot spatial assembly of timber frame modules for the DFAB HOUSE [43].While this system was effective for end effector pose correction, the success of the cooperative assembly process relied on human interventions to handle deviations resulting from material imperfections and fabrication inaccuracies [18,34].
Several research projects explored methods to address material imperfections and fabrication inaccuracies.Gandia et al. explored a computational design method to simulate tolerance propagation and minimize potential as-built deviations as part of the assembly sequence planning using a limited dataset of the measured fabrication tolerances of twenty-two cut timber element samples [29].While the authors concluded that this method was useful for estimating the assembly tolerances of two case studies relying on the measured tolerance database of a specific fabrication setup and process, they stated that it would require a much more extensive database to develop predictive models to map the reality of the fabrication processes accurately [29].Furthermore, since this method still works in a feedforward control and an open-loop manner, there are still other fabrication inaccuracies that are not accounted for, such as process forces (e.g., fastening forces) applied to the in-progress structure resulting in unforeseen deviations between the as-built and the as-planned, multi-robot calibrations errors, and deflections of large-scale robotic systems due to temperature variability and other influencing factors (e.g., payload compensation).
It has be argued that robotic fabrication cannot be effectively implemented in a pure feedforward control and an open-loop manner, and adaptive feedback-driven control methods are required to effectively handle material and fabrication uncertainties as well as the construction environment uncertainties [44].Therefore, a robotic fabrication system must perceive the in-progress as-built structure and its deviations from the as-planned model to adapt its plan accordingly [44].More specifically, computer vision technologies such as twodimensional (2D) or three-dimensional (3D) laser scanning and/or force/torque sensing form a basis for robotic perception to acquire information from the robot's environment, which then informs how the robot reasons with its surroundings to adapt its plan to perform its next action in a feedback loop.This approach has been investigated and successfully demonstrated in several research projects, such as cooperative robotic hot-wire foam cutting [45,46], robotic incremental sheet metal forming [47], robotic stone carving [48], robotic wood carving [49], mobile robotic brickwork [15,44], and mobile robotic wire mesh cutting, placing, and welding to fabricate a doubly-curved reinforced concrete wall [44,50].
Looking specifically into adaptive fabrication in robotic timber construction, Helm et al. presented a method to handle material imperfections and fabrication inaccuracies inherent in the single robot assembly of timber structures made from short timber elements [51].Within their fabrication process, the in-progress subassembly is robotically measured, and accordingly, the pose of the subsequent timber element (that is being assembled), as well as the robot trajectory for placing that element within the subassembly, are updated to reduce positioning errors [51,52].The authors conclude that the required tolerance was achieved by using this method for fabricating a multilayered truss system made from short timber elements [51].However, this research includes several limitations, such as working with a single robotic arm and not multi-robot systems and cooperative processes, and working with short timber elements that usually do not exhibit the same twists and bows associated with full-height lumber.Furthermore, neither the mathematical formalization of the adaptive process and its parameters nor the quantification of the resulting tolerances employing this adaptive process was presented and discussed.
Our recent research [53] established a conceptual framework and discussed the preliminary workflow and results, enabling an adaptive method for multi-robot calibration and coordination by eliminating the reliance on a common world frame and instead directly performing adaptive pose correction relative to the as-built structure.In the proposed method, localization occurs in a much smaller region in the robot's task space and is specific to the constraints of the next task, providing a potential framework for multiple robots to cooperate precisely, given an accurate digital model of the structure.
The following section provides a detailed description of the novelty and contributions of this research, advancing our work previously summarized in Ruan et al. [53].

Contributions
In this paper, we provide a comprehensive framework that includes detailed mathematical formalization and process characterization to address the discussed challenges in multi-robot timber construction.The main novelty of this research is the application of adaptive control techniques to multi-robot construction, which, as discussed, typically embody greater levels of uncertainty than manufacturing, where these methods are widely used.More specifically, we employ direct iterative learning control (ILC) [54,55].Our methods parameterize the control inputs and outputs specifically for assembly processes and multi-robot systems, building off of ILC as a departure from standard feedback control.By integrating ILC into our fabrication process, we are able to eliminate reliance on extensive and expensive sensor systems (such as iGPS) for multi-robot calibration and increase the robustness of the construction process by handling both material imperfections and fabrication inaccuracies.
The main contributions of this research are summarized below, which are intended to address the limitations discussed in Section 1.1: • Implementation of a multi-robot system enabling adaptive construction of building-scale timber subassemblies.• Development of a bi-directional digital design-to-fabrication workflow for the feedback-driven framework between an as-built subassembly and its as-planned model.• Formalization and implementation of the framework's two complementary adaptive fabrication methods, pose-based and topologybased, for multi-robot timber construction.• Experimental evaluation of our developed framework through the analysis of fabrication deviations in building-scale physical experiments.

Nomenclature
In our equations, we use bold lowercase and bold uppercase letters to denote vectors and matrices, respectively.A hat symbol ^ indicates an estimated variable, a star ⋆ indicates a desired value taken from the asplanned digital model, and a prime symbol ′ indicates an updated variable.

Methods
The conceptual foundation of our research hinges on developing and implementing a multi-robot timber fabrication system combined with a A. Adel et al. bi-directional digital design-to-fabrication workflow enabling adaptive construction.Building on this conceptual foundation, the methods section is structured to detail four main areas: the multi-robot timber fabrication setup and process (Section 2.1), the digital design-tofabrication workflow (Section 2.2), and two adaptive fabrication methods, pose-based (Section 2.3) and topology-based (Section 2.4).Our multi-robot system employs industrial robots equipped with specialized tools to process and assemble timber elements, while the digital workflow facilitates seamless information exchange between the digital model, the as-built conditions, and the physical fabrication process.Two adaptive methods are introduced to dynamically update the fabrication parameters, ensuring accuracy, efficiency, and robustness.

Multi-robot timber fabrication setup and process
Building on our previous research on multi-robot timber assembly [18,28,34,56], we developed a comprehensive fabrication setup (Fig. 1) to undertake this research.One of the main objectives for this setup was to enable adaptive just-in-time fabrication of bespoke building subassemblies (e.g., floor-height timber frame walls), and, accordingly, we developed the necessary end effectors and additional tools discussed in this section.
Our fabrication setup comprises two six-axis industrial robotic arms with a payload of 120 kg and a reach of 2700 mm, 2 mounted on parallel linear tracks.Each robotic arm has access to an automatic tool-changing station, which enables seamless end effector swapping.In our experiments, each robotic arm utilizes a custom pneumatically controlled gripper end effector to grasp lumber elements with varying profile dimensions (e.g., nominal 2 × 4 or 4 × 6).Additionally, we used a 2D laser profiler 3 for all scanning operations (e.g., scanning the as-built subassembly), and the robotic arms swapped out their grippers with this laser profiler whenever necessary during the fabrication process.
We also designed, engineered, and built a three-axis CNC saw (Fig. 2) and mounted it between the two tracks to be accessible by both robots.The saw has a blade with a radius of 300 mm and a kerf of 6.25 mm.The saw blade can rotate − 180 to +180 degrees and tilt between 0 and 45 degrees, enabling perpendicular cuts as well as a wide range of miter and compound miter cuts.
We located a raised assembly platform between the two tracks to easily clamp a timber element to this platform, secure the subassembly during fabrication, and have the possibility of inserting screws or nails from the bottom face of the structure if needed.The cooperative assembly envelope of this setup (located on the assembly stand) is approximately 5 m in length, 1.8 m in width, and 3 m in height.
Building on our previous research [18,28,34,56], we implemented a prototypical just-in-time multi-robot timber assembly process.This process starts with the human operator loading a stock piece of 2 KUKA KR 120 R2700 [57] 3 LMI Technologies Gocator 2350 [58] dimensional lumber (e.g., a standard 2 × 4 board) along the center of the saw table, with one end roughly aligned with the edge of the table.The active robotic arm then grasps the timber element, which also aligns the stock with the axis of the gripper, locating the stock to a known axis.To perform the first cut, the robotic arm moves the stock into position based on the specified element length and holds the stock in place, while the saw performs the first cut based on the specified cut angles.Subsequently, the robotic arm repeats these steps for the second cut and its respective cut parameters.The grasping frame of the desired timber element is computed to avoid potential collisions between the gripper and the saw blade while minimizing the discarded offcut volume resulting from the first cut and maximizing the remaining material resulting from the second cut for future use.After the cutting process is completed, the robotic arm carries the cut timber element onto the assembly platform and inserts it into its final pose within the subassembly.The human operator then either clamps the element to the assembly platform or fastens the element to previously placed elements with screws before the robotic arm releases the element and retracts.Subsequently, the in-progress as-built subassembly is scanned by either of the robotic arms (depending on the fabrication steps), and the digital model is then updated based on the as-built conditions.The robotic trajectory for the following element and the other robotic arm is adapted accordingly, and the timber assembly process repeats until the subassembly is complete.We will discuss the bi-directional digital design-tofabrication workflow enabling this multi-robot fabrication process in the following section and the detailed perception and adaptation methods in Sections 2.3 and 2.4.

Bi-directional digital design-to-fabrication workflow
Building on our previous research [28], we developed a digital design-to-fabrication workflow (Fig. 3) that enables a bi-directional information flow between the as-planned digital model and the as-built subassembly.This workflow forms the basis of our feedback-driven framework, facilitating the integration of robotic simulation, control, and fabrication into the digital design environment and enabling a seamless transfer of digital information from the dynamically updated computational model to the robotic fabrication process.
This digital workflow is implemented using IronPython [59] within the computer-aided design (CAD) software program Rhinoceros 3D [60] and its plug-in, Grasshopper 3D [61].Rhinoceros 3D and Grasshopper 3D provide interaction with the digital 3D model of the structure and the main graphical user interface (GUI) for this digital workflow.We employed a custom computational design tool, Super Matter Tools (SMT) [62], to simulate industrial robotic processes within Rhinoceros 3D and generate manufacturing instructions.We also integrated a programmable logic controller (PLC, Beckhoff TwinCAT [63]) into our setup for the coordination between different processes and the control of external machines (e.g., the CNC saw), as well as the collection and transfer of sensor data acquired from the laser profiler during the assembly process.
As shown in Fig. 3, the bi-directional digital design-to-fabrication workflow is implemented as a loop, including a forward and a backward process.The forward process of the workflow derives the necessary poses and cut parameters from the digital model of each timber element, which our control algorithm can interpret for trajectory planning, saw configuration (e.g., setting the angles of the saw blade), and gripper states (e.g., grasp or release).The planned trajectory, saw configuration, and gripper states are then automatically post-processed into KUKA Robot Language (KRL) [64] code to be executed by the robotic arms and the CNC saw.
The backward workflow process facilitates the necessary feedback into the as-planned digital model based on perceived as-built conditions and enables adaptive correction of the as-planned model for future fabrication steps.We investigated and tested two adaptive methods for our framework: pose-based and topology-based adaptive fabrication.Sections 2.3 and 2.4 will discuss the specific system formalization, perception, control algorithm, and adaptive correction for these two methods.

Pose-based adaptive fabrication
This section formalizes an adaptive fabrication method for minimizing deviations between the as-planned digital model and the as-built subassembly by tracking element poses.In Section 2.3.1, we first formalize the fabrication process outlined in Section 2.1 into a statespace representation, which sets the foundation for both adaptive methods.We then expand on how a robot perceives a pose by scanning the element extents and reconstructing the element geometry in Section 2.3.2.Next, Section 2.3.3 outlines a direct ILC algorithm to minimize the tracking error between the reference and perceived poses, which iteratively self-corrects for any fabrication inaccuracies in the robotic setup.Finally, Section 2.3.4 details the method for adaptively updating an element's reference pose based on the current as-built conditions, enabling multi-robot coordination with a common localization reference.The overall control process is visualized in Fig. 4.

System formalization
A rigid body pose p ∈ ℝ 6 in 3D space is comprised of a translation (x, y, and z coordinates) and a rotation (roll φ, pitch θ, and yaw ψ Euler angles), relative to the world coordinate system: The discrete fabrication process can be defined as follows: where n = 1, 2, …, N is the element index.x, u, and y ∈ ℝ 6 , are our primary system components, denoting the true as-built pose (which is inherently unknowable), input target pose, and perceived output pose, respectively.w and v ∈ ℝ 6 refer to noise terms: process noise and measurement noise, respectively.A and B ∈ ℝ 6×6 are the uncertain system matrices of the fabrication process.
Because we are working exclusively with position and orientation parameters, we chose to express all pose vectors as homogeneous transformation matrices, which have the benefit of simplifying transformation operations.A pose p can be converted to a homogeneous transformation matrix P ∈ ℝ 4×4 using an intrinsic rotation about the axes z, y, x, in that order, composed of a rotation matrix P R ∈ ℝ 3×3 and translation vector P t ∈ ℝ 3 : We can then convert our fabrication process from Eq. (2) into a system of transformations: where X, U, and Y ∈ ℝ 4×4 are the transformation matrices of x, u, and y, respectively, T 1 and T 2 ∈ ℝ 4×4 are transformation matrices that have combined the uncertainties in A and w, and B and v, respectively.T 1 represents the deviation during the fabrication of an element (which can include gripper calibration errors, the element shifting during fastening, etc.), while T 2 represents the error in the perception process (i.e., errors in the laser profiler calibration and measurement noise).In the following sections, we describe how we perceive the output pose Y(n) and use that to update the input pose U(n + 1) for the next element.

Pose perception
Each element in the as-built subassembly has a corresponding asplanned digital model of that element with a desired pose X ⋆ (n).We use the following perception process to determine its perceived pose Y(n), which then informs the ILC and adaptive pose processes.First, a scan path is computed using the desired pose to estimate where the element is located, which assumes that the fabricated element has not deviated beyond the width of the scanning field of view (FOV).This scan path consists of multiple profiles spaced along the axes of the element (visualized in Fig. 5), which captures the element's extents.In our experiments, we scanned 5 profiles evenly spaced along the length of the element and 3 profiles along each cut end, for a total of 11 profiles.While, in theory, only two profiles are required to extract each edge, the A. Adel et al. additional profiles build robustness against certain material deformations common to timber, such as knot holes and wanes.The resulting point cloud is manually filtered to remove background objects, after which we extract the edges of the scanned element and reconstruct its geometry as a right parallelogrammic prism with a known element height.The position of the perceived pose Y(n) is located at the centroid of this prism, while the x-axis is aligned with the major axis (i.e., along the element length) and the z-axis is perpendicular to the scanned face.

Iterative learning control calibration
After perceiving an element pose Y(n), we utilize ILC to update the input pose U(n).In ILC, the tracking error e(n) is defined as the difference between a reference target pose y r (n) and the perceived output pose y(n), while the primary objective of ILC is to minimize this tracking error over successive iterations [65]: Similar to our fabrication process, we can represent the tracking error using a series of homogeneous transformations: where I is the identity matrix.We can simplify Eqs. ( 4) and ( 6) as follows: where T is the combined transformation matrix between the input pose U and perceived pose Y.Note that homogeneous transform matrices are non-singular and, therefore, invertible.Section 2.3.4 details an extended method for deriving the reference target pose Y r (n) in relation to the asbuilt subassembly, improving the adaptive fabrication method for multirobot systems.The remaining variable we need to resolve before we can compute U(n), then, is T(n).Before beginning fabrication, we initialize T(0) by fabricating a sacrificial element (i.e., an element that is fabricated then discarded) with U(0) = Y r (0): For each subsequent element, we first compute an estimate T(n) by evaluating the fabrication process's memory and taking the average of the previous transformation matrices {T(0) , T(1) , …, T(n − 1) }.To compute an average, each transformation matrix T(k) with k = 0, 1, …, n − 1 is broken down into its rotation matrix T R (k) and translation vector T t (k).Taking the average of the translation vectors, denoted as Tt (n), is straightforward.However, the rotation matrices cannot be directly averaged element-wise since that may result in an invalid rotation.Instead, we use singular value decomposition (SVD) to compute the average rotation matrix, which ensures that the resulting rotation matrix is still valid [66]: The components of T(n), TR (n) and Tt (n), can then be reassembled into a homogeneous transformation matrix following Eq.( 3), and we rewrite Eq. ( 7) using our estimation of T(n) to determine U(n): Finally, the element is fabricated, and the actual value of T(n) is computed in Eq. ( 12) using the measured output Y(n), and stored in memory to compute the next element's input U(n + 1) using Eq. ( 11):

Adaptive pose correction
The above ILC process is normally sufficient to minimize deviations in a single-robot fabrication process.However, the remaining elements would be misaligned if the as-built subassembly were to be perturbed (e.g., accidentally shifted while fastening).Moreover, in the case of a multi-robot system, the accuracy of each robot's localization to the world frame can vary.Our ILC process improves pose accuracy relative to the robot root frame and, therefore, does not directly address coordination between multiple robots.To remedy this, we devise an adaptive pose correction process where multiple robots reference a common object for localization, namely the as-built subassembly.For each robot, there exists a global transformation G ∈ ℝ 4×4 that localizes the perceived output to the world frame, which minimizes the deviation between the as-built subassembly and as-planned digital model: where N is the total number of elements, and X ⋆ is the as-planned pose, taken from the digital model.We can estimate G for each element after the first by evaluating the deviation of the previous element from the asplanned digital model: This global transformation allows us to derive an updated reference target for each step of the ILC outlined in the previous section, localizing each robot to the as-built subassembly and remedying any inaccuracies in the calibration of the robot's root frame relative to the world frame:

Topology-based adaptive fabrication
In this section, we formalize an adaptive fabrication method for minimizing deviations at the joints between elements using the topology of the as-planned digital model.Here, we refer to topology as the relationship between elements within a subassembly and how elements must be fastened together to form a structural connection.This approach acknowledges that ensuring a proper connection with current as-built elements is more important than its absolute pose for many elements in an assembly task.For example, in a timber wall frame, a stud should be flush with the surface of the wall, and its ends should be cut flat against the top and bottom plates.The stud's pose has a small degree of freedom within these constraints.We first formalize the cut parameters for an element in Section 2.4.1.Section 2.4.2 describes how we scan a patch of the as-built structure, namely the contact surfaces where the element must interface.Finally, we update the element's combined state (pose and cut parameters) to fit the scanned contact surfaces in Section 2.4.3.The overall control process is visualized in Fig. 6.

Cut parameter formalization
Each cut for element n has parameters q(n, i), where i is the cut index and i = 1, 2 for a timber element with a cut on either end.For simplicity of expressing the equations in this section, we refer to q i ≡ q(n, i), which can be defined as: where i is the cut index, d is the distance from the element pose to the center of the cut face along the central axis, and α and β are, respectively, the input rotation and tilt angles of the saw used to make this cut (see Fig. 2).Similar to our formalization of the pose-based adaptive fabrication method (Section 2.3), we use homogeneous transformation matrices to simplify the transformation operations.Given the cut parameters q ⋆ i from the as-planned digital model, we compute a rotation matrix Q ⋆ R,i ∈ ℝ 3×3 based on the desired pose X ⋆ of the element: where: We can locate the as-planned endpoint Q ⋆ t,i of the cut surface (see Fig. 7) by offsetting the as-planned element pose along its x-axis by the cut distance d ⋆ i : with μ(i) indicating the offset direction based on the cut index i: By combining the rotation matrix Q ⋆ R,i and endpoint position Q ⋆ t,i using Eq. ( 3), we obtain a transformation matrix for the desired cut frame Q ⋆ i .The topology-based adaptive fabrication process aims to adapt this cut frame to match the scanned contact patch (with a slight tolerance offset), as described in Section 2.4.2.Once we obtain an updated cut frame Q' i and pose X ′ , as explained in Section 2.4.3, we can extract the cut parameters for input into the fabrication process as follows: ) where Γ ′ i j,k is the element in the j-th row and k-th column of the matrix Γ' i and ‖ ⋅ ‖ is the Euclidean norm.

Patch perception
We first identify each element's contact points in the subassembly based on the topology of the as-planned digital model.In the most common case, an element will have two areas of contact, each requiring a cut to form a butt joint at either end.We extract the geometry of this contact patch from the as-planned digital model and generate a scan path consisting of three profiles perpendicular to the length of the connecting element (see Fig. 7).We manually filter the scan to remove background objects, after which we compute a best-fit plane F i for the contact patch corresponding to cut i using linear least squares [67], which has its normal directed outwards from the connecting element surface (i.e., the dot product between F N,i and positive z-axis of the Fig. 7. Element contact patch perception process, with locating the as-planned contact patches (left), scanning of the current as-built subassembly (middle), and processing of the resulting point cloud to extract the updated cut parameters (right).

Fig. 6. Overview of the topology-based adaptive fabrication method.
A. Adel et al. scanner is negative).We also extract the line segments ℓ 1,i and ℓ 2,i that define the edges of this patch, where the endpoints of each line segment ℓ j,i are a j,i and b j,i ∈ ℝ 3 and co-planar on F i .From these edges, we can locate the central axis ℓ 0,i projected onto the contact patch, which is formed from the midpoints of lines a 1,i a 2,i and b 1,i b 2,i .

Adaptive state correction
From the as-planned digital model, we obtain the element pose X ⋆ and cut parameters q ⋆ i for each cut i = 1,2.As described in Section 2.4.1, we convert these cut parameters to transformation matrices Q ⋆ i .To update these parameters and align the element with the scanned as-built contact patch, we first orthogonally project the cut endpoint to the central axis of the contact patch: with v i = b 0,i − a 0,i and Δ i = F R,i [0 0 δ] T an additional tolerance offset along the contact surface normal to ensure proper fit during element insertion.
We have determined δ = 1 mm as a suitable value to avoid collision during element insertion through previous experimentation.To compute the updated rotation matrix Q ′ R,i for the cut parameters, we can use the central axis vector v i and the normal of the contact patch F N,i : where: and μ(i) is a sign function based on the cut index as used in Eq. ( 18).The final updated cut plane Q ′ i can be constructed as a homogeneous transformation matrix similar to Eq. ( 3) using Q ′ R,i and Q ′ t,i for each cut.With both cut planes defined, we can update the element pose X ′ as: )/ 2 where: and GS(⋅) is the Gram-Schmidt process [68] applied to the columns of the matrix to orthonormalize the rotation axes.In this formulation, we align the x-axis of the updated pose X ′ with the new central element axis connecting the two cut planes Q ′ i , and the y-axis is close to the asplanned element y-axis X ⋆ R,y .
We obtain the updated cut parameters α ′ i , β ′ i , and d ′ i from the cut planes Q ′ i and element pose X ′ following Eq.( 19).Finally, for the pose input U(n), we retrieve the last calculated value of T from memory, computed from previous iterations of the pose-based adaptive fabrication method (Section 2.3): Note that because the topology-based adaptive fabrication process does not scan the entire element to measure the perceived output pose Y, we are unable to complete the ILC steps outlined in Section 2.3.3, and thus the self-calibration is assumed from previous elements where the ILC process was used.The perceived output pose was not measured due to technical challenges in generating collision-free scan paths over the element extents when maneuvering in confined spaces, such as during the assembly of spatial elements.However, this can be addressed in future work with more advanced perception techniques and additional learning methods.

Experimental setup
We conducted two sets of case-study prototyping experiments to evaluate the developed framework and adaptive methods discussed in the previous sections.The first set of experiments utilizes the pose-based adaptive fabrication method outlined in Section 2.3 in a simple timber stacking assembly, focused on minimizing deviations in the pose of each element.The second case study is a spatial timber wall frame assembly task utilizing the topology-based adaptive fabrication method outlined in Section 2.4, focusing on minimizing deviations at the element connections.
In the first experiment, the robotic setup described in Section 2.1 was tasked to fabricate a flat, 1-m-wide nail-laminated timber (NLT) panel by perpendicularly cutting, stacking, and fastening 10 elements cut from standard 2 × 4 dimensional lumber, alternating between both robots.In applying the pose-based adaptive fabrication method, we tested two scenarios: (1) neither robot adapts (base case used for benchmarking, Fig. 8), and (2) both robots utilizing the pose-based adaptive fabrication method (Fig. 9).While this assembly task is relatively simple, the experiment demonstrates the inherent additional deviations that can  occur in a multi-robot assembly system due to discrepancies of their world frames, and how we can correct for them within a restricted number of degrees of freedom (planar translation and rotation).
In the second experiment, the robotic setup was tasked to fabricate a full-scale spatial timber wall module (Figs. 10 and 11).The module consists of 13 elements connected via butt joints and measures 2.05 m long by 2.32 m tall.This experiment demonstrates the potential and necessity of multiple robots during fabrication to stabilize the inprogress subassembly during each step [18,34].In this type of assembly, it is common for the floor and top plate to have some material deformation, causing parts collision when inserting the studs without adaptation during robotic fabrication and interrupting the fabrication process.In this experiment, we utilized the developed topology-based adaptive fabrication method to ensure the assemblability of the wall module.
After the completion of each subassembly in both experiments, we scan the as-built structure using the laser profiler end effector to collect a high-resolution point cloud, which is used in the following section to analyze the surface deviation from the as-planned digital model.The profile resolution of the laser profiler is 0.150 mm, and the depth resolution is 0.019 mm, with a linearity of ±0.01% of the measurement range.To scan the structure, we complete multiple passes of the laser profiler in overlapping bands.The laser profiler has a limited width for scanning (365 mm far field of view); therefore, multiple passes are required to cover the extent of the surface.These partial scans are then stitched together using the recorded pose of the laser at each scan.

Data analysis
This section reviews the methods used to analyze the data collected from the experiments.In the NLT panel experiment, we evaluate the ILC process of Section 2.3.3 by analyzing the tracking error described in Section 3.2.1.For both experiments, we evaluate their respective adaptive methods, pose-based adaptive fabrication (Section 2.3) for the NLT panel experiment and topology-based adaptive fabrication (Section 2.4 for the wall module experiment, by measuring the deviation between the scanned point cloud of the as-built subassembly and the as-planned digital model, as described below in Section 3.2.2.We can identify how accurately the adaptive fabrication methods matched the as-planned digital model by evaluating the point cloud deviation despite material imperfections and fabrication inaccuracies.

Tracking error
During the fabrication of the first experiment, the reference pose y r and perceived output pose y are recorded for each element.We can subtract the two to calculate the tracking error in Eq. ( 5).Due to the design of the subassembly and constraints of the fabrication setup, however, the degrees of freedom for adaptation are limited, causing some tracking errors along certain dimensions to be inevitable.More specifically, we constrain the tracking error analysis to 2D because the z height and roll and pitch rotations are all subject to material variation in the stacking assembly of the NLT panel and cannot be adapted for in the current fabrication process (e.g., we cannot plane the element to reduce its thickness or flatten its surface).We, therefore, focus primarily on analyzing the positional tracking error along the x and y axes, measured as the Euclidean distance in the xy plane, and the yaw rotational tracking error θ, measured as an absolute value.

Point cloud deviation
For both experiments, we scan the complete as-built subassemblies as described at the end of Section 3.1, and analyze the point clouds to identify deviations from the as-planned digital models.Due to the position of the subassemblies on the assembly platform and the robot's kinematic constraints, we could not scan both sides to form a complete point cloud.However, a single side is sufficient to measure deviations.
The scanned point clouds are first filtered manually to remove background objects.We then approximate the global transformation G of Eq. ( 13) that best fits the as-planned digital model to the scanned point cloud over the entire structure through an iterative closest points (ICP) algorithm [69].This algorithm requires a point cloud of the reference model -we convert the as-planned digital boundary representation (BREP) model to a mesh and utilize MeshLab's Poisson-disk sampling to generate a well-distributed point cloud over the surface [70,71].We apply the output transformation from the ICP to the mesh model and measure the closest distance from each point in the scanned point cloud to the mesh model.The closest distance can be computed by first indexing the mesh vertices into a K-D tree [72] using SciPy [73,74], querying the closest vertex on the mesh to each point in the point cloud, then calculating the minimum distance to each triangle associated with that vertex [75,76].

NLT panel experiment
Table 1 lists the translational and rotational tracking errors for each timber element of the NLT panels for the benchmark and adaptive scenarios in the first experiment, which are computed using the method described in Section 3.2.1.This table also summarizes the average and standard deviation of the translational and rotational tracking errors for each scenario.Surface deviations between the as-built subassembly and the as-planned digital model for each scenario are visualized as a heat map and plotted in a histogram with bin size 0.1 mm in Fig. 12 using the methods detailed in Section 3.2.2.The average surface deviation of the benchmark scenario is 1.42 mm, with a median of 1.12 mm and a standard deviation of 1.16 mm.The average surface deviation of the adaptive scenario is 0.73 mm, with a median of 0.67 mm and a standard deviation of 0.58 mm.
While this particular experimental structure could be completed with a single robot, we wanted to utilize this simple assembly task to demonstrate the difference between single-and multi-robot assembly.If we isolate the odd-and even-numbered elements in the benchmark scenario, we can track the individual performance variance of the North and South robots, respectively (Fig. 1).For translational deviation, the North robot had a standard deviation of 0.22 mm, and the South robot 0.39 mm.For rotational deviation, the North robot had a standard deviation of 0.013 • , and the South robot 0.034 • .Comparing these values to  2 Absolute value of the yaw rotation ψ about the z axis.
the combined standard deviations (Table 1) highlights the additional fabrication inaccuracies that can occur when utilizing multiple robots and why adaptive methods are required beyond individual robot pose correction.Many construction tasks, such as the following timber wall module experiment, cannot be completed by a single robot, as mentioned in Sections 1 and 1.1, so addressing these deviations becomes critical.
As shown in Table 1, our developed pose-based adaptive fabrication method demonstrated a significant improvement in reducing translational and rotational deviations compared to the benchmark scenario with no adaptation.The average translational deviation showed a marked improvement from 1.28 mm to 0.40 mm, and the rotational deviation decreased from an average of 0.24 • to 0.09 • .However, we also observed that this adaptive method could result in progressive deviations of the as-built elements from the as-planned digital model as the assembly process progresses (Table 1).This is likely because the ILC process improves over time with more data, and any perceived pose errors in the first few elements have a greater impact on the adaptive method.This could be mitigated by utilizing a larger initial dataset (e.g., a sample dataset produced from previous fabrication tasks).Another reason could be the potentially inaccurate measurement of the perceived as-built pose due to the sparse profile scanning method (Section 2.3.2),which could be prone to uncertainties due to larger material imperfections such as lumber twists and bows.In future research, we could alleviate this by extracting the as-built pose from a higher-resolution point cloud, increasing the robustness of the pose perception; however, this would increase the computational complexity of the overall system.

Wall module experiment
We evaluated the point cloud deviation between the final as-built wall module and the as-planned digital model using the methods detailed in Section 3.2.2.There were two input point clouds for the ICP algorithm: the filtered scanned point cloud and the sampled model point cloud, which contained around 1.8 million and 1.5 million points, respectively.The deviations are visualized as a heat map and plotted in a histogram (Fig. 13) with a bin size of 0.2 mm.The average deviation is 2.43 mm, with a median of 2.06 mm and a standard deviation of 1.97 mm.
The results of the wall module experiment demonstrate the potential for our feedback-driven framework to address accumulative tolerances inherent to construction, resulting from manufacturing inaccuracies and material imperfections over the assembly process.Accumulative tolerance is one of the main reasons for parts collision during robotic fabrication and would have resulted in a failed wall module.However, since we applied our developed adaptive topology-based correction in each step of the assembly process, the robotic process could adapt the fabrication steps according to the perceived deviation of the as-built subassembly and successfully fabricate the wall module without parts collision.
While our topology-based adaptive method proved effective for successfully fabricating the wall module, demonstrating the robustness of the assembly process against fabrication inaccuracies, we observed that the deviation in the top plate of the wall increases towards the right, as illustrated in Fig. 13.This is likely due to the over-compensating effect of shortening the elements.The wall module was built from left to right, and fastening each successive stud pulled the top element slightly lower, creating a lever and tilting it down slightly.This downward tilt could cause the next element to be slightly shorter and exacerbate the downward pull when fastened.One approach to address this issue and increase the precision of the topology-based adaptive method would be to revise the assembly sequence and assemble the wall's two edge studs, followed by the top plate of the wall, and then assemble the infill studs [18,34].This assembly sequence decreases the potential for deviations due to the overhang, but increases the complexity of the robot trajectories required to insert stud elements without collision.Another approach would be to modify the assembly trajectory path such that studs are inserted at an angle and rotated into place, correcting for any deviations in the top plate by pushing it back upwards.To achieve such a process, more advanced motion planning and task allocation would be required, alongside force/torque sensing, which is left as a future point of research.

Overall discussion
We see an opportunity for employing the pose-based adaptive method to fabricate elements where their pose relative to the world is critical, such as the bottom and top plates of a wall, since they might be interfacing with other components of a building or structure and require a tight tolerance.On the other hand, we see the potential of the topology-based adaptive method for fabricating timber elements that fill out the assembly where the tight fit between elements is critical (e.g., for structural performance).As part of future research, the two adaptive fabrication methods can be integrated, combining the precision afforded by the pose-based adaptive method with the robustness of the topologybased adaptive method.We envision that our developed feedbackdriven framework can minimize the effect of material imperfections and fabrication inaccuracies in a completely autonomous multi-robot fabrication setup.

Conclusion
We presented a framework consisting of two feedback-driven adaptive methods, pose-based and topology-based, for cooperative multirobot timber construction to effectively handle perceived uncertainties due to material imperfections and fabrication inaccuracies.Our proposed framework relies on using the as-built subassembly as a localization target for self-calibration and fabrication adjustment, avoiding issues of calibrating to a common-world coordinate system using external measurement equipment.In particular, our developed posebased adaptive fabrication method proved effective in decreasing deviations of the as-built NLT panel from its as-planned digital model for a cooperative assembly task performed by two robots.Furthermore, our topology-based adaptive method enabled the successful completion of a wall module, which was fabricated cooperatively by two robots.Although the case study of our research was timber framing, our methods could be extended and utilized in multi-robot construction processes involving other discrete elements such as bricks and light steel frames.Therefore, this research contributes to the body of knowledge required to facilitate flexible and accurate cooperative multi-robot construction at the building scale.The following section presents the limitations of this research and potential avenues for further investigation.

Limitations and future work
While we focused our studies on a two-robot system, the research could be easily extended to multi-robot systems, and our developed feedback-driven framework could be applied to any number of robots.This extension will require further investigation by developing an assembly setup that includes more than two robotic arms and conducting prototyping experiments to evaluate the methods empirically.Furthermore, the research could be extended to on-site construction.However, besides material imperfections and manufacturing inaccuracies, uncertainties inherent in unstructured construction sites should also be integrated into the adaptive methods.
Addressing these construction site uncertainties will require more sophisticated perception techniques.The pose perception process in this research still included manual processing steps due to setup limitations and can be made more robust with filtering and segmentation computer vision methods.Further development of the perception process also allows for the potential to utilize machine learning to not only automate pose perception but also optimize fabrication steps based on available stock, environmental conditions, and design intent.
Fabrication speed is also another consideration for future research; our experiments operated the robots at a fairly slow speed (maximum 0.25 m/s Cartesian speed) in consideration of safety, and speed was not critical to our research scope.However, increasing the robot speed and evaluating the fabrication process' productivity will be crucial for industry adoption, and will likely introduce additional research questions regarding real-time motion planning and tool development.
Moreover, we focused our research only on lumber studs and assumed negligible deformations during the robotic assembly process.Future research could investigate multi-material timber subassemblies and extend the adaptive methods to handle materials, such as plywood sheathings, that could exhibit considerable deformation.This extension will enable the assembly of fully prefabricated wall, floor, and ceiling subassemblies.We see an opportunity to investigate multi-modal perception coupling vision and force/torque sensors and robot learning to develop real-time adaptive robotic manipulation and assembly of deformable building materials.

Declaration of competing interest
Arash Adel reports financial support was provided by National Science Foundation.Wesley McGee reports financial support was provided by National Science Foundation.

Fig. 3 .
Fig. 3. Overview of the bi-directional digital design-to-fabrication workflow and process.(This diagram does not include the initial calibration element 0, as described in Section 2.3.3).

Fig. 4 .Fig. 5 .
Fig. 4. Overview of the pose-based adaptive fabrication method, including the developed ILC and adaptive pose correction processes.The control symbol denoted by Π represents matrix multiplication, with inverted inputs indicated by × − 1 .

Fig. 8 .
Fig. 8.The benchmark experimental scenario with both robots alternating to assemble an NLT panel without adaptation.

Fig. 9 .
Fig. 9.The adaptive experimental scenario, with both robots alternating to assemble an NLT panel using the developed pose-based adaptive fabrication method.

Fig. 10 .
Fig. 10.The second experiment with both robots cooperatively assembling a full-scale timber wall frame utilizing the topology-based adaptive fabrication method, showing the fabrication steps (top) and the finished module (bottom).

Fig. 11 .
Fig. 11.Contact patch scanning of the as-built subassembly in the topologybased adaptive fabrication method.

Fig. 12 .
Fig. 12. Deviation from each point in the scanned point cloud of the as-built subassembly to the as-planned digital model, visualized in a histogram (top) and a heat map (bottom) for the benchmark scenario with no adaptation (a) and with both robots utilizing the pose-based adaptive fabrication method (b).A portion of the point cloud is missing from the bottom element in both scenarios due to occlusion from the clamping mechanism.

Fig. 13 .
Fig. 13.Deviation from the scanned point cloud of the as-built wall module to the best-fit transformation of the as-planned digital model.

Table 1
NLT panel experiment tracking error.
1Euclidean distance in the xy plane.