Procedure for the Orientation of Laser Triangulation Sensors to a Stereo Camera System for the Inline Measurement of Rubber Extrudate

: Rubber production is a labour-intensive process. In order to reduce the needed number of workers and the waste of material, the level of digitalisation should be increased. One part of the production is the extrusion to produce gaskets and similar objects. An automated observation of the continuous rubber extrudate enables an early intervention in the production process. In addition to chemical monitoring, the geometrical observation of the extrudate is an important aspect of the quality control. For this purpose, we use laser triangulation sensors (LTS) at the beginning and the end of the cooling phase of the extrudate after the extrusion. The LTS acquire two-dimensional profiles at a constant frequency. To combine these profiles into a three-dimensional model of the extrudate, the movement of the extrudate has to be tracked. Since the extrudate is moved over a conveyor belt, the conveyor belt can be tracked by a stereo camera system to deduce the movement of the extrudate. For the correct usage of the tracking, the orientation between the LTS and the stereo camera system needs to be known. A calibration object that considers the different data from the LTS and the camera system was developed to determine the orientation. Afterwards, the orientation can be used to combine arbitrary profiles. The measurement setup, consisting of the LTS, the stereo camera system and the conveyor belt, is explained. The development of the calibration object, the algorithm for evaluating the orientation data and the combination of the LTS profiles are described. Finally, experiments with real extrusion data are presented to validate


INTRUDUCTION
Digitizing rubber production is of great ecological and economical interest due to high amount of waste and input material.One part of the production is the extrusion.Uncured material is funnelled into the extruder and forced by a screw under controlled temperature through a die that defines the profile shape of the extrudate.The extrudate is a continuous material flow and can be cut to length after the cooling and vulcanisation of the material.Different parameters such as temperature, pressure and the ingredients of the base material affect the quality of the final product (Princi 2019).This research is part of the DIGITRUBBER project, organised under the platform MaterialDigital (2023).Goal of the project is the digitalisation and the support of the rubber production via autonomous measurements especially inline during the production process.In addition to chemical monitoring, the geometrical observation of the extrudate is an important aspect of quality control.In this study, multiple laser triangulation sensors (LTS) were used for that purpose.An LTS consist of a laser diode, a cylindrical lens and an image sensor with known orientation.The diode creates a laser plane, which is deformed depending on the distance between the sensor and the object.The image sensor registers these deformations and a twodimensional profile is calculated.In order to receive a complete model of the object, either the object or the sensor needs to be moved and the motion has to be tracked (Luhmann et al. 2023).After the extrudate leaves the extruder, it is transported on a conveyor belt.Two systems, each consisting of several LTS, are used to register profiles at the start and the end of the conveyor belt.This way, the change of the extrudate's geometry before and after the cooling can be monitored.For a better comparison and comprehension of the swelling behaviour, creating threedimensional models is useful.This is achieved by combining the two-dimensional profiles.A stereo camera system is used to track the required movement for the combination.To properly use the tracking information, the orientation between the LTS system and the camera system must be known.Determining the orientation is a non-trivial task due to the lack of overlap in the field of view of the two systems and the different measurement characteristics.This challenge was solved by the development of a calibration object and an algorithmic workflow to calculate the orientation of the systems by measuring the calibration object.The results can be used to combine real profiles of rubber extrudate.Two approaches using the orientation are developed.One uses the nominal geometry of the conveyor belt to calculate the absolute position of the extrudate.The second one uses only the data of the stereo camera system to track the movement of the conveyor belt sequentially.In an alternative approach, the moved distance is extracted from the image data directly and used to combine the profiles along the sensor plane.The following sections are organized as follows: After providing perspective on related work, regarding quality control via LTS systems and the support of the rubber production, the measurement setup, and algorithms towards profile orientation are described.The proposed methods are then evaluated against other approaches, results are discussed and the paper concludes with possible further research ideas and improvements.
Figure 1.The measuring setup consisting of two LTS systems, a stereo camera system on the top of the left LTS system and the conveyor belt, the extrudate moves from left to right; different coordinate systems for the cameras and the LTS have to be considered.

RELATED WORK
LTS are widely used for monitoring various production processes and quality control.Cinal et al. (2023) used LTS to assess wooden planks.After a plank was placed on a conveyor belt, it was moved and an encoder triggered the LTS at regular intervals.To combine the profiles, it is required that the conveyor belt and LTS are aligned precisely so that the laser plane is orthogonal to the direction of the conveyor belt movement.The physical adjustment of different systems and its maintenance is a difficult process, particularly in a real production environment.Alternatively, the orientation can be measured and taken into account during data processing.Huang et al. (2021) used LTS to monitor the production of metal objects based on wire arc additive manufacturing.They used a rectangular calibration object to calculate the orientation between the LTS coordinate system and the manufacturing coordinate system beforehand.Lafirenza et al. (2023) and Zou et al. (2023) used multiple spheres to create calibration objects to improve the quality of a point cloud based on LTS data or to orientate multiple LTS.These methods focus mainly on pure LTS data and do not consider additional sensors.A concept to monitor the quality of a rubber extrusion process via LTS is shown in Karunasena and Wickramarachchi (2010), but only using the two-dimensional profiles.In a laboratory experiment, they used a simple setup of a web camera and a line laser to measure trapezoidal shaped rubber profiles.Perlo et al. (2016) applied magnetic resonance imaging to measure the inner and outer profiles of an extrusion process.This way, wall thickness and internal cavities can be monitored.They used image processing to upscale the comparatively low resolution of the system and extract profiles automatically.Vukicevic et al. (2019) developed a semi-automatic system as a replacement for profile projectors.A camera captures the image of a slim slice of the rubber extrudate.Rubber profile contours are extracted automatically and can be compared to the technical drawing.Liguori et al. (2004) applied a stereo camera system to measure the profiles of ready to use extrudates.First, they are tracking the contours in each image via edge-based segmentation.Then three-dimensional points are extracted and projected onto a plane afterwards to get an examinable profile.Cabera et al.
(2010) developed a system to support the operation of the extruder to increase the quality of the product and reduce the waste.The real-time monitoring considers parameters such as pressure, temperature and velocity, and enables an early intervention by the operator.A collection of sensors to monitor these parameters is found in Abeykoon (2020).While most of the referenced research in the context of rubber extrusion focuses on two-dimensional data, in this contribution, approaches are described that combine LTS profiles to threedimensional point clouds.By orienting LTS to a stereo camera system, an alternate method to encoders is presented that does not need a precise adjustment between LTS and conveyor belt, and opens up new opportunities for the combination of sensor data in the future.

MEASURING SETUP AND CLIBRATION OBJECT
Our developed method is specified for a combination of a measurement setup and a calibration object.A stereo camera system, two LTS systems and a conveyor belt are the main components of the measurement setup.During multiple iterations, the calibration object made of construction profiles, high-precision Polyoxymethylene (POM) spheres and photogrammetric markers was developed.

Measurement Setup
The measurement setup for the geometrical monitoring consists of two LTS systems and a stereo camera system.One LTS system was placed at the beginning of the conveyor belt directly after the extrusion and the other one was located at the end of the conveyor belt, following a cooling phase (Figure 1).The LTS systems have been developed and built by the Institute of Measurement and Control Technology of the Leibniz University Hannover, and incorporate multiple individual LTS.The orientation between the sensors was calibrated and all profiles were transformed into a common coordinate system.On top of the first LTS system, the stereo camera system was placed.It is made up of two machine vision cameras which capture the first third of the conveyor belt.The cameras were calibrated and oriented to measure 3D coordinates of photogrammetric markers on the calibration object or the conveyor belt.This way, the movement of the calibration object or the extrudate was tracked.During the orientation measurements, the velocity of the conveyor belt was controlled for minimum slip of the extrudate.Additional measurements were carried out while the extrudate is moving over the conveyor belt.The measurements of the LTS systems and the stereo camera system were synchronised, but operate at different sampling frequencies, so that only for every other profile a RGB image is captured.In between, the tracking parameters were interpolated.The combination of a profile and its respective image pair is called an epoch in the following.

Calibration Object
There are two main conditions to consider when constructing the calibration object.Firstly, the orientation has to be performed even though there is no overlapping field of view between the stereo camera system and the LTS systems.
Secondly, the difference in the registered data has to be considered.While the LTS system only measures twodimensional profiles with no other information, the stereo camera system measures 3D coordinates of photogrammetric markers.The solution is a combination of spheres and a platform with markers connected by aluminium construction profiles (Figure 2).The advantages of the spheres are the possibility to combine neighbouring profiles to calculate the centre point and the sphericity as a distinct quality and comparative parameter.The used spheres have a nominal radius of 10 mm.For the orientation algorithm a 3D model of the used calibration object is needed to extract the centre points of the spheres and the 3D coordinates of the photogrammetric markers.All coordinates must be known in the same coordinate system.In the carried-out experiments, a handheld 3D scanner was used to acquire the necessary information.
Figure 2. Two calibration objects with the photogrammetric markers for the camera system and spheres for the LTS systems.

ALGORITHMS FOR ORIENTATION AND COMBINATION
Three algorithms to combine LTS profiles were developed.Two of them need a previous orientation step using either a geometry-based or a sequential approach for the combination, while the other combines the profiles along the normal of the LTS sensor plane.During the orientation, the described calibration object is used.It has to be carried out outside the regular rubber extrusion process.All algorithms were implemented in Python.In addition to the standard libraries, OpenCV (Bradski 2000), Open3D (Zhou et al. 2018) and SciPy (Virtanen 2020) were used.

Orientation
Before extrudate profiles of a real inline measurement can be combined, the orientation between the camera system and the LTS systems has to be calculated.Multiple coordinate systems have to be considered.The first one is the camera coordinate system.All points that are calculated from the image pairs are described in this coordinate system.If a marker is moved between two epochs, its coordinates change.The 3D model of the calibration object is described in the calibration coordinate system.In this system each marker has a distinct coordinate and the centre points of the spheres are known.Every marker has an individual code, so homologous points can be found easily.
With at least three of them, the points in the camera coordinate system can be transformed into the calibration coordinate system.If the transformation parameters from the camera coordinate system into the calibration system for one epoch are known, every point in the camera coordinate system of this epoch can be transformed into the calibration system.However, the points of the profiles are originally only known in their own LTS coordinate system.The goal of the orientation is to calculate the transformation matrix to transform the profiles into the camera coordinate system (Figure 3).Since the position of the cameras and each LTS system is fixed, one transformation for all profiles should be valid.The first step of the orientation is the data acquisition.The calibration object is placed on the conveyor belt.Then it is moved as deep as possible into the LTS system.Now, the measurements are started.The calibration object is slowly moved out of the LTS system.Besides the measured data, the algorithm requires the coordinates of the photogrammetric markers and the centres of the spheres in the calibration system.First, the 3D coordinates of the photogrammetric markers, including their unique codes, for each epoch are extracted from the image data.Using the homologous points from the 3D model, tracking parameters are calculated.The tracking parameters consist of three translations and three rotations to describe the Helmert transformation (fixed scale) from the camera coordinate system to the calibration coordinate system.Without processing, all profiles are located on one plane (Figure 4a).To extract spheres, a three-dimensional representation is required.An approximated transformation matrix is calculated by matching the direction of the normal vector of the LTS plane to the direction of the mean move vector of the markers.It should be noted that only the directions are matched.The matrix cannot be used to transform the LTS profiles to another coordinate system.After the profiles are transformed by this matrix, the tracking can be applied to get an approximated 3D point cloud (Figure 4b).From the point cloud, spheres can be extracted (Figure 4c).A method based on Honti et al. ( 2020) is used.A random seed point and a fixed number of its nearest neighbours are chosen.
Based on these points a sphere is calculated considering the nominal radius.All points around the centre point within the radius are searched, and an improved sphere is calculated.The mean distance between the points and the sphere is used as the quality parameter, called the sphericity.If the sphericity is below the limit, the sphere is accepted and all points in the point cloud are deleted.If the sphericity exceeds the limit, the sphere is ignored.New seed points are chosen, until the required number of spheres is reached.At least three spheres are needed for the next steps to calculate the transformation into the calibration coordinate system.The sphere points are transformed back into the LTS coordinate system (Figure 4d).This way, different independent transformations can be applied in following steps.The 3D point cloud is transformed into the calibration coordinate system via the homologous centre points (Figure 4e).To determine them, a RANSAC approach is performed to find homologous spheres.Subsequently, the sphere points are transformed into the camera coordinate system via the inverted matrices of the tracking parameters (Figure 4f).The sphere points are also known in the LTS coordinate system, so that the transformation from the LTS coordinate system into the camera coordinate system can be calculated.The resulting transformation matrix is used in two of the three combination algorithms.

Combination
Two approaches to combine the profiles considering the orientation were developed.The first one requires the nominal geometry of the conveyor belt which is measured independently beforehand.Alternatively, the tracking can be done sequentially without a nominal geometry extracting the needed information only from the image data.This way the time-consuming measurement of the conveyor belt can be avoided, but it is less controlled, and an error in the tracking influences all following epochs.
All combination algorithms use photogrammetric markers on the conveyor belt to track the movement.The markers are placed at the left and right edges of the conveyor belt, so that the extrudate can be placed between them.During the measurement of the conveyor belt for the geometry-based algorithm, it is important, that at least four points at the start and the end are captured twice.These homologous points are used to expand the nominal geometry to consider multiple circulations of the conveyor belt and fit the number of registered cycles, which can be done by considering the point numbers.
The tracking parameters are calculated based on the extended geometry.Profiles can be transformed into the camera coordinate system directly by using the transformation matrix from the orientation step.The result of the combination is a 3D point cloud in the conveyor belt coordinate system.
To avoid the additional measurement of the conveyor belt, a sequential approach can be used.Between two consecutive epochs, homologous points on the conveyor belt should be found.The points in the first epoch are fixed and define the coordinate system.Points in following epochs are matched to the points of the previous epoch and transformed into the initial coordinate system.Because only two epochs are compared at once, there is no need to explicitly consider the circulation.
For the last approach, the orientation step is not required.Instead, the moved distance of the conveyor belt is directly calculated based on the image data.The points of two consecutive image pairs are compared.For each point, the moved distance between the captures is determined.By using the mean distance, the profiles can be combined along the normal of the LTS sensor plane.This way a combination is possible without an orientation step or a nominal geometry of the conveyor belt.The calculated distance is free of any coordinate system and could also result from an encoder that is widely use with conveyor belts.

COMPARISON AND RESULTS
We used real orientation and extrudate data to validate our algorithms (Figure 5).Because rubber extrudate is flexible, has a dark and low-textured surface, and changes during the cooling, it is difficult to get a nominal geometry after the extrusion to compare it with our approaches.Instead, we combined data of orientation measurements.This way we can compare the point cloud from the LTS with the point cloud from the handheld laser scanner (Figure 6).The point clouds were aligned using an ICP algorithm and the cloud-to-cloud distance for each point was calculated.While there is no visible difference in quality between the geometry-based and sequential combinations, the solution of the normal-based approach seems to have a distortion along the direction of movement.To further quantify the accuracies, the sphere centres were extracted and compared to the nominal geometry of the orientation object by calculating the distance between the centres.Considering the mean RMSE of three different orientation measurements, the sequential combination enables the best solution (Table 1).

Approach RMSE [mm]
Geometry-based 0.62 Sequential 0.19 Normal-based 1.94 Table 1.Mean RMSE for the three different approaches comparing the extracted to the nominal sphere centres.
Without the need for an orientation step, the normal-based approach provides a fast solution, but only with less accuracy.It shows, that the simplified assumption of a movement along the normal of the sensor plane is not valid and an orientation is indeed needed to obtain accurate solutions in our case.We assumed, that using a nominal geometry of the conveyor belt would enable a solution with higher quality, because it would counter the accumulating errors of a sequential tracking.This assumption could not be validated in our experiments.The approach using the orientation, but not the nominal geometry provided the best solution.It should be considered that a conveyor belt is a flexible object up to a certain degree and influenced by its own movement.It is possible that it was stretched and compressed during the measurement so that the nominal and real geometry do not correspond completely.Furthermore, the moved distance of the conveyor belt during the orientation was relatively short and consequently the potential accumulating error is low, so a longer measurement could still benefit from using a nominal geometry.This assumption is supported by a comparison between the nominal geometry of the conveyor belt and the corresponding points from the sequential tracking (Figure 7).A difference in the geometries can be observed that will also influence the combination.Moreover, without the nominal geometry, a failed tracking between two epochs would lead to a break in the complete combination.It should be mentioned that using data from different orientation measurements the calculated orientations were similar, but not identical (Table 2).It cannot be excluded that there are small changes in the orientation between the measurements in reality, but because all measurements were conducted under the same conditions in a small time frame, it is more likely that there are deviations in the calculation, limiting the repeat accuracy.This affects the position of the points in the lower tenths of a millimetre range.167.11 -29.76 -97.30 593.19 80.26 -333.49 -167.29 -29.74 -97.37 593.67 80.28 -333.30 -167.34 -29.60 -97.82 592.92 79.47 -332.98Table 2. Transformation paramaters based on different orientation measurements Figure 8 shows parts of combined extrudate profiles.As mentioned, there is no nominal geometry for a comparison, but visual, the solution seems valid.The fringes on the sides are visible and the profiles form a continous stright track.

CONCLUSION
We presented a method to orientate a stereo camera system to a LTS system by using a calibration object made of photogrammetric markers and spherical geometries.Afterward, the calculated transformation matrix was used to combine profiles of a rubber extrusion process.Two approaches to use the orientation were developed.One uses a nominal geometry of the conveyor belt to track the movement of the extrudate, the other tracked the movement sequentially.The results were compared to a method without a refined orientation, which just combined the profiles along the normal of the sensor planes.
Our experiments show a higher accuracy when using the orientation.The assumption that a nominal geometry would improve the quality of the combination could not be proven, although the solution should be more robust against errors.
This paper focuses on the orientation of one of the two LTS systems.In general, the orientation of the second system works in the same way.The two calibration objects are connected by an aluminium profile.All spheres and coordinates of the photogrammetric markers have to be known in the same coordinate system.It should be noted that the normal-based combination cannot consider the orientation between the two LTS systems.
The solution of the orientation using different data were similar, but there are deviations.We are trying to solve this problem by adding an optimisation step after the first orientation, but it is not fully developed yet.We are applying various minimisation methods to adjust the six parameters, improving the alignment of the centre points with the nominal geometry.This way, a universal solution for the orientation, independent of small variation in the input data, should be acquired.
Another aspect for further research is the possible slip between the conveyor belt and the extrudate.We assume, that the material and the belt are moving with the same velocity.In our practical experiments, it was shown, that this is not always the case and that in fact there can be a minimal slip sometimes.To solve this problem the extrudate could be tracked directly using the image data.But because of the mostly black and low textured material this would be a difficult task, so a higher quality solution cannot be guaranteed.
This work is one possible step to support the digitalisation of the rubber extrusion.The geometrical monitoring acquires valuable information for the optimisation of the production process.Using the calculated orientation, three-dimensional point clouds can be created to validate the finished product, compare it to the nominal geometry, and adapt the production for high-quality products.

Figure 3 .
Figure 3. Scheme of the orientation algorithm using the nominal calibration objetc geometry (CG).

Figure 4 .
Figure 4. Point clouds from different steps of the orientation algorithm: (a) all profiles as one plane in the LTS system, (b) approximated combination of the profiles, (c) extraction of the sphere points, (d) sphere points as profiles, (e) combined profiles transformed into the calibration system, (f) sphere points transformed into the camera system.

Figure 5 .
Figure 5. Orientation with the calibration object (top) and measurement during the extrusion with rubber extrudate on the conveyor belt (bottom).

Figure 6 .
Figure 6.Combined profiles of an orientation measurement by the three approaches: geometry-based (top), sequential (middle) and normal-based (bottom), distances compared to the nominal calibration object geometry.

Figure 7 .
Figure 7.Comparison between the geometry of the conveyor belt based on an independent measurement (blue) and the sequential tracking (red).

Figure 8 .
Figure 8. Example of combined extrudate profiles using the sequential approach, color based on height, top view (left), isometric view (top), side view (middle) and front view (bottom).