Error assessment and correction for extrusion-based bioprinting using computer vision method

299Bioprinting offers a new approach to addressing the organ shortage crisis. Despite recent technological advances, insufficient printing resolution continues to be one of the reasons that impede the development of bioprinting. Normally, machine axes movement cannot be reliably used to predict material placement, and the printing path tends to deviate from the predetermined designed reference trajectory in varying degrees. Therefore, a computer vision-based method was proposed in this study to correct trajectory deviation and improve printing accuracy. The image algorithm calculated the deviation between the printed trajectory and the reference trajectory to generate an error vector. Furthermore, the axes trajectory was modified according to the normal vector approach in the second printing to compensate for the deviation error. The highest correction efficiency that could be achieved was 91%. More significantly, we discovered that the correction results, for the first time, were in a normal distribution instead of a random distribution.


Introduction
Organ shortage is a serious social health crisis. A report from the University of Minnesota states that approximately 90,000 people require kidney transplant, but only 1,500 people have undergone kidney transplants in 2018 [1] . The shortage of fitting and propitious organs for transplantation has always been a medical concern [2][3][4] . Building organs from scratch to explore entirely new cell configurations is the main feature of bioprinting, which is an emerging scientific field that has potential to solve this organ shortage crisis [5,6] .
Bioprinting methods can be divided into inkjet-based, laser-based, extrusion-based, and stereolithography-based bioprinting [15] . Compared with other printing methods, extrusion-based bioprinting is the most often used, as it has high throughput and a fast printing speed [16][17][18] . Extrusionbased bioprinting mainly relies on rotation dispensing, air pressure dispensing, or force dispensing, as shown in Figure 1A.
Although extrusion-based bioprinting has more advantages compared to other printing methods, the low printing resolution is the main reason that limits the advancement of extrusion-based bioprinting in various fields [19][20][21][22] . Moreover, many studies have pointed out that resolution is an important parameter for bioprinting; for instance, a study on artificially printing human heart pointed out that high resolution is an important requirement for stable manufacturing of the heart [23] . Hence, it is necessary to emphasize and improve on the resolution of extrusion-based bioprinting.
The defect in the deviation of the actual printing path from the predetermined path, and the deviation error value that remains in the final printed organ are the main reasons for the low resolution of extrusionbased bioprinting [24,25] . Normally, the deviation value between the reference path and the printing path is small in linear shape areas; however, the deviation error shows a significant increasing tendency when the path changes to a curved shape. The whole helix is composed of the antihelix and helix, which is a curved structure that has been printed in many studies. However, the research on helix printing still lacks a method that can precisely control the printing path and reduce the deviation error value between the printing path and the reference path; moreover, there is still much room for improvement in terms of printing resolution [26,27] . This research focuses on the complex antihelix and helix structure using the Sobel operator and the normal vector approach to reduce the printing error to an acceptable range, thus improving the printing resolution ( Figure 1B). A process control method based on computer vision was used in this study to enable the detection and correction of the path errors of the antihelix and helix. Furthermore, the deviation error was controlled to an acceptable range to ensure that the accuracy of the helix printing path shows an increasing tendency.

Equipment
Based on the initial bioprinting machine, a non-contact camera was placed above the printing platform to capture the path of the antihelix and helix after printing. The noncontact camera covered a range of 40 mm × 40 mm rectangle to ensure that the complete helix could be captured.
Two different antihelix and helix structures were designed and counted in this study. Each helix structure had different antihelix and helix. The purpose of counting the two helix groups was to determine the difference in error values carried by different shapes of helices and prove that the correction methods are universal.

Image algorithm
Since the captured image cannot be directly recognized by the correction algorithm, the helix image needs to be pre-processed. Figure 2 illustrates in detail how abstract image data is converted into spatial point cloud data to be recognized by the correction algorithm. The first step involves acquiring the antihelix and helix images through a non-contact camera placed above the printing platform. Through binarization processing, the image matrix is reduced from three channels to a single channel, and the gray value in the image matrix is only 0 or 255; additionally, noise is also removed. The purpose of this step is to enhance the contour features of the helix and prepare for the next step. In the third step, a computer vision-based method is used to extract the edges of the helix printing image. The Sobel operator is widely used in image edge detection and extraction due to its fast response and high precision. The Sobel operator usually consists of two 3 × 3 convolution kernels. The two convolution kernels would respond the most to the horizontal edge and the vertical edge, respectively. Following the processing of pixels in the image matrix by convolution, the Sobel operator outputs two images with a horizontal edge and a vertical edge, and after the two images are superimposed, the whole helix edge is extracted [35][36][37] .
In the fourth step, the estimated value of the helix centerline is calculated and defined as the spatial path of the printed helix. In the fifth step, the point cloud data of the estimated helix centerline and the coordinate data of the reference path are placed in the same Cartesian coordinate system for comparison and calculation of the error between the two trajectories. In the sixth step, the error vector between the printing path and the reference path is calculated, and the error is mapped onto the printing path through different shades of color. The specific calculation method will be discussed in detail in the next section. In the seventh step, the modified reference path is obtained by a compensation vector based on the original reference path. Figure 2G shows the helix printing path (blue solid line), original reference path (black dashed line), and modified reference path (red dashed line).

Correction algorithm
Considering that the accuracy of the helix path is affected by the mechanical axes movement, it would be an effective attempt to adjust and control the print head to compensate for errors in the printing process. The entire automated correction algorithm can be divided into the error detection part and the execution operation part, as shown in Figure 3A. In the error detection part, the deviation errors between the helix printing path and the reference path are calculated, and these errors are automatically collected to modify the reference path by compensation vector to reduce the offset value of the helix path.
The first step involves the original printing, in which the material is extruded from the print head according to the reference path without any additional operations. The deviation error value between the printing path and the reference path is fully retained in this step. The second step involves obtaining the point cloud data of the original helix printing path. Generally, the collected printed helix image cannot be recognized by the correction algorithm. Therefore, the Sobel operator is used to extract the edge of the helix printing image. Then, the centerline of the helix is calculated to represent the real-space position coordinates. The specific details of point cloud conversion have been discussed in section 2.3.
The third and fourth steps involve detecting the deviation error between the original helix printing path and the reference path as well as calculating the modified reference path. Based on Figure 3B, the normal vector approach determines the compensation vector by taking the mirror image of the error vector to the reference path. The magnitude of the compensation vector is equal to that of the error vector, but in an opposite direction. Furthermore, the modified reference path is calculated based on the compensation vector added to the original reference path.
In the fifth step, the helix printing path is a result of the modified reference path, which is different from the original reference path in the first step. In the sixth step, the deviation error value between the new helix path, resulting from the modified reference path, and the reference path is recollected. Finally, the helix printing image is recollected in the fifth step. Similar to the second step, the image processing algorithm, as discussed in section 2.3, converts the helix image data into point cloud data.

Original printing
The printing data of the original helix was successfully counted in the Cartesian coordinate system using the computer vision-based image method ( Figure 4). The actual helix printing path deviated from the as-designed reference path, thus necessitating additional supervision. The flesh-colored path represents the helix and antihelix printing path, while the black dashed line represents the reference path; the red dotted line represents the centerline of the helix and antihelix path, which reflects the coordinate position of the printing path and facilitates the calculation of the deviation value from the reference path.

Correct printing
Through computer vision-based algorithm processing, the accuracy of the two helix trajectories improved significantly. Figure 5 illustrates the helix and antihelix printing path resulting from the modified reference path and the original reference path to compare the correction effects. The white path represents the original helix and antihelix printing path, the transparent blue path represents the printing path resulting from the modified reference path, while the black double-dotted line represents the as-designed reference path.
The original reference path, modified reference path, original helix and antihelix printing path, as well as the new helix and antihelix printing path were analyzed and compared in Figure 6. The topmost figures show the as-designed original reference path and the modified reference path with the compensation vector. The black dashed line represents the original reference path, while the blue solid line represents the modified reference path. The bottom figures show the helix and antihelix printing path resulting from the modified reference path, which is closer to the as-designed reference path than the original helix and antihelix printing path. The blue solid line represents the helix and antihelix path resulting from the original reference path, the red dotted line represents the helix and antihelix printing path resulting from the modified reference path, and the black dotted line represents the original reference path.   Combined with the training of the error vector, the compensation vector was added to the original reference path to obtain a modified reference path that would result in a better position with less printing errors. Figure 7A illustrates the magnitude of error carried by the helix and antihelix before and after correction. The different shades of color surrounding the helix and antihelix represent the magnitude of error of the area; the magnitude of error changes from small to large as the color transitions from light to dark. The helix and antihelix path on top resulted from the modified reference path, while the one at the bottom resulted from the original reference path. The error bar on the right reflects the error magnitude corresponding to the shade of color surrounding the helix and antihelix. Figure 7B records the error carried by each position for the two helices.

Discussion
There are relatively few studies on reducing printing deviation errors to improve the resolution of bioprinting. Considering that the deviation error value in actual printing might be difficult to perceive with the naked eye, a novel correction control system based on computer vision was designed and used in this study to realize the identification and correction of small errors. The final quality of the helix can be improved by adjusting the reference trajectories for the helix and antihelix. There was a significant reduction in the deviation error value of the path following the addition of the computer vision-based control. This indicates that the accuracy and resolution of bioprinting have risen by a level. Figure 8 illustrates the correction results of the helix and antihelix after correction algorithm training. Original printing error existed in each range, and the error was randomly distributed as shown in the figure. Without correction control, the print heads deviated from the predetermined path in different positions. For the position where multi-axis coordinated motion is required, the error will show an increasing tendency. For single-axis motion position, the error carried will show a decreasing tendency. With the addition of the compensation vector to the reference path, the error distribution fell in the low error range on the left. More significantly, the error distribution was in a normal distribution instead of a random distribution. This shows that the correction method based on computer vision control has the same and efficient effect on different areas.
Before the correction, most areas of the helix path deviated from the as-designed reference path, and the distance between the two trajectories was significant ( Figure 6). After image algorithm training, the distance between the helix path and reference path reduced significantly, and most of the areas overlapped. This shows that the computer vision-based method plays a significant role in path guidance. Furthermore, the modified helix path of the printing part was compared to the path of the as-designed model to compute the printing accuracy, which was defined as the average deviation error of different areas. The deviation error value between the helix printing point cloud data and the reference path was calculated based on the Euclidean norm (1), which is different for each area. For each helix, the dark blue area concentrated at the turning of the helix at the top suggests that the curve path is the area with the largest deviation from the as-designed reference path, with a maximum error of 2.47 mm ( Figure 7A). On the other side, the antihelix and other helix areas rely on less curved structures, in which the shade is lighter than the area at the top of the helix and located in the medium error range. The phenomenon of high error at the curved area and low error at the straight line area is consistent with that described by other researchers [25,38] .
The high error and medium error areas in the helix path improved significantly after algorithm training. The error after correction was in the light blue low error area, which was less than the deviation value of 1 mm. The average error value of the two helices reduced from 0.93 mm and 1.21 mm to 0.30 mm and 0.45 mm, respectively. Moreover, the average value of the high error area dropped from 2.41 mm and 2.16 mm to 0.21 mm and 0.31 mm, respectively, whereas the average value of the medium error area dropped from 1.29 mm and 1.55 mm to 0.25 mm and 0.38 mm, respectively, for each helix (Figure 7). The correction efficiency of the high error area reached 91% and 85%, while that of the medium error area reached 80% and 75%. This result shows that the normal vector approach exhibits a strong correction ability in improving material placement and has a consistent correction effect for different error areas.
A breakthrough has been made in the modification strategy for trajectory correction, in which the deviation error between the reference path and the printing path has been reduced to an acceptable range. Researchers should attempt to further probe into the accuracy of the complete ear trajectory in the entire printing space. The definition of trajectory deviation error has also been transformed from two-dimensional space to threedimensional space. However, various challenges exist in this work. Although point cloud data collection in two-dimensional space based on computer vision shows superior performance, the limitations of computer vision are magnified in two-dimensional cloud data collection. Furthermore, error definition would also shift from two-dimensional space to three-dimensional space. Hence, further investigations are required in this field of research.
The conversion of image data into component point cloud data is an important step to achieve path correction. With accurate trajectory point cloud data, it is feasible to accurately describe and define the position coordinates of components in real space. With the advancement of computer science, computer vision algorithms with powerful processing capabilities in image processing would be uncovered, allowing the edges of printed components to be extracted more accurately [39] .

Conclusion
We added a computer vision-based method to this study to improve the bioprinting resolution and printing accuracy. The error between the helix printing path and the reference path is one of the main reasons that limits the printing resolution. The path error was defined through the image algorithm, and the original reference path was modified for the second printing. The error of the helix printing path resulting from the modified reference path was reduced to an acceptable range. This method showed a significant reduction effect for high error areas. More importantly, the error of the correction results was in a normal distribution instead of a random distribution, thus proving that this method based on computer vision is universal for different locations.
This research has demonstrated a method for effectively detecting defects and correcting errors in the field of bioprinting. In fact, this method is also suitable for improving the resolution of printing other organs. We hope that bioprinting can further advance and create new opportunities for regenerative medicine.