Image Processing and QR Code Application Method for Construction Safety Management

Abstract

Construction safety accidents occur due to a combination of factors. Even a minor accident that could have been treated as a simple injury can lead to a serious accident or death, depending on when and where it occurred. Currently, methods for tracking worker behavior to manage such construction safety accidents are being studied. However, applying the methods to the construction site, various additional elements (e.g., sensors, transmitters, wearing equipment, and control systems) that must be additionally installed and managed are required. The cost of installation and management of these factors increases in proportion to the size of the site and the number of targets to be managed. In addition, the application of new equipment and new rules lowers the work efficiency of workers. In this paper, the following contents are described: (1) system overview, (2) image processing-QR code-based safety management target recognition methodology, and (3) object location discrimination technique applying the geometric transformation. Finally, the proposed methodology was tested to confirm the operation in the field, and the experimental results and conclusions were described in the paper.

1. Introduction

In the construction industry, serious safety and fatal accidents are commonly measured high, regardless of the distinction between developing and advanced countries [1]. Researchers and practitioners related to the construction industry are making a lot of effort to solve this problem. However, there is still a long way to go to zero accidents/injuries [2]. Even though the devices and technologies for construction safety management are constantly developing, the construction field’s accident rate is still high. According to data released by the U.S. Department of Labor, fatal occupational injuries by primary and secondary source of injury for all fatal injuries in the manufacturing sector (i.e., natural resources and mining, construction, manufacturing) in 2018 was the highest in the construction industry at 49.05% [3].

Stakeholders in construction projects have different levels of awareness of risks and different levels of understanding of safety according to their experiences and perceptions [4]. In addition, it is difficult to organize a systematic management process because the organizational composition continuously changes and new participants are generated according to the project schedule and stage [5,6]. This structural limitation of the construction project is a factor that lowers the learning efficiency for the project participants’ safety. Continuous safety training is essential to ensure that project participants who are newly organized by project and schedule follow the project’s standards and rules for safety management. Moreover, even if safety training has been completed, a system should be established in which the safety manager can check in real-time whether the subject of management complies with the safety-related regulations to manage unsafe safety consciousness and safety behavior of workers. The ideal form of direct safety management for construction by managers is that enough managers can check all workers scattered throughout the construction site. To satisfy this condition, the site’s scale increases the more safety managers must be employed. However, the biggest problems with these actions are (1) large amounts of economic expenditures irrespective of the project production cost and (2) whether it is possible to establish that they have sufficient knowledge for safety management and capabilities regardless of their safety career. Besides, recent studies pointed out that the existing management methods for construction safety do not sufficiently reflect potential risk factors occurring in real-time at the site [7,8]. Researchers argue that the current method should be changed to a bottom-up management method to solve these problems. This method can quickly transfer information generated on-site to headquarters and the person in charge of management and help start an appropriate response process and prepare alternatives [9,10].

In this paper, the methodology that identifies workers as individuals and estimates the location of workers, and updates their historical data, is presented. The proposed method can be applied to all sites where surveillance cameras are installed and can be applied to all cameras simultaneously. Accordingly, the method can be developed as a prototype of an automatic safety information management system for construction. The system can record workers’ safety-related history information without additional devices (e.g., sensors, transmitters, wearing equipment, control systems, etc.) and can be developed as a management system for personalized safety in the future. The case study analyzes the practical application of the methodology.

2. Literature Review

2.1. Construction Safety Management

In recent years, as high-resolution cameras were popularized in construction sites at low prices, the resolution of various images (i.e., photographs and videos) generated from the site has been highly developed. Moreover, analysis research is being conducted to utilize artificial intelligence technology for multiple purposes in the construction field [11,12]. Currently, computer vision technology has been developed to a level that allows it to automatically analyze various construction activities [13,14,15,16]. Accordingly, in safety management for construction, research on automation of safety management using computer vision technology shows a proportionate increase. In traditional safety management, safety accidents were considered to have been caused by individual carelessness, so personal responsibility was important. However, even the most experienced workers do not identify all of the safety risk situations that safety managers can [17]. Therefore, safety issues should be treated as separate from the worker’s experience, construction skill, and workability, and cannot be defined as individual problems alone. As a result, corporate responsibility for construction safety management has been gradually emphasized.

The safety management method for construction provides various advantages (e.g., reducing work injuries, risk control, productivity improvement, cost reduction, etc.) to construction projects [18]. However, it takes a lot of capital and time to observe these factors directly [19,20]. Fortunately, the low-cost prevalence of high-resolution cameras and advances in computer vision technology has shown the potential to solve these problems. To apply the technology to the right target, the computer must recognize what the observer sees and recognizes as the same. For this purpose, object recognition was conducted according to changes in color-related information (e.g., RGB values, hue, saturation) for accurate detection of objects [21]. Furthermore, as the resolution of surveillance cameras deployed at the construction site gradually improves, multiple workers are recognized in one video [22], and the path tracking of each worker also possible [23,24,25]. A study on human–object interactions (HOIs) [26] and a skeleton-based study [27] have been able to provide a basis for judging the current worker’s stability by recognizing the work situation and pose of workers. As the classification of objects became clear, researchers tried to analyze the movement and activity of objects from 2D images [28,29,30,31]. Moreover, by applying a methodology to estimate the 3D coordinates of an object from multiple images, a method that can implement an object’s position on a three-dimensional plane has been studied without any sensor (e.g., distance sensor, ultrasonic sensor, etc.) [32,33,34,35]. As such, the current computer vision studies have immediately discovered the management targets in the field and are performing the targets’ location based on the distance between the surveillance camera and the object. In addition, most studies classify objects as simple class levels (e.g., workers, people) and do not try to realize their own information (e.g., worker’s name). If the statistics enable workers to recognize the frequency of exposure to an unsafe situation, a perception of safety can be improved by themselves [36]. This immediate feedback system can also be applied at an economical cost to a system dynamics approach [37,38] to determine whether the current governance structure is working properly.

2.2. You Only Look Once (YOLO) V3

In this study, image processing based on You Only Look Once (YOLO) was applied to implement object recognition. YOLO regards the bounding box and class probability in an image as a single regression problem and can guess an object’s type and location by looking at the image once. It is also a method of calculating the class probability for multiple bounding boxes through a single convolutional network [39]. In this study, the updated version, YOLO v3, was used (Figure 1).

2.3. Geometric Transformation

The video recorded by the on-site surveillance camera reflects the distance difference due to the perspective. There is a difference between the moving distance in the video and the actual one. Thus, estimating an object’s actual position in an image requires steps to offset the perspective and mapping it to a plane. In this study, matching the image of the surveillance camera with the drawing by applying a geometric transformation to the image is proposed. Geometric transformations can be classified into five types as shown in Figure 2 [40]. In this study, distortion correction is used.

2.4. Quick Response (QR) Code

The larger the amount of information that QR codes contain, the greater the number of pixels. It must also have minimal resolution performance to allow pixel dis-crimination to recognize pixel patterns. QR codes allow users to enter a variety of information, but depending on the amount of information, the number of pixels required will increase proportionally (Figure 3). Therefore, if high-capacity information is entered into the QR code, a high-resolution device is required to recognize the QR code. Conversely, the minimal information required for object recognition reduces the number of pixels in QR codes. It can also increase the recognition rate of QR codes even with low-resolution devices. In this work, we use the object’s ID (five digits) as the least information to distinguish objects.

3. Image Processing-Based Worker Location Estimation System

3.1. Image Processing Learning Model

The system is based on Yolo V3 and developed the system code and user interface using Python, Anaconda, Ubuntu, and C++. In this study, people (i.e., workers) and things (QR code) are classified into three classes and recognized. As part of building a custom class for workers and QR codes, the image data necessary to train the weight model were collected from the web and case-study sites. Yolo basically provided the weight model for the person-class. Among the obtained data, we excluded images that are difficult to label (i.e., class classification and bounding box representation), and only available images were selected and used for learning. Additional learning data were created by reproducing the images by using the horizontal flip and vertical flip. Finally, 921 directly acquired images and 2763 reprocessed images, a total of 3684 images, were used in the learning process (

Table 1. Class and sub-label composition for object classification.

Class	Training Sample	Sub-Label	Example of Training Sample
Worker	3172	Standing
		Bending
		Sitting
Person	referred from YOLO V3 data
QR code	512	Original
		Stretched
		Distorted

). In this study, the images used for learning (i.e., worker and QR code) were composed of sub-labels as shown in Table 1.

In the learning model development, 50% of the acquired images were used for learning, 30% for comparison, and 20% for verification data. The training settings applied to use the system are shown in

Table 2. Configuration of the training inputs.

Batch.	Subdivisions	Width, Height	Channels	Decay	Angle	Saturation	Exposure	Hue	Learning Rate	Burn in	Max Batches	Policy	Steps	Scales
64	4	448,448	3	0.9	0.005	3	1.5	1.5	0.0005	1000	40,000	200; 400; 600; 20,000; 30,000	200; 400; 600; 20,000; 30,000	2.5; 2, 2; 0.1 0.1

.

3.2. System Algorithm

The image processing-based worker location estimation system is executed as follows: (1) image acquisition from a field surveillance camera, (2) object recognition using image processing, and (3) information matching in the system DB. The system algorithm is shown in Figure 4.

3.2.1. Matching Coordinates between On-Site Video and Drawing

When a region for management has been selected, a surveillance camera is installed. An ID is assigned to the surveillance camera, and the video from the camera is collected in real-time. Update the design drawing to see where the camera is located on the plane. Reference points are selected from the acquired image and the drawing, and the position reference coordinates are matched. After all coordinate matching is completed, check whether the recognized object is projected to the correct position on the plane (Figure 5).

As shown in Figure 5a, the mapping process of the site topography can be targeted at 4 coordinate points in the video from the surveillance camera. After setting the coordinates, the observed object reads the person, worker, and QR code. In Figure 5b, the recognized object coordinates and rectangular reference coordinates are recognized as X, Y coordinates of the video. Record the offset X, Y length between the recognized object and 4 points based on the current state.

The rectangles and four points are shown in Figure 5b, and Figure 5c actually mean the same space and coordinates. Accordingly, it is necessary to search for a transducer of the four points and object’s position. When Figure 5b coordinate points are expressed as input values (x, y), Figure 5c coordinate points can be expressed as result values (x’, y’). (x’, y’) can be expressed as Equations (1)–(3).

$$\left(\begin{array}{c}{\mathit{wx}}^{\prime }\\ {\mathit{wy}}^{\prime }\\ w\end{array}\right)=D{R}_L\left(\begin{array}{c}x\\ y\\ 1\end{array}\right)=\left[\begin{array}{ccc}{p}_{11}& p_{12}& p_{13}\\ p_{21}& p_{22}& p_{23}\\ p_{31}& p_{32}& p_{33}\end{array}\right]\left(\begin{array}{c}x\\ y\\ 1\end{array}\right)$$

(1)

x′ = (p₁₁x + p₁₂y + p₁₃)/(p₃₁x + p₃₂y + p₃₃)

(2)

y′ = (p₂₁x + p₂₂y + p₂₃)/(p₃₁x + p₃₂y + p₃₃)

(3)

Each Equation for x-coordinate and y-coordinate can be obtained from the movement relationship of a point. Therefore, a distortion rate factor (DR_L) is obtained by sequentially analyzing eight equations from four input coordinates and output coordinates. Moreover, when the position coordinate point of the object recognized in Figure 5b is (m, n), the position coordinate of the object in Figure 5d is calculated as Equations (4) and (5) using DR_L.

$$\left(\begin{array}{c}a\\ b\\ c\end{array}\right)=D{R}_L\left(\begin{array}{c}m\\ n\\ 1\end{array}\right)$$

(4)

$$(m^{\prime }, n^{\prime })=(a/c, b/\mathrm{c})$$

(5)

3.2.2. Target Object and QR Code Recognition

An on-site surveillance camera is filmed from one direction. Unless it has a complex structure, a single device is installed in one space. In this study, the reference point of the object was determined by the worker’s feet. Accordingly, the center point of the lower part of the object bounding box is assumed to be the floor position on which the worker is standing (Figure 6).

Equation (6) represents the anchor box coordinates of the object, and Equation (7) represents the anchor box coordinates of the QR code. Both indicators form a rectangular shape made up of four points.

$$\mathrm{A}=\left[\begin{array}{cc}x1,y1& x1,y2\\ x1,y2& x2,y2\end{array}\right]=[\mathrm{A}, \mathrm{B}, \mathrm{C}, \mathrm{D}]$$

(6)

$$\mathrm{QA}=\left[\begin{array}{cc}n1,m1& n1,m2\\ n1,m2& n2,m2\end{array}\right]=[\mathrm{E}, \mathrm{F}, \mathrm{G}, \mathrm{H}]$$

(7)

In this study, Equation (8) is applied to determine whether the QR code is included in the object.

$$\left(\overrightarrow{\mathrm{AB}}\times \overrightarrow{\mathrm{AE}}\right)· \overrightarrow{k}=\left(\mathrm{B}x-\mathrm{A}x\right)\left(\mathrm{E}x-\mathrm{A}y\right)-\left(\mathrm{B}y-\mathrm{A}y\right)\left(\mathrm{E}x-\mathrm{A}x\right)\ge 0$$

(8)

Equation (8) shows the method of calculating the outer product between the AB vector and the AE vector (Figure 7). If the result of the equation has a positive value, it means that the E point is located to the left of the AB line. As a result, if the coordinates (E, F, G, H) of the anchor box QA satisfies all these formulas for the anchor box A, it is proved that the QR code is inside the object.

3.2.3. Construction Management DB

The system recognizes three classes of people, workers and QR codes. At the construction site, the object containing the QR code is the person included in the system DB, which means the project’s people. If the QR code does not recognize, it can be divided into two cases. The one is wearing work clothes, and another is not wearing work clothes. If the target is wearing work clothes, the QR code does not recognize or that the target did not include the DB. In the case of another, it is regarded as neither the worker nor the project’s people.

4. Case Study

A site was recruited for the analysis of actual cases of the proposed methodology. The site is an apartment complex construction with an area of 65,245 m², and structure construction has been completed. The site drawing provided by the manager was in CAD file format (.dwg), and a drawing in which it was easy to understand the site boundary was selected and used. For the calibration of the drawing, the actual positions of the four reference sets of X and Y were entered. As a result, it was possible to calculate the coordinate positions of all points (Figure 8). The area used for the case study is the area marked in red in Figure 8.

In the drawing, the parts entered as reference coordinates are the four red dots shown in Figure 9a, and the location and angle of view of the surveillance camera that took the image are visually displayed. The image captured by the actual surveillance camera was taken as shown in Figure 9b, and the four reference coordinates set in the video were set as T1~T4 as shown in (Figure 9c). As for the QR code marking method, the paint was directly marked on the clothes using a stencil technique to prevent peeling during work (Figure 9d). As a result of object recognition, among a total of 4 workers in the photo, all workers were recognized except a worker located at the front apartment entrance. A total of 3 workers were recognized as workers. Even though the QR code was distorted due to wrinkled clothes, the system recognizes the code correctly. The worker’s identification number “00121” was printed (Figure 9e). Finally, the worker “00121” was estimated to be located in (197.4, 31.8) in Figure 7 (the space set as the X-axis (0~384.6 m) and Y-axis (0~178.6 m)) (Figure 9f).

5. Conclusions

This study deals with a study on developing a methodology for construction safety management automation using surveillance cameras installed on site. In this paper, three methodologies were presented: (1) classification and recognition of object; (2) allocating unique identification information using QR code; (3) extracting an object’s coordinates projecting onto a 2D plane. During the recruitment process at the experiment site, the research team was able to conduct the test on the condition that it does not infringe on the work environment of the on-site staff. Accordingly, there is a limitation in that workers and managers did not conduct the surveys. The authors discussed the case study results with safety managers. Managers have commented that this methodology is easy to apply on-site because it does not require additional equipment installation and does not require special instructions to workers. Moreover, there was an opinion that immediate application could positively affect major management targets (e.g., less than 1 year of experience, workers in hazardous areas).

The images that are using for training were taken under sufficient brightness. Because these images easily distinguish between objects and backgrounds, neural networks trained with data can record high object identification rates for images with vivid colors. However, most of the construction task is done outside, and the amount of sunlight fluctuates depending on the location of the clouds and the sun. Besides, images shot in the shade become very difficult to distinguish between the object and the background. This situation increases the uncertainty and inaccuracy of object recognition. In the study of Edirisinghe et al. [41], an approach through QR code was proposed so that workers could easily access and understand information on the precautions during work and obtained meaningful results that enable workers to prepare themselves for injuries. In addition to these results, to feedback personalized statistical data to each worker, it will be possible to improve the accident prevention effect by considering the physical fatigue and injuries caused by the worker’s improper posture.

Accordingly, in future research, it is necessary to improve object recognition accuracy by applying fuzzy theory [42,43] and to obtain personal safety information by applying improper working posture recognition [44,45].

This study provides the basis for a methodology that can automatically collect each worker’s individual safety history. If the problems described above are solved, and the system’s reliability is sufficiently secured, it is expected that even the simple process of data updating [46] by a manager using a smart device can be omitted.