Automatic estimation of excavator actual and relative cycle times in loading operations

.


Introduction
Heavy-duty mobile machines (HDMMs) play essential roles worldwide and are employed in many industries, including mining, forestry, and construction.The industries, which are rapidly growing, have tough challenges, including a lack of skillful human operators, harsh environmental conditions, low productivity, and safety [1].The productivity improvement of the construction industry has been a significant requirement over recent years.Studies show that the productivity of the construction industry has only improved by 1% over the past 20 years [2].Additionally, the costs of HDMMs significantly affect the overall construction project costs.According to studies, equipment costs account for 5% to 10% of direct costs in building construction projects and up to 40% of direct costs in highway construction projects [3].
In order to enhance the performance of HDMMs, the common quote, "if you cannot measure it, you cannot improve it," [4] must be taken into account.Accurately measuring the productivity and utilization rate of the equipment is a major challenge in earth-moving projects.Traditional methods for monitoring the productivity of HDMMs are based on manual data collection and direct observation of activities that are significantly time-consuming, high-priced, and error-prone.Hence, an automatic method is needed to accurately monitor the productivity of HDMMs in earth-moving tasks under different operating conditions.Productivity monitoring of HDMMs is a major step toward semi or fully autonomous worksites and can bring about a massive reduction in operating time, fuel consumption, and expenses.It can also assist worksite managers in identifying potential project issues, optimizing planning and operating parameters, precisely pricing, and budgeting future projects, and enhancing management and economic circumstances [5,6].Moreover, the feedback provided by the machines' productivity can aid operators to improve their skills [7].
The loading operation is one of the most significant tasks in construction and mining projects.This operation involves using the excavator's manipulator to pick up and move materials from one location to another.This can include digging or collecting material from the ground during site preparation for construction or loading materials onto trucks for transport.Accurate productivity estimation of an excavator in the loading operation guides contractors and project managers in planning and budgeting the project effectively, which can lead to cost savings by ensuring that resources are used efficiently, and the project is completed on time.Also, productivity estimation helps to determine the right size and type of excavator needed for the project, which can help to maximize equipment utilization and reduce downtime [8].
The productivity of most cyclical types of machinery is typically determined by the quantity of material and the cycle time of the operation.The excavator's cycle time is composed of the time needed for (1) scooping, (2) swinging loaded, (3) dumping, and (4) swinging empty.The schematic of an excavator in the loading cycle is shown in Fig. 1.The excavator's cycle time highly depends on the operating conditions and parameters [5], including:

Excavator
• Size of the excavator • Bucket capacity 2. Relative position between the excavator and material • Digging depth 3. Relative position between the excavator and dumping position • Swing angle • Relative height • Dumping condition 4. Site conditions • Type of material • Site congestion 5. Skill of human operator 6. Weather conditions In the scooping sub-task, digging depth and type of material are key factors.The scooping sub-task requires more time to complete as the material gets harder or the location of the material gets deeper [10].Another parameter is the swing angle which can reduce or increase the time required for swinging loaded/empty, as well as the overall cycle time.Also, the machine's size significantly impacts the swinging time because smaller machines cycle faster than larger ones.Moreover, the productivity of the excavator is influenced by the operator's skills [11].
The organization of this paper can be summarized as follows: the literature reviews are presented in Section 2. The method is elaborated in Section 3. In this section, the proposed supervised learning algorithm for recognizing excavator activities and the cycle time estimation are described.Then, the swing angle, digging depth, and theoretical and relative cycle time estimation approaches are introduced.The results of implemented method on two case studies are presented in Section 4. In Section 5, we discuss our work and clarify current limitations.Finally, Section 6 concludes the paper.

Literature review
Numerous studies have been conducted to identify equipment activities, calculate activity duration, and estimate operation cycle times.The developed techniques for monitoring the excavator's productivity are divided into four primary categories according to the type of data: 1. Vision-based methods 2. Audio-based methods 3. Motion-based methods 4. Hybrid methods Vision-based methods mostly utilize cameras as videos or images.Audio-based methods identify excavator activities using generated sounds from machines.Motion-based methods utilize different sensors, such as inertial measurement units (IMUs) to obtain acceleration and orientation data of distinct parts of equipment.In hybrid methods, several types of sensors, such as visual and motion sensors, are combined to recognize excavator activities and monitor productivity.

Vision-based methods
In order to monitor work progress, identify states of excavators, and safety on construction sites, visual recording technologies such as images and videos have been widely used.Most existing works determined productivity by identifying excavator activities.In [12], an image processing technique is developed to estimate only the idling time of a hydraulic excavator and then calculate the percentage of working time in the total operation time to determine productivity.The algorithm uses the hue-saturation-value (HSV) color space.In [13], a CV-based method for identifying activities of earth-moving construction equipment, including excavators and dump trucks, is presented.The suggested approach utilizes the histogram of oriented gradients (HOG) descriptor and a support vector machine (SVM) classifier.In [14], a CV-based method is presented for the activity recognition of an excavator in earth-moving operations using highly varying long-sequence videos obtained from fixed cameras.The algorithm identifies excavator activities (swing, dig, dump, idle, and move) at each video frame.In [15], two visual technologies, photogrammetry, and video analysis are integrated for progress monitoring and volume estimation of the excavated soil in earth-moving operations.In [16], it has been clearly expressed that activity identification is an essential step toward the productivity monitoring of earth-moving operations.A vision-based method is designed to identify the excavator activities (work, travel, and idle) using tracking-learning-detection (TLD) and bags-of-features (BoF).The method is extended in [17], and a CV method based on spatio-temporal reasoning and image differencing techniques is proposed to detect the activity of excavators and dump trucks.The proposed method consists of four main steps: (1) equipment detection and tracking, (2) action recognition of individual equipment, (3) interaction analysis, and (4) post-processing.In [18], a method is proposed for productivity analysis of an earth-moving process in a tunnel using the combination of construction-process simulation and vision-based context reasoning.Convolutional neural networks (CNNs) detect construction equipment in recorded videos.In [19], a CV algorithm based on a hybrid deep learning algorithm (i.e., CNN and long short-term memory (LSTM) network) is presented to detect, track, and identify the activity of an excavator in earthwork operations using two sequential operating patterns (visual features and operation cycles).It has been underlined that two limitations of the method are the significant computational training time and the requirement for a large amount of training data.In [20], a threedimensional (3D) CNN based on temporal and spatial information is presented to identify the activity of an excavator.The identification Fig. 1.Excavator loading cycle [9].
A. Molaei et al. results are utilized for the excavator's productivity monitoring.In [21], a CV-based method is illustrated that allows for the automatic recognition of visually distinctive excavator and dump truck activities from individual frames of a video filmed at ground level.The excavator activities include idle, loading bucket, swinging bucket, dumping, and moving, and the dump truck activities are idle, moving, dumping, and filling.In [22], a deep learning-based method is proposed to identify five actions for excavators and dump trucks from video frame sequences.In this method, CNN and LSTM are used to extract image and temporal features, respectively.In [9], three CNNs are designed for the excavator activity identification, activity time, and productivity estimation.It has been noted when the light condition is not good or when multiple pieces of construction machinery are simultaneously captured, the CV-based methods face significant challenges.In [23], an expensive approach is proposed that uses multiple non-overlapping cameras at a worksite for productivity monitoring.The model can identify different activities of excavators and dump trucks, including digging, swinging full, dumping, swinging empty, moving, and stopping.In [24], a deep learning-based excavator activity analysis and a safety monitoring system are presented that can identify the excavator activities, detect the surrounding environment, and estimate poses.This research continues, and in [25], an algorithm is developed for the activity classification of the excavator that has higher performance compared with the proposed model in [24].In [26], a vision-based method is described for automatically analyzing excavators' productivity in earth-moving tasks by adopting zero-shot learning for activity recognition.The proposed method can identify activities of general construction machines (e.g., excavators and loaders) without pre-training or fine-tuning.In [27], a novel methodology is proposed to classify the activities of the excavator.First, the pre-trained CNN model extracted the sequential pattern of visual features from the video frames.Then BiLSTM classified the activities by analyzing the output of the pre-trained CNN.The summary of vision-based methods for the excavator activity recognition algorithms is presented in Table 1.

Audio-based methods
Several studies have been published that utilize audio data to identify the activity of machines and estimate the operation cycle time.These methods mostly consist of four basic steps: (1) gathering equipment sound data using a microphone, (2) signal filtering or augmentation, (3) extracting features, and (4) training classification models to determine equipment activities from the signals.In [28], an algorithm is presented that utilizes the generated sound by construction equipment to categorize the activity of equipment such as excavators, loaders, and dozers into two main groups: major and minor activities.This method utilizes the short-time Fourier transform (STFT) features and an SVM classifier.In [29], a method is illustrated to estimate the cycle time and productivity of equipment such as excavators and dozers using the audio data and Markov chain filter.The activity identification algorithm uses an SVM classifier combined with STFT, and continuous wavelet transform (CWT) features.In [30], a multi-label multi-level sound classification method is proposed based on STFT and CNN that only needs a single-channel off-the-shelf microphone.This paper has investigated situations with mixed construction sounds, but this method is still susceptible to some limitations, such as the assumption that two types of construction noises always occur simultaneously.The audio-based methods for the excavator activity recognition algorithms are summarized in Table 2.

Table 1
Overview of vision-based methods for excavator activity recognition.

Motion-based methods
In motion-based technologies, motion sensors are attached to different parts of equipment in worksites.The operation cycle time can be measured by classifying data obtained from the sensors.In [31], a method is proposed that uses motion signals generated from an accelerometer mounted inside the cabin of a medium-sized excavator to show the connection between operational efficiency and environmental performance.In [32], different supervised classifiers, including Naive Bayes, instance-based learning, k-nearest neighbors (KNN), and decision tree, are employed to analyze the pattern of acceleration data obtained from accelerometers mounted inside the cabin of four types of excavators.In [33], a non-invasive technique is proposed to estimate the cycle time of an excavator based on activity modes (e.g., wheel-base motion, cabin rotation, and arm/bucket movement of the excavator).Eight classifiers are trained based on time and frequency domain features of acceleration data.The data is collected using a smartphone mounted inside the cabin.The accuracy of cycle time estimation is equal to 75.96%.In [34], the excavator operation cycle time is calculated utilizing IMU data.It uses random forests, Naive Bayes, J48, and sequential minimal optimization (SMO) for the excavator's activity identification.The accuracy of cycle time estimation is equal to 91.83%.In [10], synthetic training data is generated by implementing time-series data augmentation techniques on acceleration and orientation data.A recurrent neural network (RNN) is utilized for the activity classification of four different types of excavators and front-end loaders.In [35], a dynamic time-warping system is illustrated that uses joystick signals to automatically classify excavator activities.In [36], a real-time activity identification method is developed using motion data (i.e., linear, and angular acceleration) of the articulated structural parts of construction equipment.Three machine learning methods, an SVM, a KNN, and an artificial neural network (ANN), are trained using collected data.In [37], an automatic activity identification algorithm is presented that uses deep learning architectures and collected acceleration data.The proposed method is applied on a roller compactor and an excavator.In [38], an approach is presented to automatically identify the working stages of an excavator based on the main pump pressure waveform.Three machine learning algorithms, an SVM, a back propagation neural network (BPNN), and logistic regression (LR), are employed as a classifier.In [2], a deep neural network (DNN) model is proposed to calculate the volume of excavated earth using telematic data from 21 days of operation.The main limitation of DNNs is the high computational load, and also, they need a large dataset.In [6], point clouds, image data, sensors, and CAD models are combined to measure the excavation volume and monitor the excavation progress at a worksite.In [11], a random forest classifier is integrated with the fractional calculus-based feature augmentation technique to propose a method for activity recognition of construction equipment.The method is implemented in three case studies: (1) two different models of excavators, (2) a scaled remotely controlled excavator, and (3) a roller.In [39], three machine learning algorithms, an LSTM network, an RNN, and an SVM, are trained to recognize the excavator's working stages using the control signals of operating handles.In [40], a deep learning-based method is proposed to measure equipment productivity using kinematic data collected from smartphone sensors mounted on an excavator.The overview of motion-based methods for the excavator activity recognition algorithms is presented in Table 3.

Hybrid methods
In some research, hybrid sensors are employed to obtain more information about the equipment and operations.In [41], a method is presented that utilizes IMUs and microphones to gather vibration and audio data to identify the activities of an excavator, such as stop, scoop, move, and swing.In [42], a deep learning-based hybrid kinematic-visual sensing approach is developed for equipment activity identification.Kinematic and visual data are gathered using built-in sensors, gyroscopes, accelerometers, and cameras of a smartphone that is mounted inside the excavator cabin.In [43], a DNN ensemble called FusionNet is suggested to classify excavator activities.This algorithm utilizes the extracted features from sensor data and video frames of on-site excavators.The summary of hybrid methods for the excavator activity recognition algorithms is shown in Table 4.

Research gaps and point of departure
As previously described, several methods have been proposed to estimate the cycle time and monitor the excavator's productivity in earth-moving operations using different types of sensors and data.Still, some challenges should be addressed.Several CV-based methods have been developed for the activity identification and productivity estimation of an excavator.However, these methods have significant restrictions and challenges, including high computational complexity, viewpoints of cameras, illumination conditions (i.e., too bright or too dark), object occlusions, number of equipment in a scene, background movements, shaking of cameras due to wind, the blurring of images due to rain, snow, dust, and fog, the requirement of installing multiple cameras to adequately cover a large worksite, and shortage of training data sets, which can notably decrease the performance of the methods.It is challenging to maintain a direct line of sight to targeted resources due to the high level of noise on dynamic construction sites.Moreover, it should be taken into account that the length of daylight in autumn and winter in some countries, such as Finland and Norway, is short [28,40,44].In audio-based methods, the accuracy of the models can be highly affected by background noise, and some equipment does not produce distinct sound patterns, making it challenging to identify its activity.In motion-based methods, IMUs are almost affordable and can be easily installed or have been already installed on different excavators.In recent years, equipment manufacturers and third-party companies have started mounting IMUs in the equipment to locate the bucket for automated machine guidance (AMG) systems.Therefore, the motionbased methods, which are not so constrained, can be a promising solution for the automatic productivity estimation of an excavator.The qualitative comparison of excavator activity recognition approaches is presented in Table 5.Most developed motion-based methods can only identify the activity of excavators without considering the operation cycle time and productivity estimation.The first significant challenge is the cycle time estimation of an excavator in the loading operation.In Table 3, two motion-based methods have been introduced to estimate the cycle time of the excavator in loading operation.In [33], the operation cycle time has been estimated with a low accuracy of 75.96%.The amount of error can bring about a huge error in productivity estimation.In [34], the accuracy of operation cycle time estimation is around 91.83%.In 20% of cycles, the error between the actual and estimated cycle times is over 3 s which can cause a huge error in the productivity estimation.Another important challenge is the lack of a reference to be able to analyze the obtained actual cycle time.The actual cycle time cannot lonely specify whether the machine is working at high or low performance since operating conditions can significantly influence the cycle time.Hence, a benchmark or reference is required to analyze the actual cycle time.To estimate the theoretical cycle time, the operating conditions, such as swing angle and digging depth, should be estimated during the operation.
Proposing a novel framework for the automatic estimation of

Method Advantages Disadvantages
Vision-based 1) Images and videos that have been recorded can be used as trustworthy documentation in the future [46].
1) These techniques are extremely sensitive to environmental conditions, including sunlight, dust, snow, rain, and fog, because they require light to record images and videos.
2) To monitor the entire worksite, a network of cameras is needed [28].
3) There should be no obstructions between the camera and the target.4) These techniques struggle to assess crowded and congested worksites with a lot of noise (such as moving backgrounds and varying lighting conditions) [47,48].5) To save images and video data using these approaches, huge storage spaces are needed.Additionally, these techniques require comparatively more computing than alternative approaches [49].6) These methods might not be used in worksites due to privacy concerns.7) These methods are relatively expensive.The expenses of cameras are within the range [$1000-$10,000] in small-sized worksites and within the range [$10,000-$100,000] in medium-sized worksites [45].
Audio-based 1) Unlike vision-based techniques, the existence of obstacles has no impact on the quality of the recorded data and no need for high computational capacity and large storage space.
2) They can cover a relatively large area and record the sounds from multiple machines, and no need to attach sensors to machines, unlike motion-based techniques.
1) They are inappropriate for crowded and noisy construction sites.The noise can significantly reduce the accuracy of activity detection.
2) These methods are not capable of accurately distinguishing between subtasks of the machine.
3) These techniques cannot easily extend to machines, such as tower cranes, which don't generate sounds.
Motion-based 1) IMUs are robust and resilient in challenging environments, in contrast to vision-based methods [50].For example, these methods can work easily without any lines of sight.
2) The accuracy and power consumption of IMUs are satisfactory [47].
3) The methods can achieve a high level of accuracy (80-100%).4) They do not have high computational complexity and can be easily implemented in real-time.
5) The expenses are within the range [$100-$1000] in small-sized worksites and within the range [$1000-$10,000] in medium-sized worksites [45].6) Nowadays, there are several commercial machine control systems, including Topcon, Caterpillar, Trimble, and Komatsu, which are using several IMUs on the machine to track the real-time bucket position.The measurements can be used for activity recognition purposes.
1) Sensors need to be directly attached to the equipment, which can be timeconsuming in large construction sites.
A. Molaei et al. excavator cycle time is the initial and primary purpose of this paper.Firstly, a supervised learning algorithm is designed to recognize the excavator activities in the loading operation using measurements from IMUs installed on different moving parts of the machine.The operation cycle time is estimated based on the sequence of recognized activities.
The average accuracy of the actual cycle time estimation using the proposed method is around 97%, which outperforms the presented methods in the literature reviews.Because there are some operating conditions, such as the digging depth and swing angle, that can significantly influence the excavator's productivity, the actual cycle time cannot solely represent the performance of the operation.In the second step, the theoretical cycle time of the excavator is estimated based on the operating conditions.Determining the theoretical cycle time of the excavator can provide a reference or benchmark to compare with the actual cycle time.This algorithm requires the online estimation of the digging depth, swing angle, and information about the excavatability level of material in the operation.In the next step, the relative cycle time is determined by dividing the theoretical cycle time by the actual cycle time.The machine's performance is categorized into three levels (satisfactory, average, and poor performance) using the relative cycle time and simple thresholding.To show the results of the proposed method, a dataset is collected using an excavator to train the classification model.Then, the proposed method is implemented on two case studies.The trained classification model is used online to recognize the machine's activities and estimate the actual cycle time.In the following steps, the swing angle, digging depth, theoretical, and relative cycle times are estimated.The relative cycle time can effectively show the productivity of the experienced operator is higher than the inexperienced operator.
The actual cycle time and relative cycle time index can be highly beneficial for worksite managers, contractors, and construction companies to track and monitor the operational effectiveness of each machine during the loading operation.They can identify project issues, solve problems as soon as possible, optimize planning and operating parameters, ensure resources are used efficiently, maximize equipment utilization, and precisely budget future projects.Also, the proposed algorithm can be an interesting feature for construction machinery manufacturers for automatic productivity monitoring.The researcher can use the relative cycle time index to compare the performance of autonomous solutions with experienced and inexperienced operators.It is highly important and challenging to demonstrate the productivity of autonomous operations is higher than manual operations.Moreover, training organizations can use the provided information as feedback to improve the skills of human operators.

Methodology
In the paper, a method is proposed to recognize the activities of an excavator in the loading operation.This classification model is offline trained using the collected dataset.The data collection procedure is extensively described in Section 3.1.1.After offline training, the model can be utilized online to recognize the machine's activities and estimate the actual cycle time.Moreover, using detected activities, the swing angle and digging depth of the operation are automatically estimated.The theoretical cycle time is computed using the BML model, information about the excavatability level of the material, and estimated swing angle and digging depth.Finally, the relative cycle time is calculated by dividing the theoretical cycle time by the actual cycle time.

Activity recognition
In the proposed algorithm, firstly, the excavator activities are identified using a supervised learning algorithm.The excavator activities in the loading operation include (1) scooping, (2) swinging loaded, (3) dumping, (4) swinging empty, and (5) idling.Motion sensors are utilized to obtain information about different movements and activities of an excavator.The flowchart of the proposed method is shown in Fig. 2.

Field data collection
In the first step, a dataset is required to offline train the classification models.In this section, the steps for collecting the dataset from an excavator are completely described.The field data are collected from a   3.The excavator is a Komatsu® PC138US with a mass of 13.4 tons and a standard mono boom configuration which is equipped with the Novatron Xsite® machine control system.The bucket is attached to the arm by using a quick coupler, and the excavator has a tiltrotator.The tiltrotator was not moved during the data collection.The heaped capacity of the bucket is 0.37 m 3 based on the Society of Automotive Engineers (SAE) standard J-296.The volume in the bucket under the strike-off plane plus the heaped volume above the strike-off plane is called the heaped bucket capacity.The schematics of heaped and struck capacities in the SAE standard (J-296) are shown in Fig. 4. The angles of repose for material above the strike-off plane in the SAE standard (J-296) are 1 : 1 (45 ∘ ).
The dataset covers a variety of working conditions, such as different swing angles, digging depth, weather conditions, and types of material.The swing angles of the experiments vary from 60 ∘ to 120 ∘ , and the digging depths increase up to 2 m.Two types of material, including sand and gravel, are utilized in the operations.The studies were carried out in different weather conditions during 18 months in a private worksite by two operators with different levels of competence.The experiments represent realistic construction operations, i.e., no directions were provided to the operators on how to perform the operation.This was done in order to increase the robustness of the proposed algorithms and reduce the effects of human operator behaviors in the classification algorithm.
Four IMUs are installed on the bucket, arm, boom, and cabin of the excavator to measure the orientation and angular velocities of the moving parts of the machine.The IMUs are precalibrated using Xsite® machine control system.The configuration of IMUs on the excavator is shown in Fig. 5.The 3-axis accelerometer and gyroscope are mounted within each sensor unit.Sensors' measurements are transmitted over the controller area network (CAN) bus.The CAN bus is connected to the MathWorks® Simulink model for data collection utilizing a Kvaser leaf light CAN to USB interface, and the data sampling frequency f s is equal to 200 Hz.
The duration of the dataset is around 75 min, which means that based on the data sampling frequency of 200 Hz, approximately 900,000 data points were collected for each channel of the sensor.An experienced operator carried out 65% of the experiments, and the remaining data have been collected by an inexperienced operator.
Each sensor unit calculates the quaternion orientation of the sensor based on the accelerometer and gyroscope measurements.Then, the joint angles between each moving part of the machine connected by the revolute joints are computed based on the quaternion measurements.The quaternion to Euler angles conversion is stated in Eq. ( 2) where q shows the unit quaternion, and ϕ, θ, and ψ represent the roll

Data windowing
In the proposed algorithm, a data windowing approach is employed to identify the short-term motions of an excavator.The position of a moving object is represented by a single data point at a single instant of time, whereas working cycles are composed of sequential motions distributed over a period of time (for instance, the scooping activity does not occur instantly but over a period).In the data windowing process, a data sequence is divided into numerous smaller, constant-sized pieces of data using a defined windowing function that is moved along the whole time-series data.In this study, a sliding rectangular windowing function with four different window sizes (0.5, 1, 2, 3 s) and with four alternative overlapping configurations (0%, 25%, 50%, and 75% overlap between two consecutive windows) are utilized.

Data annotation
In supervised learning algorithms, the data samples need to be paired with the so-called ground truth information.In the experiments, the operation is videoed using an external USB webcam installed inside the cabin of the machine connected to the MATHWORKS® SIMULINK model.The frame rate of the webcam is twenty frames per second.The recorded videos are used only for data annotation, and the classification models are only dependent on the motion information.Firstly, the data samples are manually categorized into five types of activities based on the recorded video.Secondly, the most frequent label in each window is considered the label of that window.

Feature extraction
Before performing the model training, feature extraction is conducted to extract useful information from each labeled data window in the dataset.The main concept behind feature extraction is to calculate variables from the raw data to maximize the amount of information connected to the phenomenon that a classifier will be used to model.Ten statistical time domain features were extracted from each window in the collected dataset, including (1) mean, (2) maximum, (3) minimum, (4) standard deviation, (5) mean absolute deviation, (6) root mean square, (7) peak-to-peak, (8) interquartile range, (9) skewness, and (10) kurtosis.

Feature selection
In a feature selection method, a subset of the initially extracted features is chosen to minimize the feature space, find the features that contain the most information related to the classification task, and provide a faster and more cost-effective algorithm.It should be noted that some features could not be useful because they do not contain value-adding information and can therefore be discarded for further investigation.In this research, four different subsets of features: (1) all features, (2) selected features using the ReliefF algorithm, (3) selected features using minimum redundancy maximum relevance (MRMR) algorithm, and (4) selected features using the chi-square test, are employed to train supervised learning algorithms.

Classification models
Although activity recognition algorithms are proposed using both supervised and unsupervised methods, supervised learning algorithms show better performance for equipment activity recognition [13].The characteristics and amount of data will determine which supervised learning algorithm should be utilized.As a result, there is no one best classifier, and each method needs to be assessed independently.Based on the most commonly used supervised classifiers in construction resource activity identification algorithms in previous studies, four classifiers, including a support vector machine (SVM), a k-nearest neighbors (KNN) algorithm, a Naive Bayes classifier, and a decision tree (DT), are trained using the introduced dataset in Section 3.1.1 to classify machine's activities.The accuracy metric that is utilized for the evaluation of the classification algorithms is calculated as follows: where TP denotes true positives, FP denotes false positives, FN denotes false negatives, and TN denotes true negatives.

Actual cycle time estimation
In the previous step, the data collection procedure and classification model training have been described.The trained classification model can be used online to recognize the excavator's activities.In this section, the actual cycle time of an excavator is estimated based on the sequence of activities in a work cycle.Firstly, the work cycle and cycle time should be defined.In [53], a definition for a work cycle of construction equipment is presented: "a work cycle is an activity performed in a finite time-frame by the equipment where all the states of the equipment are in the same range at the start and the end of the work cycle".The loading cycle consists of scooping, swinging loaded, dumping, and swinging empty activities.In [34], the cycle time is defined based on the time between two consecutive anti-clockwise rotations if there is one clockwise rotation between them.In our algorithm, to reduce the effects of classification errors and increase the robustness, the cycle time is equal to the time between two consecutive scooping activities if there are at least one swinging loaded activity, one dumping activity, and one swinging empty activity between them.To compare the estimated cycle time with the literature reviews, the introduced accuracy definition in [34] is employed.The real cycle time or ground truth information is obtained based on the recorded video by manually measuring the cycle times.The accuracy of the estimated cycle time is computed by the ratio of deviation between the estimated cycle time and real cycle time to the total real cycle time.The cycle time estimation accuracy is formalized by Eq. ( 4): where ti is the estimated cycle time, t i denotes the real cycle time, n is the total number of cycles.

Theoretical cycle time estimation
As previously described, the actual cycle time cannot solely represent the performance of a machine, since several factors can significantly influence the excavator's productivity.To evaluate or analyze the estimated actual cycle time, a theoretical cycle time is required.The theoretical cycle time provides us with a reference or benchmark to evaluate the actual cycle time of an excavator.In this section, the theoretical cycle time of an excavator in the loading operation is calculated based on the ongoing operation conditions such as swing angle, digging depth, and the excavability level of the material.Construction equipment manufacturers, Komatsu® [54] and Caterpillar® [51] designed two models to determine the theoretical cycle time and productivity of an excavator.The suggested model by Caterpillar® cannot be utilized in an automatic manner because it is a descriptive model that needs human input.Moreover, an industry guideline suggested another model [55].The BML guideline was designed by a common committee of the Central Association of the German Construction Companies (Zentralverband des Deutschen Baugewerbes) and the Federation of the German Construction Industry (Hauptverband der Deutschen Bauindustrie).In this paper, the BML method is utilized because it is more conservative and provides a more realistic theoretical cycle time compared to the Komatsu model which is more optimistic.
The cycle time of a hydraulic excavator in the loading operation based on the BML model is formalized by Eq. ( 5): where t theoretical is the theoretical cycle time [sec], t initial is the initial guess of theoretical cycle time [sec], f swing is the swing angle factor [ − ], and f depth is the digging depth factor [ − ].
In the BML model, the initial guess of theoretical cycle time t initial is computed based on the heaped bucket capacity and soil excavability categories.The soil excavability categories in the BML model are presented in Table 6.The initial guess of theoretical cycle time for materials with high excavability (e.g.sand and gravel) is formalized by Eq. ( 6): and for materials with medium and low excavability (e.g.hard compacted clay) is calculated by Eq. ( 7): where V CECE is heaped bucket capacity according to the standard of the Committee for European Construction Equipment (CECE).The angles of repose for material above the strike-off plane in the CECE standard is 1 : 2 ( ∼ 27 ∘ ).There is no estimation for the theoretical cycle time of the very low excavability category in the BML model [56].
In the next step, the initial guess of theoretical cycle time is modified using two factors based on the swing angle and digging depth of the operation.In the loading operation, the horizontal angle between the scooping and dumping positions is called the swing angle.In the BML method, the swing angle factor f swing is approximately estimated using Eq. ( 8): Variations of swing angle within the range of [45 ∘ , 180 ∘ ] influences ±10% variations in the cycle time.
In the BML method, the digging depth factor f depth for low and very low excavability soil types is approximately computed using Eq. ( 9): and for high and medium excavability material is approximately estimated by Eq. ( 10): For h d < 1 m, the digging depth factor is assumed to equal one.When the digging position gets deeper, it has only a negative impact on cycle time which can increase up to 20% in extreme cases (i.e. for h d > 8 m).
In order to automatically estimate the theoretical cycle time of an excavator in the loading operation, real-time automatic estimations of swing angle and digging depth at the end of each cycle are needed.In the next sections, the swing angle and digging depth are estimated based on the identified excavator activities.

Swing angle estimation
An essential variable that can significantly change the operation cycle time and the excavator's productivity in the loading operation is the swing angle.In [8], an approach is presented to estimate the operating conditions, including the swing angle, and digging depth, based on Otsu's method.Otsu's technique achieves optimal thresholding by maximizing the variance between classes.In our paper, a novel algorithm is presented to estimate the swing angle using the cabin encoder measurements and the excavator activity recognition algorithm.All excavators have been already equipped with the cabin encoder.The cabin encoder measurements specify the horizontal angle of the excavator cabin during the operation.The swing angle is defined as the absolute deviation of horizontal angles between the scooping and dumping positions.To obtain the horizontal angles of the scooping and dumping positions in the loading operation, the scooping and dumping activities must be detected.The start and end of each cycle are detected based on the proposed method in Section 3.2.During each cycle, there are four groups of activities (scooping, swinging loaded, dumping, and swinging empty).The average of horizontal angles (cabin encoder measurements) in detected scooping and dumping activities are considered as the scooping and dumping positions, respectively.The flowchart of the method for swing angle estimation is shown in Fig. 7.  A. Molaei et al.

Digging depth estimation
Digging depth is another significant variable that must be considered in the theoretical cycle time estimation.When the digging position gets deeper, it takes more time to complete the scooping task.In this paper, the digging depth is estimated based on the bucket position estimation and the excavator activity recognition algorithm.As described earlier, during each cycle, there are four groups of activities.The digging depth is equal to the minimum of the vertical axes of the bucket position estimation in the detected scooping activities.The scooping activity is detected using the trained classification model, and the bucket position is estimated using IMUs and the forward kinematics of the machine.The flowchart of the method for digging depth estimation is presented in Fig. 7.
The motion of the excavator manipulator is described by kinematic equations without considering the driving forces and torques.An excavator can be modeled as an open-loop articulated chain with a boom, arm, and bucket.In an excavator, a series of rigid bodies, known as links, are joined by revolute joints [57].The forward kinematics of an excavator is shown in Fig. 8.Each link has its own Cartesian coordinate system that moves with the link.The local coordinate system for each link is constructed based on the Denavit-Hartenberg (D-H) convention (Table 7), with the z-axis pointing in the direction of rotation of the revolute joint and the x-axis pointing at the other joint in the same link.Therefore, the y-axis direction is established using the right-hand rule [58].The angle θ i , i ∈ {1, 2, 3, 4} are computed using quaternion measurements, and the conversion formula (Eq.( 2)).Also, the length l i , i ∈ {1, 2, 3, 4} can be obtained from machine specifications.Forward kinematic equations are utilized to calculate the positions of the manipulator links given the joint angles and lengths of the links.A transformation matrix between two adjacent coordinate systems (from (i + 1) th to i th ) on a link can be stated by using the D-H convention.
where i T i+1 is the transformation matrix from (i + 1) th coordinate system to i th coordinate system, θ i+1 is the rotation angle about z i -axis, α i+1 is the rotation angle of z i -axis about x i+1 -axis, d i+1 is the offset along the z i -axis, and a i+1 is the length of the link [58].Any point in any local coordinate system can be shown in the origin coordinate system by the coordinate transformation matrix as where 0 P is the position vector in the origin coordinate system, 0 T n is the transformation matrix from the n th coordinate system to the origin coordinate system, and n P is the position vector in the n th coordinate system [58].

Relative cycle time estimation
Finally, the relative cycle time is obtained by comparing the actual cycle time against the theoretical cycle time: where t relative indicates the relative cycle time [ − ], t theoretical is the theoretical cycle time [sec] which is calculated according to the introduced model in Section 3.3, and t actual shows the actual cycle time [sec] which is estimated based proposed method in Section 3.2.The higher relative cycle time shows higher performance.Not only the relative cycle time can be used by worksite managers to analyze the performance of the excavator and operation and to improve the planning and timetable of projects, but also, this index can be utilized as feedback to evaluate the performance and skill of the human operator.

Implementation and case studies
In this section, the proposed approach is tested using the collected dataset.The classification models are offline trained using the collected dataset in Section 3.1.1.In the next step, the method is tested using two case studies in Section 4.2.In the case studies, the accuracy of cycle time estimation is investigated.Then, the swing angle, digging depth, and theoretical and relative cycle time are estimated in both case studies.Finally, the performance of experienced and inexperienced operators in both case studies is compared using the provided relative cycle time index.The algorithm has been implemented using MATHWORKS® MAT-LAB R2021a on a laptop with a 1.8 GHz Intel Core i7 CPU and 16 GB of RAM.

Classification model training and evaluation
Firstly, the accuracy of classifiers with different feature selection algorithms is investigated.Moreover, the impacts of different window sizes and overlapping configurations on the accuracy of classification algorithms are analyzed.Different subsets of the dataset must be used for model training and testing when comparing several data-driven modeling techniques.The entire dataset was split into training and testing datasets, with 70% of the data being used for training and 30% being used for testing.Table 8 shows the accuracy of different classifiers and feature selection algorithms with the associated time window and overlapping configurations.The highest accuracy is reported with the SVM classifier with the MRMR feature selection algorithm.Not only the MRMR feature selection algorithm has a higher accuracy than the ReliefF algorithm, but also it is more cost-effective and has lower computational complexity than the ReliefF algorithm.In Table 9, the accuracy of the SVM classifier and MRMR feature selection algorithm with different time windows and overlapping configurations are presented.The highest accuracy is achieved when the time window is equal to 2 s, and the overlapping is 75%.
The confusion matrix of the proposed supervised learning algorithm with the best configuration is shown in Fig. 9.In this classification algorithm, the SVM classifier with the MRMR feature selection algorithm is employed.
Moreover, the k-fold cross-validation is performed to show the
In the next sections, the trained supervised classification algorithm is utilized to online recognize machine activities, and then the actual cycle time, swing angle, and digging depth are calculated based on detected activities.

Case studies
The performance of the proposed method is illustrated by implementation in two case studies.The data of case studies are collected using the excavator and human operators introduced in Section 3.1.1.In each case study, two experiments were performed by experienced and     A. Molaei et al. inexperienced operators, and the digging depth is approximately equal to zero because the pile of material is on the ground surface.In the first case study, the swing angle is around 120 ∘ , and the type of material is sand.In the second case study, the swing angle is around 60 ∘ , and the type of material is gravel.The duration of each experiment is approximately equal to 5 min which means based on the data sampling frequency of 200 Hz, approximately 60,000 data points were collected for each channel of the sensors.The summary of specifications of case studies is presented in Table 10.Also, the operation is recorded using the camera to be able to obtain the ground truth for the actual cycle time.

Actual cycle time estimation
The cycle time estimations in the first case study are shown in Fig. 11.As shown, the cycle time in case study Exp. 1 which is performed by the experienced operator is lower than in case study Inexp.1.It can have a significant impact on the total productivity of the operation.The cycle time estimations in the second case study are presented in Fig. 12.As expected, the cycle time of case study Inexp. 2 is higher than the estimated cycle time in case study Exp. 2. The proposed method can effectively estimate the cycle time with an accuracy of less than 1.5 sec error in both case studies.The accuracy of cycle time estimations based on the introduced definition Eq. ( 4) is presented in Table 11.In the case studies, there are only 5 cycles out of 66 cycles (around 7.5%) that have absolute errors higher than one second.

Swing angle estimation
In order to calculate the theoretical cycle time, the swing angle and digging depth estimations are required.In this section, the swing angle is estimated based on the presented model in Section 3.3.1.The estimations of the swing angle in the first case study are shown in Fig. 13.In this case study, the operators tried to keep the horizontal angle between the scooping and dumping positions around 120 ∘ .The experienced operator can easily control the swing motion of the cabin without any significant variations.The estimations of the swing angle in the second case study are shown in Fig. 14.In this case study, the operators try to keep the swing angle around 60 ∘ .As shown, the proposed method in both scenarios can correctly recognize the scooping and dumping activities and estimate the swing angle.

Digging depth estimation
In this section, the digging depth of the operation is estimated using the proposed scheme in Section 3.3.2.The digging depth estimations in the first case study are illustrated in Fig. 15.The estimations are around zero because the pile of material is on the ground surface.The digging depth estimation in the second case study is presented in Fig. 16.As shown, the method can correctly estimate the digging depth during the operations.As expected, the experienced operator can easily control the bucket position, but there are a lot of variations in the bucket movements when the inexperienced operator performs the task.Although the    skills of human operators are significantly varied, the proposed supervised learning method can manage this challenge.

Relative cycle time estimation
In this section, firstly, the theoretical cycle time is calculated based on the estimated swing angle, digging depth, and the type of material.Then, the relative cycle time is calculated based on the estimated and theoretical cycle times.The theoretical, estimated, and relative cycle times in the first case study are demonstrated in Fig. 17.The theoretical cycle times of the two experiments in the first case study are approximately equal because the operating conditions are almost the same.In this figure, the area over 0.8 is colored green, which shows satisfactory performance, the area between 0.6 and 0.8 is colored yellow, which presents average performance, and the area less than 0.6 is colored red, which represents poor performance.The colors have been inspired by traffic lights to be easily understandable for everybody.In the case study Inexp. 1, which is operated by the inexperienced operator, in most of the cycles, the relative cycle times are in the yellow area, which shows the average performance.In the case study Exp. 1, which is operated by the experienced operator, almost in all cycles, the relative cycle times are more than 0.8, which represents satisfactory performance.The relative, estimated, and theoretical cycle times in the second case study are demonstrated in Fig. 18.In the case study Inexp.2, which is operated by the inexperienced operator, the relative cycle times are in the yellow area and show the average performance.In the case study Exp.2, the relative cycle time shows satisfactory performance.

Discussion
Nowadays, automatic productivity and performance monitoring of the HDMMs is significantly required.As per available information, an automated procedure for assessing the relative cycle time and operational effectiveness of an excavator during the loading operation under prevailing operational conditions does not appear to exist.The proposed method can automatically monitor the cycle time and performance of an excavator in the loading operation.This information can guide worksite managers and contractors to spot the issues as soon as possible.It can bring about a massive reduction in operation time, improve time scheduling, and increase productivity.Moreover, the operating conditions can affect the cycle time and productivity of the machine.The estimation of operating conditions, such as swing angle and digging depth, using identified machine activities is another novelty of the paper.
In this paper, to train the classification models, a dataset is collected using one excavator operated by one experienced and one inexperienced operator in a private worksite.Also, the test dataset is collected using the same machine.The amount of data is one of the most important limitations of the proposed method and generally data-driven algorithms.The dataset should be extended by collecting data from different excavators in different sizes and should be collected by different operators with different levels of competence.Different swing angles, digging depths, types of material, and weather conditions should be considered in the data collection phase since the operating conditions can affect the accuracy of the algorithm and also the productivity of the machine.Another limitation of the proposed method is labeling the dataset.Labeling is an essential and time-consuming step for supervised classification algorithms.
The type of material can affect the cycle time and productivity of an excavator in the loading operation.In the proposed method, it has been assumed that the excavability level of the material is known.The known material is one the limitations of the proposed method.This information A. Molaei et al. should be given to the system by project managers or operators.An artificial intelligence (AI)-based method for automatic material identification in construction sites can be a promising solution and solve some challenges in productivity estimation algorithms.Another variable that should be estimated during the loading operation is the amount of material in each cycle.The weight of the material in the bucket can be estimated using dynamics bucket payload estimation methods, or the volume of material can be estimated using high-advanced sensors such as light detection and ranging (LiDAR).Generally, the bucket volume estimation algorithms are more costly and complex and have less accuracy compared to the dynamic bucket payload estimation algorithms.

Conclusions
In this paper, an algorithm is proposed to estimate the actual, theoretical, and relative cycle times of an excavator in the loading operation.Firstly, a supervised learning algorithm is presented to recognize the excavator activities.To collect the orientation variables and angular velocities of an excavator, four IMUs are installed on different moving parts of an excavator.Then, several classification algorithms and feature selection methods are tested on the collected dataset.Also, the effects of different time windows and overlapping configurations on the activity recognition algorithm are analyzed.The obtained accuracy of the activity recognition is around 95%.In the next step, the cycle time of an excavator is estimated based on the sequence of activities identified using the trained model.In the third step, the theoretical cycle time in the loading operation is calculated based on the BML model and current operating conditions to be able to evaluate the actual cycle time.To compute theoretical cycle time, the swing angle and digging depth estimations are required.Two schemes are proposed to estimate the swing angle and digging depth using the recognized activities.Then, the relative cycle time is estimated based on the actual and theoretical cycle times.The index of relative cycle time can demonstrate the operational effectiveness of an excavator in the loading operation.A higher relative cycle time shows higher performance.Finally, the presented method is assessed by the implementation on two case studies.In each case study, two experiments were performed by experienced and inexperienced operators.The operating conditions, such as swing angle and type of material, are different in the two case studies.The results show that the method can effectively estimate the cycle time of an excavator in the loading operation.The average accuracy of actual cycle time estimation is around 97%.It has been illustrated that the index of relative cycle time in the operation which is performed by the experienced operator is higher than the inexperienced operator.The relative cycle time can be divided into three levels (satisfactory, average, and poor) using a simple thresholding.
The proposed method can be utilized in different excavators to automatically monitor the cycle time, productivity, and operational effectiveness of the machine.The concept of actual, theoretical, and relative cycle times can be further extended to actual, theoretical, and relative productivity by adding information about the quantity of material in operations.These methods can lead to significant cost savings in the overall worksite process through better utilization of the machines.Worksite managers also get key insight into the productivity of individual machines and their operators, thus allowing them to further train the operators and improve the process.In the future, the presented method is planned to be extended to other applications such as excavators in trenching and grading operations, front-end loaders in short and long loading cycles, etc.

Fig. 3 .
Fig. 3. Excavator used in data collection.In picture, cabin (1), boom (2), arm (3), and bucket (4) are highlighted with red boxes [8].(For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

(
rotation around x-axis), pitch (rotation around y-axis), and yaw (rotation around z-axis), respectively[52].The global angular velocities are directly measured using the gyroscope in the IMU.The local angular velocity of each moving body is also calculated from the global angular velocities.The local angular velocity is the true angular velocity of the individual body part from which the movement of the other machine parts has been subtracted.The local angular velocity describes the movement of the measured body part caused by the operator moving the specific part that is being measured.The global angular velocity, on the other hand, consists of all movement caused by the machine.The local angular velocities and orientation variables are visualized in Fig.6.The quaternion data was discarded from further processing, and the joint angles and angular velocities of the machine parts are utilized as input data in the excavator activity recognition algorithm.The used variables include the angular velocities of four IMUs (3 axes per each sensor unit), the local angular velocity of the boom (ω 2 ), the local angular velocity of the arm (ω 3 ), the local angular velocity of the bucket (ω 4 ), the pitch angle of the boom (θ 2 ), the pitch angle of the arm (θ 3 ), and the pitch angle of the bucket (θ 4 ).

Fig. 6 .
Fig. 6.Local angular velocities and orientation variables are visualized on an excavator's side profile.

Fig. 7 .
Fig. 7. Flowchart of methods for swing angle and digging depth estimations based on activity recognition algorithm.
et al.robustness of the proposed classification algorithm.In applied machine learning, cross-validation is mostly used to evaluate how well a model performs on unseen data.A dataset is randomly divided into k groups, or folds, of approximately similar size, and one of the folds is used as a holdout set while the remaining k-1 folds are used to fit the model.The final estimate is determined by averaging the results of the k holdout sets after this process has been repeated k times.In this analysis, k is considered equal to 4. The results of k-fold cross-validation for different classification models and feature selection algorithms are presented in Fig.10.The accuracy of the classification algorithms is similar to the presented results in Table

Fig. 12 .
Fig. 12. Cycle time estimations in the second case study.

Fig. 14 .
Fig. 14.Swing angle estimations in the second case study.

Table 3
Overview of motion-based methods for excavator activity recognition.

Table 4
Overview of hybrid methods for excavator activity recognition.

Table 6
Material categories in BML model.

Table 8
Accuracy of different classifiers and feature selection algorithms with associated time window and overlapping configurations (the highest accuracy is highlighted in bold).

Table 9
Accuracy of the SVM classifier and MRMR feature selection algorithm with different time windows and overlapping configurations.

Table 10
Specifications of case studies.
a Inexp.stands for inexperienced operator.b Exp.stands for experienced operator.

Table 11
Accuracy of cycle time estimations based on Eq. (4) for each case study.
Fig. 13.Swing angle estimations in the first case study.A.Molaei et al.