Prediction of Service Level Agreement Time of Delivery of Goods and Documents at PT Pos Indonesia Using the Random Forest Method

ABSTRACT


INTRODUCTION
Trust and customer satisfaction in the courier business or the delivery of goods and documents, one of which is determined by the accuracy and speed of delivery of shipments to recipients. Therefore, the expedition always tries to make predictions about how long it will take for the shipment to reach its destination. The existence of a target delivery time is a form of delivery from the service provider to the customer in the form of a Service Level Agreement (SLA) [1] which is contained in each shipment receipt in the form of estimated delivery time information. Thus, the sender will be able to predict when the goods or documents must be sent so that they can be received by the recipient according to the expected target time.
PT Pos Indonesia (Persero) is the oldest courier and financial service provider in Indonesia and needs to make efforts to continuously improve the quality of its operations according to the times. PT Pos Indonesia (Persero) as one of the freight forwarding agents always includes a receipt for each package or document shipment that has been assigned a shipping SLA with the amount of travel time according to the product chosen by the customer. Special Express Post/Regular Post is a service with a relatively high number of transactions with quite a variety of SLAs between 2-14 days. In this service, there are several destinations that experience delays in delivery not due to delays as a result of several SLA determinations which are calculated based on the distance between hubs manually. As for external factors such as air and sea transportation to connect between islands there are no significant obstacles, because shipments can be transported regularly to the destination location. Meanwhile, land transportation is under internal control, departures and arrivals are regulated and the hours are set.
This SLA prediction must be carried out in order to restore customer trust in PT Pos Indonesia by providing a promise of a more precise travel time than the predetermined SLA. The SLA that has been set at this time has never been analyzed, whether the SLA is appropriate or not since the SLA was enacted in 2019. (Source: Pos Indonesia Operations Division, 2022). Therefore an analysis is needed that aims to predict an unknown value based on existing data called predictive data analysis [2] Problems arise when the specified SLA does not match the predetermined time as happened in shipments to areas outside Java, namely Kalimantan, Sulawesi, Maluku, and Papua. With the target of sending special express mail in 2022, only 91.2% of the 98% target set was achieved (source: pos indonesia operation dashboard, 2022). Therefore, the SLA determination that has been previously determined has not been re-examined using machine learning methods. in order to obtain more precise SLA estimates. With machine learning through the right method, it is expected to be able to predict the best SLA, which has an impact on customer confidence in the set travel time.
Based on the problems that arise in predicting the delivery of goods as described above, researchers are trying to find the best solution to overcome these problems. There are several analytical approaches that can be used to make predictions, one of which is data mining [3]. Data mining is a process for finding new relationships that have patterns, meanings, and habits based on data owned using mathematical, statistical, artificial intelligence and machine learning techniques. [4] One of the methods contained in data mining is the Random Forest method.
Selection of Random Forest as the method used is based on various journals that have good accuracy scores with similar discussions such us, Prediction of Bus Travel Time Using Random Forests Based on Near Neighbors [5], Using machine learning and big data approaches to predict travel time based on historical and real-time data from Taiwan electronic toll collection [6], Bus travel time prediction based on deep belief network with back-propagation [7], Taxi Travel Time Prediction Using Ensemble-Based Random Forest and Gradient Boosting Model [8], dan Developing a Travel Time Estimation Method of Freeway Based on Floating Car Using Random Forests [9]. Of the five journals, the majority of Random Forest has a high accuracy value.
Random Forest is the result of the development of the Classification and Regression Tree (CART) method which applies the bagging or bootstrap aggregating method and random feature selection [10]. The basic technique of the Random Forest method is the decision tree. In other words, Random Forest is a method that consists of a set of decision trees and then forms a random forest that is used to classify or predict data. Based on the description of the background and the problem, the researcher is interested in conducting research on increasing the accuracy of the SLA by predicting travel time using the Random Forest (RF) method at PT Pos Indonesia (Persero).

RESEARCH METHOD
The research stage begins with data collection by understanding business processes to know and understand business processes at PT Pos Indonesia (Persero) Malang City branch, followed by data understanding and data preparation. After data collection, system design was carried out, followed by the experimental model with the random forest (RF) algorithm. Next is to look at the results of the evaluation and analysis of the results. The last conclusion is drawn. The research design process flow can be seen in Figure 1  The process of sending goods or documents is carried out by several hubs (shipments are collected at certain offices designated as hubs for distribution to other destination cities) except for local shipments or shipments that are passed by available means of transportation. In this study the focus of shipping goods is from the Java region to areas outside Java, such as Kalimantan, Sulawesi, Maluku and Papua. Figure 2 below is the flow of delivery of these goods and documents. In Figure 2 above, it is explained that to the destination city through several specified hubs. The delivery process is not based on point to point, but through a mechanism for collecting shipments in one predetermined hub to be forwarded to several hubs until they reach their destination. Besides the distance and the number of hubs, the data that has the potential to affect the implementation of the model is the weight of the shipment, whether the weight of the shipment also has a significant effect on the prediction calculations made.
The data used in this study are primary data, obtained from the PT Pos Indonesia delivery monitoring dashboard on the official web page of the operations department (https://board.mile.app) to retrieve operational data for goods or document shipments as an external variable, and data from the calculation of transportation time between hubs as an internal variable. The data is initially in the form of CSV (Comma Separated Values) and then processed in the form of a Sheet to facilitate the next process. Before making predictions with machine learning algorithms, the raw data will be processed first so that it is ready for use. There are two stages, namely loading the dataset and preprocessing the data so that it is mature. Pre-processing the data here will see whether the data has a missing value or not. If there is a missing value, the data row will be deleted. Data that has no correlation will also be removed from the dataset. In addition, normalization is applied so that the data has a uniform range of values. Data will also be divided based on certain presentations to be used as training data and test data. Internal and external variable data are combined into one to facilitate data processing.
The amount of data processed is 18,769 data for shipments to Kalimantan, Sulawesi, Maluku and Papua sent from PT Pos Malang City Branch for special/regular express postal services from January 2022 to December 2022. Furthermore, the system design explains how the system performs SLA predictions for shipments of goods or documents using the Random Forest (RF) Algorithm. The system design process flow in this study can be seen in Figure 3. In the random forest algorithm, the stage begins with the preliminary elimination and ranking stage, this stage draws a random sample of size z, where z is 2/3 of the amount of data, then builds a tree based on that sample. The tree formed is based on the gini criterion principle as the separator. Next, take samples of size z to s times (boostrap) and build a tree from each sample. A collection of s trees will form a forest, to obtain more accurate results, you can take more samples to form an r forest. After the r forest is formed, the value of the important variable (VI) is calculated for each variable, the VI value is used to rank and eliminate variables based on the specified threshold. The value of the important variable (VI) is related to the OOB (out of bag) error value. OOB error is an error obtained from the results of data testing that is tested on the model formed from the results of the formation of each tree. This testing data is data that is not included in the boostrap sample in the initial step. [11] In its application, RF produces which variable is the most dominant which can then be used as the next predictive calculation. While the final elimination stage is based on the selection results at the variable selection for interpretation stage, to determine the most influential variable where these variables can significantly reduce errors. The variable is said to be selected if the decrease in the OOB error is greater than the threshold. To measure the value of accuracy in this study using MAE in order to know the average difference between the absolute value of the actual with the predicted (forecasting) value. The smaller the MAE value, the better the model is in forecasting. Meanwhile, to calculate the percentage of the prediction accuracy value, MAPE is used which has a formula almost the same as MAE by adding division and percentage. The smaller the percentage of MAPE values, the better the prediction results. The following is the RF flowchart in this study.

RESULTS AND DISCUSSION
To explain the results of this study, the authors use a tool called Google Collab as an option, because Google Coollab is Python-based, web-based without installing on a PC/Laptop and easily presented with application support and data collected together on Google Drive. Processed spreadsheet data is stored in Google Drive as a data source, then Google Collab retrieves the data source from the url where the data is stored.
There are several variables that will be processed, to produce the best value, 2 scenarios are carried out, the first scenario is by processing all the influential variables, the second scenario is processing the most influential variables. For more details, which variables have an effect can be seen in table 1 below To generate predictions, one of the reference attributes is needed. The attribute used is actual time as a label and the amount of training and testing data is determined by a ratio of 75:25, with the hope that the more training data the better results will be obtained.
Before running scenarios 1 and 2, it is necessary to prove whether the current SLA is compared to the actual time whether the results are good or not. Therefore, it will be proven by MAE and MAPE between the current_sla and actual_time variables first. From the results of data processing using the Googla collab, the accuracy test results are obtained in table 2 below: The results above indicate that the current SLA is no longer suitable for use because there is a high difference in days, namely 2.26 days and the accuracy rate is only 45.05%.

Scenario 1
Scenario 1 using 5 variables and 1 reference variable without calculating which variable is the most important. In each prediction scenario, several experiments are needed by making changes to the tree or Random Forest iterations to measure the level of accuracy. This method is executed by changing the iteration values 100, 500, 1000 and 2000 which are presented in table 3. Based on the results of 4 experiments with different iterations, the results obtained were an average of 83.7% where for each the accuracy obtained from MAE was 0.72 which was a good result, while MAPE was obtained on average 16.13%. With these results, the level of prediction accuracy using FR is good because MAPE is in the range of 10-20%. While the results of the data plot between actual time and predictions are shown in Figure 5 below usig 1000 iteration. In Figure 5 above the actual time is red and the prediction is blue, from a distance of 500 km to 2,000 km, the average time accuracy is almost the same as the prediction. Meanwhile, above 2,000 km there are some differences between actual and predicted. If you look at the average, it will appear even more clearly as shown in Figure 6 below. In figure 6 for a distance of 3000 there is a significant difference from the prediction, while at a distance above 3500 km the prediction is getting further away from the actual.

Scenario 2
In scenario 2 it is processed first from several variables to determine which variable is the most important. This variable is then used as a parameter for calculating predictions because it has a high influence value. The results of determining the most important variables in Figure 7 below:  Figure 7, you can see 2 important variables, namely deviation and distance, these two variables are used in calculating predictions for scenario 2 with results as shown in Table 4 below: In scenario 2 all iterations produce the same value, namely MAE 0.7 MAPE 84.2%, from these results it can be seen that selecting important variables can produce better results. For visualization of plots and scenario 2 averages can be seen in Figure 8 and Figure 9 below:  Figure 8 shows that the actual and predicted plots are closer than in scenario 1 in Figure 5. For distances of 3,000 km and 3,500 km there is a difference that is not as sharp as shown in Figure  9, this means that with scenario 2 the prediction results are better.
All of predictions can be summarized in table 5 below which will show the results from the accuracy of the test to the results of scenarios 1 and 2. Calculating important variables will produce better accuracy, as in the results of scenario 2, but in other machine learning research, the level of accuracy of predictive models developed with certain methods depends on the type of data used as a case study. So it is possible that a method that has a certain level of accuracy, will not fully achieve the same level of accuracy when applied to different data types.

CONCLUSION
In the courier and logistics business world, the application of machine learning can be applied and can be used as a measuring tool to determine travel time predictions in compiling a Service Level Agreement (SLA) [12] for each shipment destination. Previous SLA data using manual calculations by adding up the number of days will be very useful for making predictions by displaying the actual travel time compared to the "difference" from the time or SLA set.
The prediction results with Random Forest have a good level of accuracy, which is 83.86% of the average 4 trials. This shows that the Random Forest method is the right choice for managing the existing data model [13] at PT Pos Indonesia. In the implementation of determining the next SLA for delivery destinations to the islands of Kalimantan, Sulawesi, Maluku and Papua from Malang, this prediction can be used to make changes to several destination areas so that the accuracy of the SLA to customers can be fulfilled properly, of course these results can be developed for cities that other.
This research has the potential to be further developed in the future. Besides the need to use more complete data, for example data on economic and environmental conditions that affect goods delivery activities, researchers recommend using other machine learning methods, or a combination of several machine learning methods, to increase the level of prediction accuracy.