The Comparison of K-Nearest Neighbors and Random Forest Algorithm to Recognize Indonesian Sign Language in a Real-Time

. Purpose: Comparing 2 models or prototype programs which can recognize Indonesian Sign Language System or Sistem Isyarat Bahasa Indonesia (SIBI) fonts from hand gesture and translate it’s into writing Messages in real-time. Methods: After selecting datasets and reprocessed by the researcher into 1 dataset, which are a combination of several sign image datasets of the SIBI letters images available on the Kaggle website, the dataset is converted into landmarks. The landmarks are divided into 26 sign classes and preprocessed to a total of 19,826 rows of data, and then divided into 67% training data and 33% test data. Next, both K-NN and Random Forest algorithm are implemented into different program and get tested into 2 different tests, model evaluation and real-time. At the end, the result is compared to see the increase of accuracy level of both K-Nearest Neighbors (K-NN) and Random Forest algorithm. Result: The constructed and trained model is then evaluated and the results of Precision, Recall, Accuracy, and F1-Score are 99.88% using the Random Forest algorithm. The results of real-time program testing with the K-Nearest Neighbors algorithm get higher results, where the average accuracy value reaches 99%. Novelty: From the result shows that the model built with the Random Forest algorithm is superior, but the K-Nearest Neighbors algorithm is better in real-time testing. Therefore, image data and its diversity should be increased, in order to improve recognition accuracy. The program could be enhanced by adding a function where the program can recognize hand gesture, not only one or two hands but also can recognize a hand gesture with movements so the program can recognize static and dynamic letter (


INTRODUCTION
Sign language is a language that prioritizes communication using body language and lip movements.Sign language is generally used by the deaf community, combining movements of the hands, arms, body and facial expressions to express what they want to say.In 2018, Adi Kusumo Baroto, a researcher in Indonesian Sign Language, revealed to Berita Tempo that there are two sign languages used by the deaf community in Indonesia, namely Bahasa Isyarat Indonesia (BISINDO) and the Sistem Isyarat Bahasa Indonesia (SIBI).However, as the references and records regarding the BISINDO sign language were very minimal, the Indonesian government then created a sign language which has been standardized for use in various schools or special institutions, namely the Sistem Isyarat Bahasa Indonesia (SIBI) [28] [35] [36].
Like any languages in general, sign languages also have dictionaries or translators that have been agreed upon by both the government and the deaf community.The dictionary of the Sistem Isyarat Bahasa Indonesia (SIBI), contains movements and forms of hand signals that represent letters and existing terms.With our current technology, SIBI sign language dictionary are no longer has to be in a written record form, but in a form of a program that can recognize hand gestures from the sign language.In other words, it's very possible if we made a program which can be recognize hand gestures movements of people--even in a real time moment, by using the data-training which has been embedded into the Machine Learning as knowledge based for the Sign Language recognition program.
The methodology used in research on hand gesture recognition can be applied by various algorithms, such as Deep Learning algorithm, namely Convolutional Neural Network (CNN), Support Vector Machine (SVM), K-Nearest Neighbors (K-NN), Random Forest, and other algorithms [5][10] [21][23][24] [29].K-Nearest Neighbors (K-NN) algorithm is an algorithm that is often used on gesture recognition.In previous researches, according to Taunk et al. in [32], K-Nearest Neighbor (K-NN) algorithm is a machine learning algorithm that is non-parametric and a lazy learning algorithm [11][14] [19][26] [36], which means that the algorithm does not make assumptions about the distribution of the underlying data, it does not use training data to create a model, but only stores and remembers the training data, so it can be said to be lighter in terms of computing load and time [10][11][19] [33] [36].Meanwhile, the Convolutional Neural Network (CNN) method was known less effective due to the heavy computational load or time [7][24] [31].
Random Forest is an ensemble of numerous tree predictors-based approach in which N decision trees are independently built by bootstrapping the data.Put forward by Leo Breiman and Adele Culter, random forest, an integrated non-parametric learning machine with the purpose of sample training.The training speed is fast, the implementation of the algorithm is simple and effective, a classifier with high accuracy can be generated for varieties of data.Nevertheless, will be overfitting when dealing with some classification and regression problems with loud noise, the property weights produced by random forests on the data with different value properties are not credible [37][41].
This research was conducted to develop previous research on algorithm programs in sign language recognition of letters from hand movements.Some of these researches includes study was conducted by Damatraseta et al., which discussed real-time hand gesture recognition from videos using the Convolutional Neural Network (CNN) algorithm; research using the K-Nearest Neighbors (K-NN) algorithm by Alksasbeh et al [33]; and research using the Random Forest algorithm by Wiraswendro et al [34].Specifically, this research was conducted to develop a program to hand gesture recognition of Indonesian Sign Language System (SIBI) sign language letters from hand movements directly (real-time) by applying two classification algorithms, namely the K-Nearest Neighbors and Random Forest algorithms, with the aim to get the best comparison of them especially in accuracy values, both in the model or the results terms.In addition, this research expected be a reference for creates communication media that is able to translate sign language letters from gestures into written form with the result to be a connector communication between deaf and non-deaf communities.

METHODS
The topics of this research studies include the SIBI sign language, hand gesture recognition, K-NN algorithm, and Random Forest algorithm.The data collected is in the form of a dataset sourced from the Kaggle site, which is sets of images of SIBI letters which will then be read to form coordinate points (hand landmarks).

Data Used
The dataset consists of 26 hand shape alphabet images from 'A to Y' letters of Indonesian Sign Language System (SIBI), except 'J' and 'Z' letters, as described in Figure 1.

Training and Sampling Dataset
The 26 alphabet letters are divided into 26 folders according to the name of the letter class folder.Each folder contains image data with name and type adjusted.The frequency every single letter of dataset distributed as 1880 number letter image data with the total of 48,880 dataset.The frequency of SIBI letters image shown in Table 1 Dataset division was processed using the Scikit-learn library were 13,287 training data and 6,545 test data.
The dataset converted into landmarks, and would be read to form coordinate points (hand landmarks).
According to Mediapipe's official documentation, Mediapipe's hand landmark model detects 21 coordinates based on the knuckles of the detected hand, as in the following image Figure 2. The whole coordinates of image data will be extracted using Mediapipe Solutions, then will be manipulated into a dataset by The Pandas library, such as visualized in table form as below Figure 3.The next process, landmarks are divided into 26 sign classes and preprocessed to a total of 19,826 rows of data, and then divided into 67% training data and 33% test data.The dataset will be processes and test using the model which is built using K-Nearest Neighbors, Random Forest algorithm and the Python programming language version 3.9 with Jupyter Notebook, Mediapipe and scikit-learn libraries.To visualization flow for data testing the System of Indonesian Sign Language (SIBI) is described in below Diagram Flow Figure 5.

Figure 5. Data Testing Flow
The testing process is divided into two processes, that is model evaluation testing and real-time program testing.The results of the testing processes are carried out by comparing the results of the precision, F1 Score, and recall produced during the evaluation test, as well as the accuracy of real-time hand gesture recognition of SIBI letter.

Comparative of K-Nearest Neighbors and Random Forest Algorithms
According to [11] The Random Forest algorithm is a development of the decision tree algorithm, which is a collection of classification tree structures as following: {ℎ(,  ),  = 1, . . .} The k values are random vectors that are distributed independently and identically, and each tree produces one of the most popular results given the input x.Quoting from Breiman, the classification of the Random Forest Algorithm is determined based on the voting results of the most popular results.The tree collection was obtained through the bootstrap aggregating (bagging) process [2][4] [7][8][38].This algorithm works by combining n decision trees to build a random forest.Then, this process is continued with each tree to produce predictions, which will then be carried out by a voting process using the mode (the value that appears most frequently) to select the final prediction result [22][24][31] [39].
To initialize the RandomForestClassifier class, the decision tree is determined randomly with the n_estimators command and controlled using the random_state parameter as much as 1234.Subsequently, after a hyperparameter search completed, then based on the best results of hypertuning model process, it is determined that the number of parameters such as decision trees, max_depth, and max_features is 100 trees.

RESULTS AND DISCUSSIONS Real-time Testing
Program testing was carried out in real-time using a Dell G3 3500 Laptop webcam with an RGB HD Camera type and a camera resolution of 1280 x 720 (HD) at 30 fps.In this testing process, letter recognition was measured based on the distance between the hand and the webcam.To start the hand detection and recognition process, the program captured landmark coordinates as view on the Figure 6, then the model will be using them as data 'memorized'.Based on Table 1 and Figure 7 above, it can be concluded that the model applied with the Random Forest algorithm has a higher value than the model applied with the K-Nearest Neighbors algorithm, which is model using the K-Nearest Neighbors (K-NN) algorithm resulted a score of 99.60% for its F1-Score, Recall and Accuracy values; meanwhile, the model using Random Forest algorithm resulted a score of 99.88% for its F1-Score, Precision, Recall and Accuracy values.However, the level of accuracy obtained in this study was able to exceed the level of accuracy of previous research.

Discussion
The model of this research can only recognize SIBI letters through landmark patterns of static hand sign language image, and do not recognize from videos with dynamic movements.So that, it can only recognize 24 letter classes from 26 hand sign language A to Z of the alphabet letters in real time, except for the J and Z letters, because of these two letters require movement to demonstrate.The development of this research also can only compare the result of accuracy; precision; F1-Score; and recall by applying each K-Nearest Neighbors (K-NN) algorithm and the Random Forest algorithm to the model then testing separately, so it has not yet reached the experimental stage of combining the two algorithms.

CONCLUSION
Main objective of this research is to compare two algorithms, are K-Nearest Neighbors and Random Forest in sign language recognition of SIBI letters from real time hand movements, as well as increasing results the accuracy of research previously, especially in terms of Accuracy; Precision; F1-Score; and Recall.In this our model, from 26 alphabet letters, only uses the 24 letter SIBI image data dataset (without the letters 'J' and 'Z') with the total of 48,880 datasets, then processed to landmarks coordinates with a total of 19,826 rows landmarks data.Then, the landmarks data divided into 13,322 (67%) training data and 6562 (33%) test data.The model with the application of the K-Nearest Neighbors (K-NN) algorithm resulted in a score of 99.60% with an increase in accuracy of 2.48% from previous research was a score as 97.12%.Meanwhile, the model using Random Forest algorithm resulted in a score of 99.88%, an increase from 97.12% or in other words, it resulted increase in accuracy of 1.28% from previous research.

Figure 3 .
Figure 3. SIBI Letter Image Dataset Furthermore, letter class selection is carried out, where each letter class is targeted at 1000 columns containing coordinates or landmark points from the dataset hand image, as shown in Figure 4.

Figure 4 .
Figure 4. Landmarks of SIBI Letter by Mediapipe Solutions [12][18][19][33][36], the way K-NN algorithm works is by determining the number of neighbors (k) which will be used as an estimate for class determination.The formula for calculating similarity weights with the K-NN algorithm is to use the Euclidean distance (D) formula from training data (x) and test data (y), which is written in the following equation:

Figure 6 .
Figure 6.Real-time SIBI Letter RecognitionResultAs previously mentioned, the aim of research using two the two algorithms namely K-Nearest Neighbors and Random Forest, is to see a comparison of them, especially in terms of Accuracy, Precision and Recall evaluation tests.Referring to the model we created in this research, using a dataset of 24 SIBI letter classes which is further divided into 13,322 training data and 6562 test data, calculations measured using the Multi Class Confusion Matrix produce data interpretation[39][40] as presented in Table2and below Figure7.Table2.Comparison of the calculations results between KNN and RF Algorithm in Accuracy, Recall, Precision, and F1-Score

Figure 7 .
Figure 7. Chart Evaluation Test Comparison of Model KNN and RF Algorithms

Table 1 .
. Frequency of SIBI Letters Image

Table 2
and below Figure7.

Table 2 .
Comparison of the calculations results between KNN and RF Algorithm in Accuracy, Recall, Precision, and F1-Score