Multicriteria evaluation of the trajectory of a wheeled mobile robot – a case study

The problem of maintaining accuracy of autonomous movement of vehicles and robots is an important element of keeping the right level of traffic safety. This applies not only to autonomous cars, but also to construction or other industry mobile robots that will soon appear in the human environment. With that in mind, the authors present a multi-criteria assessment process, which can be used to assess the qualitative and quantitative movement of such robots, in order to better identify passage errors, as well as improve their way of moving. The authors wanted to check the effect of the set power level on engines, repeatability of subsequent runs, and the impact of the level of contamination of the surface on the stability of the robot movement. The article describes the results of the robots movement assessment based on (1) the surface areas of the routes envelope, (2) the augmented average route length and (3) the number of trajectory intersections. The results obtained were compared to the research scenarios – differences in set power for motors and the state of contamination of the surface


Introduction
The progressive automation of manufacturing processes leads to creation of a new type of devices aimed at reducing the more unreliable and unpredictable human factor. An example of this type of activities can be the increasingly used autonomous or remote-controlled machines. These machines require proper preparation in terms of their working environment, as well as defining possible operations and rules along with determining the required degree of accuracy of their implementation.
The methods of motion tracking, its recording and subsequent analysis, are important issues undertaken by many researchers. [1] describes a method for predicting the trajectory of an autonomous vehicle in order to avoid collisions in traffic. The method combines a motion model, constant speed with acceleration of the vehicle and recognition of manoeuvres. A technique based on wavelet analysis of cleaning up the data collected on the trajectory of the vehicle and reconstruction of lost data is presented in the article by [2]. The author of [3] includes contextual information from the driving scene, e.g. road structure, and sets of trajectory data collected from a test vehicle, to model the behaviour of an autonomous vehicle on the basis of its trajectory.
The object of the authors' considerations was the Wheeled Mobile Robot (WMR) constructed of Lego Mindstorms elements and programmed to perform strictly determined manoeuvresforming a movement loop of the "lazy eight with parking" type [4]. Initial attempts to describe manoeuvres, assuming the perfect coverage of angles and distances delineated by the robot expressed in the parameters of programming methods, showed a very large influence of unknown factors on the robots' movement (e.g. instead of the planned turn by 45o, the robot performed a turn from 25 to 35 degrees).
The object of the authors' considerations was the Wheeled Mobile Robot (WMR) constructed of Lego Mindstorms elements and programmed to perform strictly determined manoeuvresforming a movement loop of the "lazy eight with parking" type [4]. Initial attempts to describe manoeuvres, assuming the perfect coverage of angles and distances delineated by the robot expressed in the parameters of programming methods, showed a very large influence of unknown factors on the robots' movement (e.g. instead of the planned turn by 45o, the robot performed a turn from 25 to 35 degrees).
In the research described in [5], Microsoft Kinect v2 was used to record the trajectory of the object in 3D (man's movement performing kinaesthetic rehabilitation exercises). The motion quality assessment was carried out at three levels: (1) based on a logarithm of probability of correct execution of a sequence of movements, (2) rescaling these values into success expressed as percentage, (3) analysis of time segments in a window of defined length.
The aim of the article is to examine applicability of assessment methods with vector indicators of the motion quality of a WMR-type robot as a whole, without the time segmentation. Application was made of a method of determining solutions not dominated in the Pareto sense [6] and searching for the optimal solution in the sense of min-max [7] as well as the modified lexicographical method. For this purpose, the following quantitative criteria for robot's trajectory evaluation have been proposed: firstly (C1) the surface areas of the routes envelope figure, secondly (C2) the augmented average route length and thirdly (C3) the number of trajectory intersections. The evaluation criteria did not apply to single laps like in [8] but captured information from many laps (in the set conditions of other variables). Application of the author's method of individual trajectories concatenation allowed to generate an external and internal contour of the robot's motion during subsequent laps, which led to the delineation of the area beyond which the robot did not venture.
The following research hypotheses were put forward:  H1: Increasing the driving power of the WMR robot's motors increases the motion imprecision.

Construction of the test stand
The test stand consisted of the following components: a board (75x150cm) surrounded by a barrier with a marked starting point, a camera with a support frame, a WMR robot with loaded software executing the basic task ("lazy eight with parking"). The block diagram of the constructed station with the presentation of functional dependencies is shown in Fig. 1.

The construction of the experimental model
The WMR model was created on the basis of a set of modular Lego Mindstorms parts in the EV3 version (Fig.  2). The model used in the research was equipped with a two-wheeled differential drive with an additional support point in the form of a sphere, acting as a free turning wheel. The robot had no steering wheels, but two fixed wheels with the same axle. This type of drive system is similar to the differential gear used in cars, where each of the drive wheels can receive different speeds but each drive wheel is powered separately. The differential control model is widely used in robotics because of the ease of use together with relatively simple and cheap construction. An example of the use of such a control model can be seen in a multipurpose loader such as BobCat.
This construction was already used by the authors in research on discovery learning of two age groupskids aged 5-10 and students aged 20-30 [9]. Both groups agreed about the great didactic potential of such device.

Preparation of the basic task
The basic task of the WMR was to travel on a specific track, performing manoeuvres described by the duration and turning angle ( Fig. 3 black line). The track chosen for the robot was the "lazy eight with parking" one, due to the variety of operations occurring there.

Scenarios of the experiment
The research scenarios assumed measurement of the trajectory of the robot's movement under different environmental conditions (passing through clean and dirty surfaces) and with different value of the motors power (MP). The type and condition of the road surface have a determined effect on behaviour of the vehicle [10].
The experiment was conducted on the basis of four scenarios:  S1: driving on a clean surface at 50% MP,  S2: driving on a dirty surface at 50% MP,  S3: driving on a clean surface at 100% MP,  S4: driving on a dirty surface at 100% MP.
The authors accepted as an initial condition that they did not implement mechanisms for correcting the route, such as the above-mentioned collecting of data from the environment [3], feedback control or trajectory predicting techniques [1] of a nonholonomic WMR. In addition, the authors did not consider such factors as the wear of mechanisms and motors, or the change of MP values caused by battery usage. Consequently, only data about the motion of the WMR with no feedback control and without any additional enhancements or programming intervention like in [11,12] were collected.

Description of the multi-criteria assessment method
The following set of criteria was adopted in the multicriteria evaluation:  C1: the surface areas of the routes envelope figure ([cm 2 ], minimisation), the article "An optical method for the evaluation of the programmable electric vehicle trajectory" (in Polish) [4],  C2: the augmented average route length ([cm], minimisation),  C3: the number of trajectory intersections ([-], maximisation).

Criterion C1
Criterion C1 is the surface area created by the outer and inner contour marked by generating the envelope based on the course of all the available robot trajectories for each of the adopted scenarios (Fig. 3). It was assumed that if this field had a smaller value, it meant that the robot's movement was more repeatable.

Criterion C2
Criterion C2 is the augmented average route length calculated on the basis of the contour (external and internal) of all the robot laps for each of the adopted scenarios. It is calculated as half of the contour length, and is actually the average value reflecting the generalised trajectory (Fig. 4). It was assumed that if the calculated length was closer to the trajectory of the basic task (Fig.  3) then the robot's movement was more similar to the programmed curve. Fig. 4. Visualisation of the augmented average route length (in green) and the glued contour (in red), for all the robot laps according to scenario S1.

Criterion C3
Criterion C3 is the number of trajectory intersections when determining the contour envelope from all the robot laps for each of the adopted scenarios (Fig. 5). This number indicates the number of segments from different trajectories that were used to generate the contour. It was assumed that the greater the number of intersections is, the closer individual trajectories are to each other. Moreover, the more they penetrate each other, a greater repeatability of the robot's movement is.  5. List of the 6 trajectories of the robot's journey: bluescenario S1, purplescenario S2.

Multi-criteria assessment
The multi-criteria assessment relied on two methods that implement various paradigms and do not require normalisation of measures of individual criteria to dimensionless values. In the first method, all criteria were equal. The search was for solutions that were not dominated in the Pareto sense [6] , and then for an optimal solution in the sense of min-max [7]. In the second one, the assessment modified by the lexicographic method required establishing the hierarchy of criteria validity. The method was looking for the optimal solution due to the criterion values in accordance with the accepted hierarchy. After determining the first solution, it was removed from the set of acceptable solutions and the best solution was re-determined without changing the previously adopted order of criteria validity.

Software preparation
The task was developed in the LabView Lego MindStorms software provided by the manufacturer, using a set of blocks.
The authors modified the parameters of functions performed by the robot's control program until the shape of routes drawn by it began to reflect the route of the basic task. The fragment of the pseudocode presenting the operations carried out by the robot as part of the basic task is shown in Fig. 6:  The motors list includes the ports of connected robot motors, while the list of buttons list contains the ports of connected buttons.  The start() function is called only once when the program is started, and the update() function is executed every processor cycle.

Data processing
Acquisition of data regarding the quality of subsequent journeys took place in the following form (Fig.7.):  Step 1 -Recording the motion of the robot with a camera set above the test stand, with the lens directed perpendicular to the surface, at a distance enabling the entire track to be covered without the need to change the position of the camera during the experiment (Fig.  7-1),  Step 2 -Conversion of the recording from the raw format of recording data in MPG format and MPEG Audio codec to the file in the AVI container and codec H264-MPED-4 AVC with Pazera Free MP4 to AVI Converter 1.9 ( Fig. 7-2),  Step 3 -Use of motion tracking on the source video file and selected reference object with HitFilm 2 Express (Fig. 7-3),  Step 4 -Removing the distortion path caused by curving and deforming the lens with PhotoShop CS3 from the image (Fig. 7-4),  Step 5 -Vectorisation of the robot motion trajectory with AutoCAD using the manual method, due to the desire to obtain more accurate vectorisation results ( Fig.  7-5).  Step 6 -Normalisation of data to real dimensions by applying to the master grid with AutoCAD ( Fig. 7-6).

Fig. 7.
Diagram of the acquisition process of the robot motion data.

Implementation of the experiment and evaluation of results
The implementation of the task consisted of three trials for each of the presented scenarios. The final effect was to obtain 12 charts of subsequent journeys by scenarios S1-S4. The relevant graphs (obtained for the same research scenario) were imposed on each other in order to concatenate them, which led to the determination of the external and internal contour of the area occupied by the moving WMR robot. After completing all the data obtained, the evaluation of the accuracy of implementation by the WMR of the tasks was started, performing the qualitative and quantitative assessment on the basis of the adopted set of criteria.

Qualitative assessment
The qualitative assessment consisted in comparing with reference to the trajectory of the basic task of the area separated by envelopes generated on the basis of available trajectories for each scenario. The difference between the robot trajectories on a clean and dirty track could be clearly determined. One quality criterion was defined consisting in a visual assessment of trajectory displacement for each scenario relative to the base route and other routes. Figures 8 and 9 show the prepared sets of routes with the base task route shape (black curve) for all scenarios.
The biggest differences compared to the basic task route is shown by the journeys on clean surface with the use of 100% MP (Fig. 9, S3). This indicates WMR susceptibility to errors resulting from the braking and twisting method. If no mileage correction is applied, all errors made during manoeuvres accumulate. Fig. 8. Comparison of the shape of the crossing contour according to: a) scenario S1blue, b) scenario S2purple. Intersectionorange. This is particularly visible in the final phase of the WMR movementthe parking manoeuvre. In the case of laps on dirty surface with 100% MP, a WMR tendency can be seen to go outside the planned route (Fig. 9, S4). This clearly indicates the relationship between the state of the surface and the given speed of the robot's movement. Fig. 9. Comparison of the shape of the crossing contour according to: a) scenario S3red, b) scenario S4green. Intersectionorange.
As the speed increases, the tyres of the robot moving on the dirty surface lose their grip, resulting in the robot not being able to stay in the area constituting the interior of the figure, whose outline reflects the trajectory of the underlying task.

Quantitative assessment with Single-criteria evaluation
The quality assessment was presented in relation to each of the adopted criteria. The calculated values for criterion C1the surface areas of the routes envelope figures are shown in Figure 10. The values of criterion C1 indicate that the area increases as the MP increases and when we change the type of surface from clean to dirty. This means that there is more slippage between robot's wheels and the ground as well as when the power of the robot's motors increases both when there is surface contamination.
The calculated values for criterion C2, the augmented average route length, are shown in Figure 11. The values of criterion C2 indicate that the average trajectory lengths for scenarios S2 and S3 are smaller than the route of the base task, while for the others they are larger. The situation becomes clearer when we consider information from the qualitative assessment ( Fig. 8 and  9).
More visible shifts of the movement trajectory for S1 and S3 scenarios from the trajectory of the basic task are the reason for the larger values of criterion C2. The high value for the S4 scenario results from increased slippage due to the dirty surface and high MP value.
The calculated values for criterion C3, the number of trajectory intersections, are shown in Figure 12. The values of criterion C3 indicate that the number of intersections for laps along the route with the dirty surface is significantly lower than in the case of clear ground.
This means that the individual trajectories for the S2 and S4 scenarios are more distant (displaced), which indicates that the robot's repeatability is less predictable.

The method of determining optimal solutions in the Pareto sense
The research scenarios considered are the four solutions studied. Non-dominated solutions are scenarios S1 and S3. S2 and S4 are dominated, which means that all criteria values were worse than for S1 and S3 (Fig. 10-12). Therefore, with the more surface contamination, the robotic movement is more unstable. Solutions S1 and S3 are, in the Pareto sense, equal. The search for the optimal solution in the sense of min-max [7] (Chebyshev standard was applied) was solved by the S1 solution. Conducting an additional analysis based on individual values of the components of solutions S1 and S2 (for S1 two values of criteria: C1 and C3 are better than for S2), we can also conclude that the trajectory of the robot on the clean surface is more repeatable. The same conclusions refer to the comparison of solutions S3 and S4.

The lexicographic method of determining optimal solutions
In the lexicographic evaluation, on the basis of the above analysis of the values of individual criteria, two different ordering criteria were proposed. The first order (P1): C1, C3, C2 and the second order (P2): C3, C1, C2. Optimisation using both orders leads to an indication of S1 as the best solution. After withdrawing this solution from among the others, the best is S3. The received solutions show that the most stable traffic is obtained when the robot moves on a clean surface with less power of the motor. Movement of the robot with more power of the motor is more stable on a clean surface than on a dirty one.

Conclusions
The H1 hypothesis about increased motion inaccuracy when the robot's MP increases was confirmed. This is evidenced by the obtained values of criteria C1 and C3 for all research scenarios. The H2 hypothesis about the stabilisation of the robot's motion while reducing track contamination was confirmed based on the values of criteria C1 and C3 for all research scenarios.
Multicriteria evaluations carried out by the methods of searching for optimal solutions in the Pareto sense and using the lexicographic method also confirmed both hypotheses.
Criterion C1 defined as the surface areas of the routes envelope figures, and criterion C3the number of trajectory intersections, well describe the set of WMR trajectories and are useful for linking MP and surface contamination with the drawn motion curve. The definitions of criteria C1 and C3 are the original method of aggregation of individual robot trajectories. Verification of the criteria based on the conducted experiment indicates that the criterion C2augmented average route length, does not provide any relevant information for the robot's movement assessment, because it does not give unambiguous results. The presented criteria and methods in the multi-criteria assessment of the robot actual trajectory are possible to apply regardless of the robot control model.