A dynamic reward-enhanced Q-learning approach for efficient path planning and obstacle avoidance in mobile robotics

1. Introduction

Path planning is a fundamental challenge in mobile robot (MR) safe and efficient navigation in an unknown environment that may contain obstacles. Path planning encompasses a diverse array of approaches, with the optimal choice contingent upon the unique attributes of the environment and the robot in question. Contrary to the known environment [1], where the robot possesses knowledge of the terrain and path planning can be straightforward, in a partially known environment, only partial mapping is available and the robot grapples with uncertainty regarding concealed obstacles [2]. Completely unknown environment [3] is the most challenging scenario where the robot confronts uncharted terrain, devoid of any prior mapping. Each of these scenarios requires distinct path-planning techniques to address the specific challenges posed by varying degrees of environmental familiarity. Path planning in a known environment is relatively straightforward, as the robot knows where all the obstacles are. However, path planning in a partially known or unknown environment is more challenging, as the robot must first map the environment and then find a path that avoids obstacles. Partially known environments are often the most practical, as they allow the robot to benefit from the knowledge of previously mapped areas while still being able to navigate in new areas.

The selection of a path planning algorithm significantly impacts a robot’s navigation in terms of safety, efficiency and robustness. Path planning in robotics is broadly categorized into static and dynamic approaches. In static planning, the robot charts a fixed route around stationary obstacles in the environment [4]. Conversely, dynamic planning adapts to moving obstacles, requiring the robot to continuously adjust its path to navigate safely through the evolving surroundings [5]. While static planning is generally simpler, it may not suffice when obstacles are in motion, rendering dynamic planning essential for safe navigation.

This study introduces the dynamic reward-enhanced Q-learning (DRQL) algorithm, merging Q-learning techniques with dynamic reward mechanisms. This innovative approach allows robots to navigate unknown environments, adapting to environmental changes and task variations during movement. Empirical results demonstrate that this algorithm outperforms traditional Q-learning, showcasing improved convergence speed, optimization and adaptability. Our contributions can be outlined in three main aspects. First, we introduce dynamic rewards, leveraging information limitations in unknown environments. Static rewards relate to state node characteristics, while dynamic rewards vary based on target point distance, preventing blind searches and excessive exploration, enhancing learning efficiency. Second, the DRQL algorithm encompasses three steps: (1) exploration, (2) exploration and exploitation and (3) exploitation, addressing limitations of classical Q-learning in path planning. Third, experiments validate the effectiveness of DRQL in tackling complex path-planning challenges encountered by MR in diverse environments.

The subsequent sections of this paper are arranged as follows: the second section contains an exploration of related works in the field. Section 3 describes the formulation of the modified Q-learning algorithm, describing its four core components. Section 4 details the simulation methods used. Finally, the final section summarizes the key findings and indicates potential avenues for future research.

2. Related works

Following an extensive assessment of the literature, the navigation methodologies in robotics are categorized into two principal paradigms: classical approaches and reactive strategies (see Figure 1).

Historically, robotics has heavily focused on classical approaches, such as cell decomposition [6], roadmap approach [7] and artificial potential field [8]. However, these methods suffer limitations in computational complexity, susceptibility to local minima, uncertainty handling, reliance on precise data and the need for accurate real-time sensing. As a result, doubts persist regarding their practicality in real-time applications. Efforts to enhance these approaches through strategies like artificial potential fields and hybrids have not consistently surpassed reactive methods, especially in real-time scenarios.

Reactive strategies excel in navigating unfamiliar environments, leveraging their simplicity, adaptability to uncertainty, efficient behavior and real-time performance, often outperforming classical methodologies. Meta-heuristic methodologies revolutionize path planning by iteratively generating candidate solutions and selecting the best-fit trajectory for execution, encompassing various methods like genetic algorithm, simulated annealing, Tabu search, particle swarm optimization, ant algorithm, bacterial foraging optimization and bee algorithms. Yet, despite their advantages over classical methods, these approaches are not devoid of limitations. Genetic algorithms [9], while effective, can face challenges in complex environments due to their reliance on population-based optimization, potentially struggling with computational intensity in scenarios with extensive search spaces. Simulated annealing [10] might face limitations in swiftly adapting to rapidly changing environments due to its gradual cooling process, potentially leading to suboptimal paths or slower convergence in dynamically evolving scenarios. Tabu search [11] might struggle in navigating complex and high-dimensional spaces due to its reliance on memory structures, potentially limiting its efficiency in certain intricate environments. Particle swarm optimization [12] can suffer from premature convergence and getting stuck in local optima, hindering its ability to thoroughly explore complex search spaces efficiently. Ant algorithms [13] might struggle with scalability and large search spaces due to their reliance on pheromone trails, potentially leading to suboptimal solutions or increased computational requirements. While bacterial foraging optimization (BFO) [14] presents strengths in exploration and optimization, its sensitivity to parameters, slower convergence and potential challenges in dynamic environments might limit its suitability for real-time and highly dynamic robotics. Bee algorithms [15] might face challenges in handling dynamic environments efficiently due to their reliance on fixed communication patterns among agents, potentially limiting their adaptability to real-time changes. The firefly algorithm [16], although an effective optimization technique in certain contexts, presents drawbacks in path planning and obstacle avoidance for mobile robotics. The algorithm’s performance can be affected by parameter tuning, and finding the right balance between exploration and exploitation can be challenging.

Fuzzy logic, while adept at managing uncertainties and enabling adaptive decision-making for MR in intricate terrains [17], faces limitations in computational intensity due to complex rule bases, challenges in representing dynamic uncertainties, dependency on expert knowledge and struggles in highly dynamic settings, requiring potential integration with other methods to address these constraints.

Neural networks, known for their ability to learn complex patterns [18], face challenges in MR path planning and obstacle avoidance due to reliance on extensive training data, potential interpretability issues, susceptibility to overfitting or underfitting in diverse environments and computational complexity, calling for hybrid or complementary approaches to mitigate these limitations.

The A* algorithm, known for its efficiency in finding near-optimal paths [19], faces challenges in scaling to complex environments with high-dimensional spaces or intricate obstacles, potentially struggling in dynamic settings or when heuristic estimates are inaccurate.

Reinforcement learning, applicable in various environments, particularly in path planning, features Q-learning as a prevalent algorithm, creating state-action pairs with associated Q-values denoting anticipated rewards for actions in specific states [20]. The learning process involves the agent navigating through trial and error, initially exploring random actions and assessing their resulting rewards. Over time, it identifies rewarding actions within specific states while balancing the trade-off between trying new actions (exploration) and choosing known rewarding ones (exploitation) [21]. In path planning, the agent is incentivized for finding collision-free paths but penalized for actions resulting in collisions [22]. Despite its potency in path planning, Q-learning’s computational demands and parameter selection are crucial considerations. Reinforcement learning stands out for path planning due to its adaptability in diverse environments, showcasing versatility and adeptness in acquiring navigation skills, particularly in challenging and dynamic settings [23]. Reinforcement learning, while beneficial for path planning, poses challenges like computational complexity, demanding substantial resources, requiring careful hyperparameter selection and involving time-consuming learning iterations to develop effective navigation strategies in varied environments [24].

In summary, diverse path planning algorithms offer unique advantages for robot navigation in unknown environments yet face persistent challenges: ensuring safety in unknown environments using general rules, redundant calculations in consecutive searches due to a lack of prior knowledge, slow convergence rates and limited dynamic path planning. Addressing these challenges requires an algorithm enabling efficient navigation in unknown environments with strong generalization, rapid convergence and reduced computational load, such as our introduced DRQL algorithm.

3. Designing the modified Q-learning algorithm with four key components

The Q-learning algorithm has four important elements: state, action, reward and Q-table. The definition of these elements is specific to the application context of the Q-learning algorithm. Figure 2 presents reinforcement learning (RL), which is a machine learning paradigm focused on learning optimal actions through interactions within an environment to achieve a certain goal. Its framework comprises the following components:

1. State (s): The current situation or configuration of the environment that the agent perceives. It represents all the relevant information necessary for decision-making.

2. Action (a): The choices available to the agent in each state. Actions lead to transitions from one state to another.

3. Reward (r): The immediate feedback the agent receives from the environment after taking an action in a certain state. It quantifies the desirability of the agent’s action. For example, if a robot navigates to a hazardous location near obstacles, he will receive a negative reward. On the other hand, reaching the destination gives a positive reward.

4. Q-function (Q): Estimates the value of taking a particular action in each state. It helps in decision-making by evaluating action values.

The RL system operates through an iterative process, where the agent interacts with the environment, observes states, takes actions, receives rewards and updates its policy and value functions based on these experiences. The objective is for the agent to learn an optimal policy that maximizes cumulative rewards over time. This learning occurs through exploration (trying new actions) and exploitation (using learned knowledge to select the best actions).

Running example: In Figure 3, the map’s size is specified as 4×4, and the robot’s objective is to navigate from its starting position (1,1) to the goal (4,4). The simulation allows the robot movement in four directions – forward, backward, left and right – while avoiding collisions with obstacles or the environment’s edges. For recording purposes, state nodes are numbered based on the robot’s exploration sequence. The environmental state nodes are categorized into four distinct types: the starting point, forbidden points, the target point and free points. Figure 3 illustrates forbidden points where the obstacle space is depicted by the black area.

4. Simulation

We compare our proposed dynamic reward solution DRQL against a conventional approach that relies on a static reward mechanism. To facilitate this comparison, we begin by elucidating the methodology employed for computing the static reward. The static reward can be calculated as shown in equation (3):

1. C₁, St = Sg: This condition indicates that if the current state (S_t) is equal to the goal state (S_g), a static reward C₁ is assigned. This reward is given when the robot reaches its intended destination without any proximity to obstacles.

2. -C₁, d_obs = 0: If the distance between the robot and the nearest obstacle (d_obs) is zero, meaning there is no separation between the robot and the obstacle, a static reward C₁ is given. This situation represents a collision with an obstacle, so a reward is assigned.

3. C₂, d_t < d_t-1, d_obs ≠ 0: This condition states that if the current distance to the target location (d_t) is less than the previous distance (d_t-1) and there is some non-zero distance (d_obs) between the robot and the nearest obstacle, a static reward C₂ is assigned. This reward encourages the robot to get closer to the target while avoiding obstacles.

4. -C₂, d_t > d_t-1, d_obs ≠ 0: Conversely, if the current distance to the target location (d_t) is greater than the previous distance (d_t-1) and there is some nonzero distance (d_obs) between the robot and the nearest obstacle, another static reward C₂ is given. This reward motivates the robot to move further away from obstacles while still progressing towards the target.

In summary, these static rewards provide a way to guide the robot’s behavior based on its current state, proximity to the goal and proximity to obstacles, helping it make decisions that lead to successful path planning and obstacle avoidance.

Table 2 shows the parameters incorporated into Equation (2), i.e. α and γ, and the variables used in Algorithm 1, such as N and ξ. The specific values are set to control various aspects of the learning and decision-making process of the robot, such as the balance between exploration and exploitation.

In all experiments, the configuration of MR was always represented by q(x; y), with x and y representing coordinates. Table 3 describes three configurations (Config₁, Config₂ and Config₃) of the test environment used for the experiment. All these configurations share a uniform starting point and endpoint (goal). Each configuration of the test environment consists of n obstacles called O_j(x_j, y_j), each obstacle positioned at a coordinate point (x_j, y_j). These carefully designed test environments have been designed to thoroughly evaluate and analyze the effectiveness, performance and accuracy of the DRQL learning algorithm.

To maintain the reliability of the experimental results, we iteratively reproduced the same experiment 20 times. Afterward, we calculate average values that include the number of moving steps and the average reward. Table 4 provides data on the number of moving steps required for the MR to reach the endpoint under these three configurations (Config₁, Config₂ and Config₃).

In this context, the “Moving Step Count for SRQL” represents the number of moving steps the robot takes when using a static reward mechanism. The SRQL, which refers to the static reward Q-learning algorithm, is very similar to the algorithm previously introduced DRQL, which differs only in the methodology of the calculation of the reward, which is calculated statically. Similarly, the “Moving Step Count for DRQL” indicates the number of moving steps when employing a dynamic reward mechanism.

The key observation here is that the dynamic reward approach results in significantly fewer steps (lower moving step count) compared to the static reward approach across all three configurations (Config₁, Config₂ and Config₃). This reduction in steps indicates that the dynamic reward strategy is more efficient in guiding the robot to reach the goal point. The percentages provided (e.g. 9.23, 20.78 and 12.05%) represent the extent to which dynamic reward reduces the number of steps compared to static reward for each respective configuration.

Table 5 presents data on the average reward achieved by a MR under different configurations (Config₁, Config₂ and Config₃).

The key observation here is that, for each configuration, the dynamic reward approach results in a lower average reward compared to the static reward approach. This difference is quantified as a percentage reduction in the average reward. For instance, in the first configuration (Config₁), the average reward for the dynamic reward approach is 12% lower than that for the static reward approach.

However, it’s important to note that despite the lower average reward values, the dynamic reward strategy is more efficient in terms of path planning and obstacle avoidance, as indicated by the lower number of moving steps (moving step count) required to reach the endpoint, which was discussed in a previous explanation. This suggests that the dynamic reward approach provides a better trade-off between reward and path efficiency.

Figure 4 presents results comparing the performance of two algorithms, DRQL and SRQL, over a series of iterations. The values represent the accumulated rewards or Q-values for both algorithms at specific iteration points.

At the beginning of the iterations (Iteration 0), both DRQL and SRQL start with an initial reward value of 0, which is expected as they have not yet learned the optimal policy.

As the iterations progress, both algorithms begin to learn and improve their Q-values; however, notable differences emerge. DRQL demonstrates a more rapid learning rate compared to SRQL. By iteration 40, DRQL achieves a Q-value of 1.3, while SRQL reaches 0.9, indicating that DRQL learns faster and accumulates higher rewards.

The divergence between the two algorithms continues throughout the iterations. DRQL consistently outperforms SRQL in terms of accumulated rewards, suggesting that the incorporation of dynamic reward mechanisms enhances the learning efficiency and performance of the algorithm compared to the static reward approach. These results emphasize the effectiveness of dynamic rewards in guiding the learning process of an agent in a reinforcement learning environment.

5. Conclusion

We introduce innovative solutions to the persistent challenges of long convergence rates and extensive planning cycles encountered by MR in unknown and complex environments. Using dynamic rewards in the Q-learning framework called the DRQL, this approach significantly improves navigation performance. DRQL strategically combines the inherent learning capability of Q-learning with an adaptive and dynamic reward mechanism. As a result, this methodology reduces the need for a comprehensive exploration and accelerates convergence rates. Extensive simulations confirm DRQL’s superiority over the existing methods. It manifests accelerated convergence toward an optimal action strategy that requires shorter time and exploration steps. In addition, DRQL has consistently obtained higher average rewards, which means that it is more efficient in road planning in an unknown environment.

In our ongoing research, we are exploring avenues to refine the DRQL approach by optimizing parameters (α, γ) in Equation (2), focusing on adaptive techniques to dynamically adjust these values based on real-time feedback.