Visual servoing control of omnidirectional mobile robots based on disturbance observer

This study studies a critical issue that linear velocity and steering angular velocity of the wheels of omni-directional mobile robot are easily affected by the unknown external disturbances such as wind direction and road flatness during the visual servoing stabilization task. To overcome such challenging problems, this study proposes a disturbance observer based visual servoing control strategy. In configuration, the controller is divided into two parts: the quasi-min-max model predictive controller is designed to accommodate the state constraint and input constraint of the system, and complete the visual stabilization task without disturbance. A nonlinear disturbance observer is designed to estimate the external disturbance and feedback compensate the effect of the actual disturbance on the wheel speed. The simulation results and real experiments show that the proposed control strategy can stably accomplish the visual servoing stabilization task even with a large external disturbance.


Introduction
Visual servoing has always been a hot research field in mobile robot technology, 1,2 which mainly controls the position and pose of mobile robots through visual sensor feedback information at the current instance, so that mobile robots can achieve the functions of stabilization, trajectory tracking, and navigation. In recent years, there have been many researches on visual servoing stabilization methods for mobile robots. Liang et al. 3 proposed an adaptive image-based trajectory tracking control scheme for wheeled mobile robots with uncalibrated fixed cameras. Yan et al. 4 developed a hybrid visual trajectory strategy for wheeled mobile robots equipped with vehicle-mounted vision systems, in which 2.5D visual servoing framework was used to enhance trajectory tracking behavior and helped preserve visual objects within the camera's visual range. In He et al., 5 a model predictive controller design method based on quasi-min-max strategy was proposed for the limited mobile robot visual servoing system to achieve asymptotic stability of the closed-loop vision system of mobile robots.
The above research approaches are based on the mobile robot visual servoing task without considering the disturbance of the external environment. However, in practical situations and applications, the structure of mobile robot is complex, and mobile robot itself is easily affected by the changes of system parameters, coupled with the external environment disturbance, such as external force disturbance, wheel skid, and other factors, which increase the difficulty of mobile robot system control. How to reduce the external disturbance of the system and improve the control of mobile robot have been critically difficult problems in robot control analysis and design. To effectively suppress the influence of disturbance on the system, adaptive control method, disturbance observer method, and the adaptive method combined with the two methods can be adopted. [6][7][8] In recent years, active disturbance rejection control (ADRC) technology has achieved good results in robot control by establishing an observer to estimate disturbances because it does not need an accurate model of the object. Ma et al. 9 estimated the system disturbance through the extended state observer, and introduced the 1 State Key Laboratory of Robotics and Systems (HIT), Harbin Institute of Technology, Harbin, China disturbance compensation term into the active disturbance rejection controller, so as to reduce the uncertainty of system parameters and the impact of external disturbances on the system. Wang et al. 10 from Hebei University, China designed an active disturbance rejection controller based on linear extended observer for the unknown wheel sliding and skidding phenomenon of mobile robots, which enabled mobile robots to achieve better trajectory tracking effect. In addition, there is a kind of model-based control technology, such as sliding mode control, which also has good anti-interference effect. In Li et al., 11 in order to overcome the torque disturbance of wheeled mobile robots, a sliding mode speed tracking controller with disturbance observer is proposed in literature, and finally the mobile robot can track the desired path. In Huang et al., 12 a sliding mode control method based on high-order disturbance observer is designed for the general underactuated system to improve the disturbance estimation ability of the underactuated system. However, when the synovial track enters the dead zone, the sliding model controller will be an unpredictable chattering interval. Therefore, Qiu et al. 13 proposed an uncalibrated visual servoing adaptive model predictive control method based on disturbance observer, which solved the visual servoing tasks of mobile robots under the conditions of unknown internal and external parameters of the camera, external interference and system constraints, and improved the antiinterference performance of the system. Yu et al. 14 took the two-wheel differential mobile robot model as the object and designed a predictive control estimation method based on nonlinear disturbance observer to compensate the slow-varying disturbance of the system, and got a good path tracking effect. Sun et al. 15 proposed a model predictive control method for disturbance suppression and designed two disturbance observers, one to estimate the unknown disturbance and the other to compensate for the disturbance of the known harmonic frequency, finally realizing the nonoffset tracking of the trajectory. To compensate the external disturbance of the multi-input multi-output robot system, Liu and Liu 16 constructed a nonlinear observer to estimate the disturbance of the external system, and adopted the backstepping method to suppress the disturbance and improve the performance of position tracking. In Chen et al., 17 disturbance estimation technology is comprehensively summarized, including disturbance observer control (DOBC), active Disturbance Rejection control (ADRC), disturbance regulation control (DAC), etc., and the future development direction of disturbance compensation technology is also discussed.
Inspired by the quasi-min-max predictive control method, 5 this paper studies the visual servoing stabilization control problem of omnidirectional mobile robot in the presence of external environmental disturbances. Different from nonholonomic two-wheel differential mobile robot, omnidirectional mobile robot can move 360°in all directions, so it is more suitable for application in the scene with limited range of activities and has greater practical significance. The major contribution of the study can be briefly described below.
(1) This paper establishes completely the error model and kinematics model of omnidirectional mobile robot with three off-centered direction wheels. The full model integrates the omnidirectional wheeled mobile robot and visual system. (2) In order to improve the effectiveness and robustness of the control, we designed the quasi-min-max MPC based on a non-linear disturbance observer that provided robust performance regardless of the modeling uncertainties and perturbation. (3) The simulation results show that the proposed strategy has the advantage of anti-disturbance performance.
The rest of this paper is organized as follows. Section ''Problem formulation'' presents the visual servoing error model and omnidirectional mobile robot kinematics model. Section ''Design of control strategy based on disturbance observer'' presents the detailed design process based on disturbance observer control strategy. Section ''Simulation'' shows off the simulation results to demonstrate the performance of the control strategy. Finally, Section ''Conclusions'' concludes the study.

Nonholonomic OWMR description
As shown in Figure 1, an omnidirectional wheeled mobile robot (OWMR) is assumed to move in a plane, which is independently driven by three off-centered direction wheels 18 and equipped with a monocular camera above the OWMR. O w -X w Y w Z w is the world coordinate frame attached to the ground, O r -X r Y r Z r is the robot coordinate frame fixed to the OWMR, the origin O r is located at the mass center of the OWMR, O c -X c Y c Z c is the camera coordinate frame attached to the camera, the origin O c is located at the Z r axis of the robot coordinate frame, and the X c and Y c axes are in the same direction as the X r and Y r axes. Assuming that the OWMR moves on the plane under the nonholonomic constraints of pure rolling and non-slipping, then the speed and the pose of the omnidirectional mobile robot satisfy the following relationship: where (x, y) is the coordinates of the robot's mass center in the world coordinate system, u is the angle between the world coordinate system and the robot coordinate system, and counterclockwise is positive.
T is the speed of the omnidirectional mobile robot, that is, the control input, v x and v y are the longitudinal and lateral linear velocities of the omnidirectional mobile robot, and v is the angular velocity of the omnidirectional mobile robot. Define the coordinates of feature point P in the world coordinate system as (X, Y, Z), in the robot coordinate system as (x r , y r ), and in the camera coordinate system as (x c , y c , z c ). The equation can be obtained according to the geometric relationship in Figure 1: Since the feature point P is stationary, X and Y are constants. Taking the derivative of equation (2), and then combining with equation (1) we can obtain the change of robot motion position of feature points in the camera coordinate system, that is, the robot-camera-target model is as follows: There are various disturbances encountered frequently in the actual environment in the visual servoing task of omnidirectional mobile robot, such as lateral wind, ground slip, and so on, which will affect the linear velocity and angular velocity of mobile robot. When external disturbances are considered, model (3) of the omnidirectional mobile robot system can be written in the following general form: Where Visual servoing error model of OWMR As shown in Figure 1, P is the feature point the world coordinate frame. We assume that (x p , y p ) is the image coordinate of the projection point p of feature point P in the image coordinate system, (u p , v p ) is the corresponding image pixel coordinate, (u 0 , v 0 ) is the center of the image plane, f is the focal length of the camera. According to the image projection model, 19,20 the relationship between the camera coordinate system and the image coordinate system can be derived by where f x = f=Dx, f y = f=Dy, f is the focal length of the camera, and Dx and Dy are actual size per unit pixel on the axes X p and Y p , respectively. By normalizing the image coordinates, we have that with the normalized image coordinates (x p #, y p #).
Taking the derivatives of (6) and combining (5), it is obtained that Let T be the velocity of the camera on the robot. Since the robot moves in plane, that is, v z = v x = v y = 0 and v z = v, the motion velocity of p is obtained by Define the posture variables (r 1 , r 2 ) of the robot with its desired posture (r 1 * , r 2 * ) as , where (u p * , v p * ) are the desired coordinates of the point p in the image pixel coordinate frame. Then we have that The error signals of the robot can be defined as where u * is the desired orientation of the robot and u can be the estimated using the motion-estimated technique.
Taking the derivative of (11) and using (10), the visual servoing error system of the robot is described by Kinematic model This paper considers an omnidirectional WMR consisting of three eccentric steering wheels, whose top view is shown as Figure 2, where M is the mass center of the WMR, and A, B, and C are the fixed points of the three wheels relative to the robot body. These fixed points are located on the central axis of the vertical steering axis of each wheel when moving. The three wheels of the omnidirectional WMR are assumed to be exactly the same and the angles between MA, MB, and MC are identical to be 2p/3. In Figure 2, l is the distance from the center of the vertical axis of the wheel to M, a is the constant angle from the X r axis of the robot coordinate system to MA, d, r, b, and u are the eccentric distance, radius, turning angle, and rolling angle of the wheel, respectively. Note that the angles are positive when rotating counter-clockwise and the turning angle b is from MA to the axis of the wheel. The conversion relation between the rotation velocity _ u and deflection velocity _ b of the three wheels and the velocity v r of OWMR can be obtained as follows 21 where T are the rotation velocity of the three wheels respectively, the deflection velocity of the three wheels respectively, and Combining (12), the visual servoing error system of the mobile robot with respect to the rotation velocity and deflection velocity of the wheel can be obtained, that is, To ensure safety and effective execution of visual servoing systems, the robot should satisfy the velocity constraints and the visibility constraints for each sampling time k, that is, e 1 min 4e 1 (k)4e 1 max , e 2 min 4e 2 (k)4e 2 max ð16aÞ where i = 1, 2, 3 and the subscript ''min'' and ''max'' indicate the lower and upper bounds of variables, respectively. The objective of this paper is that the OWMR is driven to reach the feature point while meeting the constraints (16), and in the design process, the error e(k) and the velocity u(k) eventually approach zero.

Design of control strategy based on disturbance observer
In order to ensure the successful completion of the visual servoing task of omnidirectional mobile robot, the state constraints and output constraints of the visual system need to be considered. Therefore, a stability control law u(k) is designed for system (15) using the quasi-min-max MPC strategy. But in the actual system, the disturbance will disturb the rotation velocity and deflection velocity of the robot. By designing a nonlinear disturbance observer to estimate the external disturbance, and feedback compensation control input, the stabilization control of OWMR is finally realized. The visual servoing control block diagram of omnidirectional mobile robot based on disturbance observer is shown in Figure 3.
In the framework shown in the figure, the visual servoing task is divided into two sub-problems: (1) Without considering the existence of external interference, model mismatch, and parameter uncertainty in the model, the quasi-min-max MPC controller 5 is designed to obtain the control quantity based on the error state quantity at the current moment, which ensures the stabilization control of omnidirectional mobile robot, and satisfies the state constraint and input constraint control in the process of motion; (2) When there is lateral wind, complex ground environment and other conditions, that is, there is external disturbance, the disturbance is estimated and obtained by designing disturbance observer. The disturbance feedback is compensated into the control input channel to weaken or even eliminate the influence of the disturbance on the system.
Integrating the quasi-min-max MPC controller and the disturbance observer, the actual control quantity of the OWMR is u Ã = u Àd u + d u , withd u = J(b i )d, i = 1, 2, 3, d u = J(b i )d, i = 1, 2, 3. It can be seen that if the designed disturbance observer can converge asymptotically, the influence of the actual disturbance can be compensated by the disturbance observer, so as to ensure the asymptotic convergence and feasibility of the global control algorithm.

Design of controller based on quasi-min-max MPC
Consider the visual servoing error system (15) and let g(u) = J 21 (b)u. Then the system (15) can be rewritten as: where the matrix Notice that g(u) is a static nonlinear function. Hence, the system (17) is a multi-variable time-varying Hammerstein system which typically consists of a static nonlinear input followed by a linear dynamic model. 22 Let u = g(u) and we have the linear time-varying dynamic model of (17) It is further noted that B(e) is linear on the states e 1 and e 2 . Then due to the constraints on those states, it is obtained from Baranyi 23 that B(e) can be covered by the convex polytope O, that is, where C 0 denotes convex hull and the matrices  dynamic model (18) can be modeled as a polyhedral LPV system e(k + 1) = e(k) + with the unknown parameters l i 5 0 satisfy that l 1 + l 2 + l 3 + l 4 = 1.
Consider the visual servoing system (20) and define the infinite horizon cost function where (i|k) is the value at k + i predicted at the current time k and weighted matrices Q . 0 and R . 0. In order to minimize the function (21), we consider the structure of (18) and use the system (20) to be the predicted model for (21). Notice that at each current time k, the matrix B(e(k)) can be determined via the measure of e(k) whilst B(e(k + i)) is not known but is covered by the convex polytope (19). Here the quasi-min-max MPC 24-26 is adopted to achieve the control goal. Online solve the optimization problem to obtain the solution u * (k). Then from the system (17) or (18), it is known that the corresponding optimal control u(k) = u mpc (k) is uniquely determined by which yields the closed-loop visual servoing system e(k + 1) = e(k) + B(e(k))g(u mpc (k)), k = 0, 1, Á Á Á ð23Þ

Design of disturbance observer
Considering that it is very difficult to establish an accurate interference model or accurately measure the external interference through the sensor, a disturbance observer is used to estimate the disturbance. According to the OWMR system model (4), the disturbance observer is designed as follows: where z is the internal state of the nonlinear disturbance observer, p(x) is the function to be designed, L(x) is the gain of the observer, and Theorem 1: If l(x) in disturbance observer (24) can satisfy that À l(x)g 2 (x) is Hurwitz, then disturbance observer (24) can have tracking disturbance d with error, and the maximum tracking error e d k k ' is proportional to d; For a constant bounded disturbance, the disturbance observer (24) can track disturbance d asymptotically.
Proof: Define estimation error whered is the estimated error. It is assumed that the disturbance changes slowly compared with the dynamic characteristics of the disturbance observer, therefore we can assume _ d \ d, d is a given constant.
When the disturbance d is a slow-varying disturbance, the dynamic estimation error can be written as Differential equation solution of (27) can be obtained: According to norm property, the following inequality can be obtained by (28) If l(x) satisfies that À l(x)g 2 (x) is Hurwitz, then there exists K51, a \ 0 such that Therefore, combining (29) with (30), it can be obtained Therefore, by analyzing (31), it can be known that when t ! ' and a \ 0, d, K is a constant greater than 0, then that is, e d (t) k kis bounded. When the disturbance d is a constant bounded disturbance, the dynamic (27) of the estimation error can be written as Since l(x) satisfies that À l(x)g 2 (x) is Hurwitz, the disturbance observer (24) can track disturbance d asymptotically.
Theorem 2: Select Through l(x), the linear function p(x) to be designed can be obtained according to (25). Then the disturbance observer (24) obtained is asymptotically stable.  ½ T are added to the robot system, the Figure 6 shows the error of OWMR state simulation  diagram, Figures 7 and 8 shows rotation velocity and deflection velocity of wheels of OWMR. Figure 9 shows the trajectory of the OWMR in the world coordinate system Xw-Yw plane, and Figure 10 shows the trajectory of the feature points in the camera plane during the process of motion.
As shown in Figure 6, the red solid line represents DOBC, and the dotted line represents quasi-min-max MPC. When the disturbance observer is not added, the dotted line error e 1 has static error, so the omnidirectional mobile robot cannot reach the desired target point under large external interference and cannot complete the corresponding visual servoing task. When the disturbance observer is combined with the disturbance observer, the disturbance observer can compensate the external disturbance well, and then eliminate the influence of the external disturbance. The mobile robot can reach the target point, and complete the visual servoing task better finally.
As shown in Figures 7 and 8, the red solid line represents DOBC, and the dotted line represents quasi-minmax MPC. When no disturbance observer is added, the rotation velocity and deflection velocity of the wheels     of OWMR are greatly affected by external disturbance, and the fluctuation changes are obvious. And the velocity of rotation is not stabilized to zero. In the control strategy based on disturbance observer, the rotation velocity and deflection velocity of the robot are less affected by external disturbance, and the final velocity tends to zero, that is, the OWMR reaches the target position.
It can be seen from Figures 9 and 10 that under the control strategy presented in this paper, the OWMR in the world coordinate plane and the feature points in the camera coordinate plane finally reach the desired positions. Figure 9 shows the trajectory of the robot from the initial posture to the desired posture, where the red triangle denotes the desired posture. The feasibility of the designed control strategy is proved. According to Figure 10, the feature points are always located within the visual field of the camera, which ensures the visual visibility of the mobile robot.

Conclusions
This study has proposed a disturbance observer based visual servoing stabilization predictive control procedure. Firstly, it studies the visual servoing error system to model of the omnidirectional mobile robot dynamics with the rotation velocity and deflection velocity of the eccentric direction wheel. On this basis, the asymptotic stability of the closed-loop visual servoing system without disturbance is guaranteed by combining the quasimin-max MPC strategy. A nonlinear disturbance observer is designed to compensate the impact of external disturbances on the system. Simulation and comparison experiments verify the feasibility and effectiveness of the proposed strategy. The procedure has been tested by simulation and real experiments. In addition, new solutions to problems such as camera occlusion should be explored and evaluated in practical applications. For such control system design, a potential approach worthwhile being investigated is the model free sliding mode control. 27

Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.