This reseach is intended to control the mobile robot by the visual sensor. We consider an estimation algorithm to obtain the position and moution of the camera and the moving object using Kalman filter to process the image data. Applying the estimation algorithm, a design of control system for the mobile robot is presented. Numerical and experimented researctes show the validity of the visual sensor.