This thesis focuses on the research of human hand gesture detection and recognition for the indoor surveillance system, which is capable of interacting with a mobile robot. Through recognizing human hand gesture, the developed system can direct the robot to carry out various actions. In the strategy of hand gesture detection, the original image is transformed to YCbCr color space and the subspace of human skin color in the face and hands can then be segmented. By means of the posture feature of human body, the fuzzy segment model is designed to identify the hand location. The procedure of human gesture recognition includes finding the contour of hand image. Thereafter, the numbers of fingertip can be determined by using the curvature quantity from geometrical features of hand contour. The recognition result can be encoded as various instructions and transmitted to the mobile robot through TCP/IP wireless network. Thus, it can offer an interactive manner between the user and the mobile robot. The experimental results show that the developed system can be used in various computer vision-based applications that require real-time detection and recognition of hand gesture.