In recent years, new applications like augmented reality, driverless cars, intelligent environmental surveillance, human action analysis, and in-home robotics are all becoming possible due to the advancement of computer vision (CV) developments, and object recognition plays an essential role in all those tasks. Objects are basic meaningful units of surrounding environments, and good performance of those CV applications can be achieved if all related objects can be successfully tracked, detected, segmented, recognized and analyzed. However, there are still limitations in current 2D CV methods. Color data combined with depth data are then considered to improve performances and solve problems encountered by traditional algorithms. Depth sensors are now cheap and readily available, and they bring new opportunities and challenges to many CV areas. In this thesis, a thorough system is designed to solve essential problems of object recognition with RGB and depth data (RGB-D data). The system aims to assist those emerging CV applications to help people live more conveniently and have more fun. In this thesis, essential algorithms of object recognition are developed and integrated to provide a thorough solution for problems encountered by previous works. First, 3D structure analysis is developed to segment the input RGB-D data into basic 3D structure elements. The indoor scene is parsed into planes and physically meaning clusters with depth and surface normals. Further analysis and processing are the performed on those clusters as object candidates. De-noising and accurate object segmentation algorithm are then proposed. Depth data is useful in segmenting raw clusters, but it is often noisy and broken at edges of objects. Color images and depth data are then combined to accurately segment out objects due to the higher quality of passive color images. Detailed boundaries are preserved by color superpixels, and 3D structure is then used to build foreground and background color models. Using superpixels and color models, accurate object visual segmentation is achieved automatically without any user input. 3D object tracking is also developed. The targeted object can be tracked in real time with huge variations in size, translation, rotation, illumination, and appearance. Using RGB-D Data, the high performance can be achieved, and features like positions, global color and normal models, and local 3D descriptors are processed for tracking. Compared to previous 2D tracking, no sliding window or particle filtering is needed since 3D structure elements are available. Pixel-wise accurate video segmentation can also be achieved with the proposed segmentation method. Finally, a novel on-line learning method is proposed to train robust object detectors. By using the proposed object tracking, training data with labels can be automatically generated, and efficient on-line learning and detection methods are used for real-time performance. Combining object detectors with tracking, object recognition and tracking recovery can be achieved. In previous 2D CV learning tasks, training datasets often suffer from lack of variability, cluttered background, and inability in automatically locating and segmenting targets of interest. The proposed on-line learning is provided for those problems. Totally speaking, a highly integrated algorithm for object recognition is designed, and RGB-D data is used and studied for object segmentation, tracking, on-line learning, and detection.