Nowadays, robots have been utilized to perform a wide variety of tasks instead of humans, in which those equipped with hand-eye cameras have been proved to be useful in factories for handling metal parts. However, there have been difficulties for robots with vision sensors like cameras to detect a target objects’ 3D pose accurately, especially in the case that the target object is unique and the shape is arbitrary, and further, the object is moving. Visual servoing to moving target may enable the robot to pursue an animal to catch it. Aiming at achieving identification ability of a unique object, this paper utilized our previous study concerning photo-model-based object recognition. Based on the study, this paper applies the photo-model-based pose estimation method to video-rate stereo-vision position and orientation (pose) tracking, and it has been applied to an arbitrary-shaped moving target. To confirm the abilities of the proposed stereo-vision-based method to pursue an arbitrary target, real-time 3D pose estimation experiment to track the relative pose between a moving target and hand-eye robot have been conducted. The real-time experimental results show the proposed system can recognize the designated arbitrary-shaped target and track its pose in real-time.