In this paper, we propose a method of emotion monitoring using physiological signals such as RR intervals and blood volumes obtained by analyzing hemoglobin concentrations from facial color images. Emotion monitoring has great potential in areas such as market research, safety, medical and robot systems. The most popular method of emotion monitoring uses physiological signals. However, it is difficult to apply the commonly used methods in practice because special instruments such as electrodes or laser speckle flowgraphy are required to obtain physiological signals. The proposed method uses a simple RGB camera. Using 27 features calculated from the physiological signals obtained from facial RGB images, we classified five emotional states, amusement, anger, disgust, sadness and surprise, with 94% accuracy.