Gesture recognition is an image sensing technology that allows people to operate devices with natural movements. Gesture recognition applications include patient monitoring, surveillance, robotics, sign language recognition, and more. However, there are many places where gesture recognition using normal cameras cannot be used from a privacy consideration. For example, personal spaces such as toilets and bathrooms, public spaces, and more. We have proposed a method of capturing shadow pictures using single-pixel-imaging to realize privacy-conscious gesture recognition. Single-pixel-imaging is a method of image reconstruction using random mask patterns and a single point detector. As an implementation method of single-pixel imaging in public spaces, we have studied using a high-frame-rate LED display as a light source. By using a high-frame-rate LED display, random patterns can be latent while the observer perceives an apparent image. However, the image reconstructed by single-pixel-imaging using a high-frame-rate LED display is influenced by the apparent image, making gesture recognition difficult. In this study, we show that the influence of the apparent image can be removed by restoring the restored image using deep learning with a convolutional network called U-Net.