Learning robot actions using a simulator has many advantages as compared to one using a real robot. However, transferring the policy learned in simulation to the real robot is difficult because of the influence of the “reality gap”. In particular, the visual reality gap is a severe problem for the End-to-End controller, which uses images as a state. In this paper, we propose a real-to-sim image transfer combining domain randomization with latent dynamics. Our proposed method can predict future real-to-sim images, even if we could not obtain images. We validate the effectiveness of the proposed method by using real images in a manipulation task.