This paper describes a method of recognizing “Ooba” (green perilla leaves) using deep neural network. We are developing a robot for ooba harvesting support robot. In order to manipulate ooba exactly, the robot needs to recognize ooba’s parameters such as width, height and direction using images. It is difficult to adjust parameters because the leaf colors change depending on the weather and the growth policy of each farmer. To reduce the time for adjusting parameters, we adopt deep neural network (DNN). We train DNN for recognizing ooba region in the image. We use image processing techniques to annotate labels efficiently. We also compare the performance of the method for sevral numbers of layers of the DNN.