We construct a speaker authentication system, where 3-layerd neural networks with ensemble learning algorithm are used, and investigate the effect of ensemble learning on the performance of the system. From the results, we found that the authentication rates of the system for a person became to 100% by using ensemble learning. Therefore, the new ensemble leaning used in this study is thought to be useful for the speaker authentication system with layered neural networks. In addition, a new multi-step authentication system for many persons by extending the system for a person was suggested. In the system, the ensemble learning was also useful for the speaker authentication system of neural networks for many persons.