Janus ellipsoids as mesoscale building blocks can aggregate into various micelle-like structures in solution that have potential applications in many fields such as novel surfactants, photonic crystals, drug delivery and biochemical sensors. In this work, we present a novel nonspherical-particle model to investigate the self-assembly of Janus ellipsoids, which quantitatively reflects interaction dependence on the particle shape. The phase diagrams of Janus ellipsoids depending on the aspect ratio and the component ratio are achieved and various aggregates are observed such as a sandwich-type structure, columnar aggregates, vesicles, liquid crystals, random aggregation structures, spherical micelles and wormlike micelles. The specific heat capacity curves and temperature evolutions illustrate the formation processes of assembled superstructures detailedly. We analyze the potential energy surfaces (PESs) of interaction between two Janus ellipsoids and the minimum energy paths (MEPs) between saddle points on the PESs. It is found that the number of metastable conformation and the activation energy along MEPs rely not only on the ellipsoidal shape but also on the component ratio. This work provides rich and valuable information for a deep understanding of the self-assembly mechanism of Janus ellipsoids and design of new mesoscale building blocks.