Ceria doped with Sm and Gd (SDC and GDC) has been suggested as a promising candidate for the electrolyte used in solid oxide fuel cells (SOFCs), since it has relatively high oxygen ion conductivity at intermediate temperature. There have been many previous experimental and computational studies to investigate the properties, structure, and effect of vacancies, etc. for SDC and GDC. However, in these previous studies, it is commonly assumed that the interaction between oxygen vacancies is negligible and many focus only on the mono-vacancy system. In addition, the possibility of anisotropic vibrational motion of the oxygen ions around vacancies is often neglected. In this paper, using both first-principle density-functional theory and classical molecular dynamics calculations, we investigate the structural and vibrational properties of the optimized SDC and GDC structures, such as bonding analysis, phonon density-of-state and mean-square-displacement of the oxygen ions. Also, we report the direction-dependent vibrations at the specific frequency of the oxygen ions near the vacancies, activation energies, and diffusion coefficients of SDC and GDC which can extend our understanding of diffusion dynamics in doped ceria-based electrolytes for SOFC applications.