This paper investigates the charging control strategy design problem of a large-scale plug-in electric vehicle (PEV) group, where each PEV aims to find an optimal charging strategy to minimize its own cost function. It should be noted that the collective behavior of the group is coupled in the individual cost function, which complicates the design of decentralized charging strategies. To obtain the decentralized charging strategy, a mean-field game (MFG) formulation is proposed where a penalty on collective consensus is embedded and a class of mean-field coupled time-varying stochastic systems is targeted for solving the MFG which involves the charging model of PEVs as a special case. Then, an augmented system with dimension extension and the policy iteration algorithm are proposed to solve the mean-field game problem for the class of mean-field coupled time-varying stochastic systems. Moreover, analysis of the convergence of proposed approach has been studied. Last, simulation is conducted to illustrate the effectiveness of the proposed MFG-based charging control strategy and shows that the charging control strategy can achieve desired mean-field state and impact to the power grid can be buffered.