The aspheric surface testing system based on the shearing interference principle is used to better analyze how the installation and fabrication errors of linear polarizer affects the precision of the aspheric surface. The effects of the angle error of the linear polarizer and the phase-shifting error of the wave plate on the reconstruction of the aspheric surface are studied. Constructing the model of the Jones matrix with the error term system, the reconstruction surface and residual after the corresponding error system are obtained using the four-step phase-shifting and differential Zernike method. The polarizer array of an angle of θ was varied from ( θ − 1 deg ) to ( θ + 2 deg ) every 0.2 deg, and the changes of the fitting surface and the residual surface were simulated. The phase-shifting error of the simulated one-fourth wave plate was <λ / 300. Then, the change of the fitting surface and residual surface when these two errors exist at the same time were simulated. The simulation results show that the influence of the phase-shifting error of the wave plate on the fitting surface is much greater than the angle error of the polarizer array. When assembling the interference system, the phase-shifting error of the wave plate should be <λ / 300 when the light transmission angle error of the polarizer array is between −0.2 deg and 0.2 deg, which will ensure the testing precision of the aspheric surface testing system.