Cutting forces are often used to optimize cutting parameters and analyze cutting characteristics of the tool. Accurate prediction of cutting forces is critical to understand deeply of the cutting process, especially for helical milling of the difficult-to-cut materials. Generally, the cutting force coefficients are not affected by the cutting parameters in the end milling process, but due to the special cutting mechanism of helical milling, the change of the cutting force coefficients must be carefully studied. However, the identification of cutting force coefficients is very important to simulate cutting forces accurately, and the relationship between cutting force coefficient and cutting parameters needs to be carefully considered. To analyze the change of cutting forces, this paper respectively proposes linear and nonlinear cutting force models applied for the helical milling process, then according to the helical milling experiments, cutting coefficients are identified using an inverse method through instantaneous force in the linear model, at the same time, the coefficients are identified by similar average force method in the nonlinear model, simulated three-direction cutting forces under two different situations are compared to the experimental results. The relationship between cutting force coefficients and cutting parameters is analyzed in detail, the linear and nonlinear models as well as the fitting relations of different cutting force coefficients are compared. The aim of this paper is to study the influence of cutting parameters on the cutting coefficients, thus create a model to predict reliably the cutting force with different cutting parameters.