The first magnetization loop and the first stress–strain cycle of magnetoactive elastomers (MAEs) in a magnetic field differ considerably from the following loops and cycles, possibly due to the internal restructuring of the magnetic filler particles and the matrix polymer chains. In the present study, the irreversible magnetization processes during the first magnetization of MAEs with different filler compositions and tensile moduli of the matrix are studied by first-order reversal curve (FORC) measurements. For MAEs with mixed magnetic NdFeB/Fe fillers the FORC distributions and magnetization distributions of the first major loop reveal a complex irreversible magnetization behavior at interaction fields H_u < −50 kA m⁻¹ due to the magnetostatic coupling between the magnetically hard NdFeB and the magnetically soft Fe particles. This coupling is enhanced either if the interparticle distance is reduced by particle motion and restructuring or by an increase in the particle densities. If the stiffness of the matrix is increased, the structuring and thus the interparticle interactions are suppressed and the magnetization reversal is dominated by domain processes in the NdFeB particles at high coercive fields of H_c > 600 kA m⁻¹.