Approximate Message Passing (AMP) is a class of iterative algorithms that have found applications in many problems in high-dimensional statistics and machine learning. In its general form, AMP can be formulated as an iterative procedure driven by a matrix M. Theoretical analyses of AMP typically assume strong distributional properties on M, such as M has i.i.d. sub-Gaussian entries or is drawn from a rotational invariant ensemble. However, numerical experiments suggest that the behavior of AMP is universal as long as the eigenvectors of M are generic. In this paper we take the first step in rigorously understanding this universality phenomenon. In particular, we investigate a class of memory-free AMP algorithms (proposed by Çakmak and Opper for mean-field Ising spin glasses) and show that their asymptotic dynamics is universal on a broad class of semirandom matrices. In addition to having the standard rotational invariant ensemble as a special case, the class of semirandom matrices that we define in this work also includes matrices constructed with very limited randomness. One such example is a randomly signed version of the sine model, introduced by Marinari, Parisi, Potters, and Ritort for spin glasses with fully deterministic couplings.