We study the quantitative stability of the quadratic optimal transport map between a fixed probability density ρ and a probability measure μ on ${\mathbb{R}}^d$, which we denote $T_{\mathrm{\mu }}$. Assuming that the source density ρ is bounded from above and below on a compact convex set, we prove that the map $\mathrm{\mu }\mapsto T_{\mathrm{\mu }}$ is bi-Hölder continuous on large families of probability measures, such as the set of probability measures whose moment of order $p>d$ is bounded by some constant. These stability estimates show that the linearized optimal transport metric ${\mathrm{W}}_{2,\mathit{\rho }}(\mathrm{\mu },\mathrm{\nu })=\Vert T_{\mathrm{\mu }}-T_{\mathrm{\nu }}{\Vert }_{{\mathrm{L}}^2(\mathit{\rho },{\mathbb{R}}^d)}$ is bi-Hölder equivalent to the 2-Wasserstein distance on such sets, justifying its use in applications.