Limitations of principal components in quantitative genetic association models for human studies

Principal Component Analysis (PCA) and the Linear Mixed-effects Model (LMM), sometimes in combination, are the most common genetic association models. Previous PCA-LMM comparisons give mixed results, unclear guidance, and have several limitations, including not varying the number of principal components (PCs), simulating simple population structures, and inconsistent use of real data and power evaluations. We evaluate PCA and LMM both varying number of PCs in realistic genotype and complex trait simulations including admixed families, subpopulation trees, and real multiethnic human datasets with simulated traits. We find that LMM without PCs usually performs best, with the largest effects in family simulations and real human datasets and traits without environment effects. Poor PCA performance on human datasets is driven by large numbers of distant relatives more than the smaller number of closer relatives. While PCA was known to fail on family data, we report strong effects of family relatedness in genetically diverse human datasets, not avoided by pruning close relatives. Environment effects driven by geography and ethnicity are better modeled with LMM including those labels instead of PCs. This work better characterizes the severe limitations of PCA compared to LMM in modeling the complex relatedness structures of multiethnic human data for association studies.


Introduction
The goal of a genetic association study is to identify loci whose genotype variation is significantly correlated to given trait. Naive association tests assume that genotypes are drawn independently from a common allele frequency. This assumption does not hold for structured populations, which includes multiethnic cohorts and admixed individuals (ancient relatedness), and for family data (recent relatedness; Astle and Balding, 2009). Association studies of admixed and multiethnic cohorts, the focus of this work, are becoming more common, are believed to be more powerful, and are necessary to bring more equity to genetic medicine (Rosenberg et al., 2010;Hoffman and Dubé, 2013;Coram et al., 2013;Medina-Gomez et al., 2015;Conomos et al., 2016a;Hodonsky et al., 2017;Martin et al., 2017a;Martin et al., 2017b;Hindorff et al., 2018;Hoffmann et al., 2018;Mogil et al., 2018;Roselli et al., 2018;Wojcik et al., 2019;Peterson et al., 2019;Zhong et al., 2019;  inflation factors Devlin and Roeder, 1999 or QQ plots), a large fraction (6/17) did not measure power (or proxies such as ROC curves), and only four used more than one number of PCs for PCA. Lastly, no consensus has emerged as to why LMM might outperform PCA or vice versa (Price et al., 2010;Sul and Eskin, 2013;Price et al., 2013;Hoffman and Dubé, 2013), or which features of the real datasets are critical for the LMM advantage other than family relatedness, resulting in unclear guidance for using PCA. Hence, our work includes real and simulated genotypes with higher model dimensions and F ST matching that of multiethnic human cohorts (Ochoa and Storey, 2021;Ochoa and Storey, 2019), we vary the number of PCs, and measure robust proxies for type I error control and calibrated power.
In this work, we evaluate the PCA and LMM association models under various numbers of PCs, which are included in LMMs too. We use genotype simulations (admixture, family, and subpopulation tree models) and three real datasets: the 1000 Genomes Project (Abecasis et al., 2010;Abecasis et al., 2012), the Human Genome Diversity Panel (HGDP) Rosenberg et al., 2002;Bergström et al., 2020), and Human Origins (Patterson et al., 2012;Lazaridis et al., 2014;Lazaridis et al., 2016;Skoglund et al., 2016). We simulate quantitative traits from two models: fixed effect sizes (FES) construct coefficients inverse to allele frequency, which matches real data (Park et al., 2011;Zeng et al., 2018;O'Connor et al., 2019) and corresponds to high pleiotropy and strong balancing selection (Simons et al., 2018) and strong negative selection (Zeng et al., 2018;O'Connor et al., 2019), which are appropriate assumptions for diseases; and random coefficients (RC), which are drawn independent of allele frequency, and corresponds to neutral traits (Zeng et al., 2018;Simons et al., 2018). LMM without PCs consistently performs best in simulations without environment, and greatly outperforms PCA in the family simulation and in all real datasets. The tree simulations, which model subpopulations with the tree but exclude family structure, do not recapitulate the real data results, suggesting that family relatedness in real data is the reason for poor PCA performance. Lastly, removing up to 4th degree relatives in the real datasets recapitulates poor PCA performance, showing that the more numerous distant relatives explain the result, and suggesting that PCA is generally not an appropriate model for real data. We find that both LMM and PCA are able to model environment effects correlated with genetics, and LMM with PCs gains a small advantage in this setting only, but direct modeling of environment performs much better. All together, we find that LMMs without PCs are generally a preferable association model, and present novel simulation and evaluation approaches to measure the performance of these and other genetic association approaches.

Ind.(n)
*For admixed family, ignores additional model dimension of 20 generation pedigree structure. For real datasets, lower range is continental subpopulations, upper range is number of fine-grained subpopulations. † m 1 = round(nh 2 /8) to balance power across datasets, shown for h 2 = 0.8 only. ‡ Model parameter for simulations, estimated value on real datasets.

Overview of evaluations
We use three real genotype datasets and simulated genotypes from six population structure scenarios to cover various features of interest ( Table 2). We introduce them in sets of three, as they appear in the rest of our results. Population kinship matrices, which combine population and family relatedness, are estimated without bias using popkin (Ochoa and Storey, 2021; Figure 1). The first set of three simulated genotypes are based on an admixture model with 10 ancestries ( Figure 1A; Ochoa and Storey, 2021;Gopalan et al., 2016;Cabreros and Storey, 2019). The 'large' version (1000 individuals) illustrates asymptotic performance, while the 'small' simulation (100 individuals) illustrates model overfitting. The 'family' simulation has admixed founders and draws a 20-generation random pedigree with assortative mating, resulting in a complex joint family and ancestry structure in the last generation ( Figure 1B). The second set of three are the real human datasets representing global human diversity: Human Origins ( Figure 1D), HGDP ( Figure 1G), and 1000 Genomes ( Figure 1J), which are enriched for small minor allele frequencies even after MAF <1% filter ( Figure 1C). Last are subpopulation tree simulations ( Figure 1F, I, L) fit to the kinship ( Figure 1E, H and K) and MAF ( Figure 1C) of each real human dataset, which by design do not have family structure. All traits in this work are simulated. We repeated all evaluations on two additive quantitative trait models, fixed effect sizes (FES) and random coefficients (RC), which differ in how causal coefficients are constructed. The FES model captures the rough inverse relationship between coefficient and minor allele frequency that arises under strong negative and balancing selection and has been observed in numerous diseases and other traits (Park et al., 2011;Zeng et al., 2018;Simons et al., 2018;O'Connor et al., 2019), so it is the focus of our results. The RC model draws coefficients independent of allele frequency, corresponding to neutral traits (Zeng et al., 2018;Simons et al., 2018), which results in a wider effect size distribution that reduces association power and effective polygenicity compared to FES.
We evaluate using two complementary measures: (1) SRMSDp (p-value signed root mean square deviation) measures p-value calibration (closer to zero is better), and (2) AUC PR (precision-recall area under the curve) measures causal locus classification performance (higher is better; Figure 2). SRMSDp is a more robust alternative to the common inflation factor λ and type I error control measures; there is a correspondence between λ and SRMSDp , with SRMSDp > 0.01 giving λ > 1.06 (Figure 2-figure supplement 1) and thus evidence of miscalibration close to the rule of thumb of λ > 1.05 (Price et al., 2010). There is also a monotonic correspondence between SRMSDp and type I error rate ( Figure 2-figure supplement 2). AUC PR has been used to evaluate association models (Rakitsch et al., 2013), and reflects calibrated statistical power (Figure 2-figure supplement 3) while being robust to miscalibrated models (Appendix 2).
Both PCA and LMM are evaluated in each replicate dataset including a number of PCs r between 0 and 90 as fixed covariates. In terms of p-value calibration, for PCA the best number of PCs r (minimizing mean |SRMSDp| over replicates) is typically large across all datasets (Table 3), although much smaller r values often performed as well (shown in following sections). Most cases have a mean |SRMSDp| < 0.01 , whose p-values are effectively calibrated. However, PCA is often miscalibrated on the family simulation and real datasets ( Table 3). In contrast, for LMM, r = 0 (no PCs) is always best, and is always calibrated. Comparing LMM with r = 0 to PCA with its best r , LMM always has significantly smaller |SRMSDp| than PCA or is statistically tied. For AUC PR and PCA, the best r is always smaller than the best r for |SRMSDp| , so there is often a tradeoff between calibrated p-values versus classification performance. For LMM, there is no tradeoff, as r = 0 often has the best mean AUC PR , and otherwise is not significantly different from the best r . Lastly, LMM with r = 0 always has significantly greater or statistically tied AUC PR than PCA with its best r .

Evaluations in admixture simulations
Now we look more closely at results per dataset. The complete SRMSDp and AUC PR distributions for the admixture simulations and FES traits are in Figure 3. RC traits gave qualitatively similar results ( Figure 3-figure supplement 1).
In the large admixture simulation, the SRMSDp of PCA is largest when r = 0 (no PCs) and decreases rapidly to near zero at r = 3 , where it stays for up to r = 90 ( Figure 3A). Thus, PCA has calibrated p-values for r ≥ 3 , smaller than the theoretical optimum for this simulation of r = K − 1 = 9 . In contrast,  the SRMSDp for LMM starts near zero for r = 0 , but becomes negative as r increases (p-values are conservative). The AUC PR distribution of PCA is similarly worst at r = 0 , increases rapidly and peaks at r = 3 , then decreases slowly for r > 3 , while the AUC PR distribution for LMM starts near its maximum at r = 0 and decreases with r . Although the AUC PR distributions for LMM and PCA overlap considerably at each r , LMM with r = 0 has significantly greater AUC PR values than PCA with r = 3 (Table 3). However, qualitatively PCA performs nearly as well as LMM in this simulation.
The observed robustness to large r led us to consider smaller sample sizes. A model with large numbers of parameters r should overfit more as r approaches the sample size n . Rather than increase r beyond 90, we reduce individuals to n = 100 , which is small for typical association studies but may occur in studies of rare diseases, pilot studies, or other constraints. To compensate for the loss of power        Table 3 continued on next page due to reducing n , we also reduce the number of causal loci (see Trait Simulation), which increases perlocus effect sizes. We found a large decrease in performance for both models as r increases, and best performance for r = 1 for PCA and r = 0 for LMM ( Figure 3B). Remarkably, LMM attains much larger negative SRMSDp values than in our other evaluations. LMM with r = 0 is significantly better than PCA ( r = 1 to 4) in both measures ( Table 3), but qualitatively the difference is negligible.
The family simulation adds a 20-generation random family to our large admixture simulation. Only the last generation is studied for association, which contains numerous siblings, first cousins, etc., with the initial admixture structure preserved by geographically biased mating. Our evaluation reveals a sizable gap in both measures between LMM and PCA across all r ( Figure 3C). LMM again performs best with r = 0 and achieves mean |SRMSDp| < 0.01 . However, PCA does not achieve mean |SRMSDp| < 0.01 at any r , and its best mean AUC PR is considerably worse than that of LMM. Thus, LMM is conclusively superior to PCA, and the only calibrated model, when there is family structure.

Evaluations in real human genotype datasets
Next, we repeat our evaluations with real human genotype data, which differs from our simulations in allele frequency distributions and more complex population structures with greater F ST , numerous correlated subpopulations, and potential cryptic family relatedness.
Human Origins has the greatest number and diversity of subpopulations. The SRMSDp and AUC PR distributions in this dataset and FES traits ( Figure 4A) most resemble those from the family simulation ( Figure 3C). In particular, while LMM with r = 0 performed optimally (both measures) and satisfies mean |SRMSDp| < 0.01 , PCA maintained SRMSDp > 0.01 for all r and its AUC PR were all considerably smaller than the best AUC PR of LMM.
HGDP has the fewest individuals among real datasets, but compared to Human Origins contains more loci and low-frequency variants. Performance ( Figure 4B) again most resembled the family simulations. In particular, LMM with r = 0 achieves mean |SRMSDp| < 0.01 (p-values are calibrated), while PCA does not, and there is a sizable AUC PR gap between LMM and PCA. Maximum AUC PR values were lowest in HGDP compared to the two other real datasets.
1000 Genomes has the fewest subpopulations but largest number of individuals per subpopulation. Thus, although this dataset has the simplest subpopulation structure among the real datasets, we find SRMSDp and AUC PR distributions ( Figure 4C) that again most resemble our earlier family simulation, with mean |SRMSDp| < 0.01 for LMM only and large AUC PR gaps between LMM and PCA.
Our results are qualitatively different for RC traits, which had smaller AUC PR gaps between LMM and PCA ( Figure 4-figure supplement 1). Maximum AUC PR were smaller in RC compared to FES in Human Origins and 1000 Genomes, suggesting lower power for RC traits across association models. Nevertheless, LMM with r = 0 was significantly better than PCA for all measures in the real datasets and RC traits ( Table 3).

Evaluations in subpopulation tree simulations fit to human data
To better understand which features of the real datasets lead to the large differences in performance between LMM and PCA, we carried out subpopulation tree simulations. Human subpopulations are related roughly by trees, which induce the strongest correlations, so we fit trees to each real dataset LMM r = 0 vs best r PCA vs LMM r = 0     and tested if data simulated from these complex tree structures could recapitulate our previous results ( Figure 1). These tree simulations also feature non-uniform ancestral allele frequency distributions, which recapitulated some of the skew for smaller minor allele frequencies of the real datasets ( Figure 1C). The SRMSDp and AUC PR distributions for these tree simulations ( Figure 5) resembled our admixture simulation more than either the family simulation ( Figure 3) or real data results ( Figure 4). Both LMM with r = 0 and PCA (various r ) achieve mean |SRMSDp| < 0.01 ( Table 3). The AUC PR distributions of both LMM and PCA track closely as r is varied, although there is a small gap resulting in LMM ( r = 0 ) besting PCA in all three simulations. The results are qualitatively similar for RC traits ( Figure 5figure supplement 1, Table 3). Overall, these subpopulation tree simulations do not recapitulate the large LMM advantage over PCA observed on the real data.

Numerous distant relatives explain poor PCA performance in real data
In principle, PCA performance should be determined by the dimension of relatedness, or kinship matrix rank, since PCA is a low-dimensional model whereas LMM can model high-dimensional relatedness without overfitting. We used the Tracy-Widom test  with p < 0.01 to estimate kinship matrix rank as the number of significant PCs ( Figure 6-figure supplement 1A).
The true rank of our simulations is slightly underestimated ( Table 2), but we confirm that the family simulation has the greatest rank, and real datasets have greater estimates than their respective subpopulation tree simulations, which confirms our hypothesis to some extent. However, estimated ranks do not separate real datasets from tree simulations, as required to predict the observed PCA performance. Moreover, the HGDP and 1000 Genomes rank estimates are 45 and 61, respectively, yet PCA performed poorly for all r ≤ 90 numbers of PCs ( Figure 4). The top eigenvalue explained a proportion of variance proportional to F ST ( Table 2), but the rest of the top 10 eigenvalues show no clear differences between datasets, except the small simulation had larger variances explained per eigenvalue (expected since it has fewer eigenvalues; Figure 6-figure supplement 1). Comparing cumulative variance explained versus rank fraction across all eigenvalues, all datasets increase from their starting point almost linearly until they reach 1, except the family simulation has much greater variance explained by mid-rank eigenvalues ( Figure 6-figure supplement 1). We also calculated the number of PCs that are significantly associated with the trait, and observed similar results, namely that while the family simulation has more significant PCs than the non-family admixture simulations, the real datasets and their tree simulated counterparts have similar numbers of significant PCs ( Figure 6figure supplement 2). Overall, there is no separation between real datasets (where PCA performed poorly) and subpopulation tree simulations (where PCA performed relatively well) in terms of their eigenvalues or kinship matrix rank estimates. Local kinship, which is recent relatedness due to family structure excluding population structure, is the presumed cause of the LMM to PCA performance gap observed in real datasets but not their subpopulation tree simulation counterparts. Instead of inferring local kinship through increased kinship matrix rank, as attempted in the last paragraph, now we measure it directly using the KINGrobust estimator (Manichaikul et al., 2010). We observe more large local kinship in the real datasets and the family simulation compared to the other simulations ( Figure 6). However, for real data this distribution depends on the subpopulation structure, since locally related pairs are most likely in the same subpopulation. Therefore, the only comparable curve to each real dataset is their corresponding subpopulation tree simulation, which matches subpopulation structure. In all real datasets, we identified highly related individual pairs with kinship above the 4th degree relative threshold of 0.022 (Manichaikul et al., 2010;Conomos et al., 2016b). However, these highly related pairs are vastly outnumbered by more distant pairs with evident non-zero local kinship as compared to the extreme tree simulation values.
To try to improve PCA performance, we followed the standard practice of removing 4th degree relatives, which reduced sample sizes between 5% and 10% ( Table 4). Only r = 0 for LMM and r = 20            for PCA were tested, as these performed well in our earlier evaluation, and only FES traits were tested because they previously displayed the large PCA-LMM performance gap. LMM significantly outperforms PCA in all these cases (Wilcoxon paired 1-tailed p < 0.01 ; Figure 7). Notably, PCA still had miscalibrated p-values two of the three real datasets ( |SRMSDp| > 0.01 ), the only marginally calibrated case being HGDP which is also the smallest of these datasets. Otherwise, AUC PR and SRMSDp ranges were similar here as in our earlier evaluation. Therefore, the removal of the small number of highly related individual pairs had a negligible effect in PCA performance, so the larger number of more distantly related pairs explain the poor PCA performance in the real datasets.

Low heritability and environment simulations
Our main evaluations were repeated with traits simulated under a lower heritability value of h 2 = 0.3 . We reduced the number of causal loci in response to this change in heritability, to result in equal average effect size per locus compared to the previous high heritability evaluations (see Trait Simulation). Despite that, these low heritability evaluations measured lower AUC PR values than their high heritability counterparts (    The gap between LMM and PCA was reduced in these evaluations, but the main conclusion of the high heritability evaluation holds for low heritability as well, namely that LMM with r = 0 significantly outperforms or ties LMM with r > 0 and PCA in all cases (Table 5).
Lastly, we simulated traits with both low heritability and large environment effects determined by geography and subpopulation labels, so they are strongly correlated to the low-dimensional population structure. For that reason, PCs may be expected to perform better in this setting (in either PCA or LMM). However, we find that both PCA and LMM (even without PCs) increase their AUC PR values compared to the low-heritability evaluations (  (Table 6). Additionally, on RC traits only, PCA significantly outperforms LMM in the three real human datasets (Table 6), the only cases in all of our evaluations where this is observed. For comparison, we also evaluate an 'oracle' LMM without PCs but with the finest group labels, the same used to simulate environment, as fixed categorical covariates ('LMM lab.'), and see much larger AUC PR values than either LMM with PCs or PCA ( Table 6). However, LMM with labels is often more poorly calibrated than LMM or PCA without labels, which may be since these numerous labels are inappropriately modeled as fixed rather than random effects. Overall, we find that association studies with correlated environment and genetic effects remain a challenge for PCA and LMM, that addition of PCs to an LMM improves performance only marginally, and that if the environment effect is driven by geography or ethnicity then use of those labels greatly improves performance compared to using PCs.

Discussion
Our evaluations conclusively determined that LMM without PCs performs better than PCA (for any number of PCs) across all scenarios without environment effects, including all real and simulated genotypes and two trait simulation models. Although the addition of a few PCs to LMM does not greatly hurt its performance (except for small sample sizes), they generally did not improve it either ( Table 3,  Table 5), which agrees with previous observations Janss et al., 2012) but contradicts others (Zhao et al., 2007;Price et al., 2010). Our findings make sense since PCs are the eigenvectors of the same kinship matrix that parameterized random effects, so including both is redundant. The presence of environment effects that are correlated to relatedness presents the only scenario where occasionally PCA and LMM with PCs outperform LMM without PCs ( Table 6). It is commonly believed that PCs model such environment effects well (Novembre et al., 2008;Zhang and Pan, 2015;Lin et al., 2021). However, we observe that LMM without PCs models environment effects nearly as well as with PCs (Figure 8), consistent with previous findings (Vilhjálmsson and Nordborg, 2013;Wang et al., 2022) and with environment inflating heritability estimates using LMM (Heckerman et al., 2016). Moreover, modeling the true environment groups as fixed categorical effects always substantially improved AUC PR compared to modeling them with PCs ( Figure 8, Table 6). Modeling numerous environment groups as fixed effects does result in deflated p-values ( Figure 8, Table 6), which we expect would be avoided by modeling them as random effects, a strategy we chose not to pursue here as it is both a circular evaluation (the true effects were drawn from that model) and out of scope. Overall, including PCs to model environment effects yields limited power gains if at all, even in an LMM, and is no replacement for more adequate modeling of environment whenever possible.
Previous studies found that PCA was better calibrated than LMM for unusually differentiated markers (Price et al., 2010;Wu et al., 2011;Yang et al., 2014), which as simulated were an artificial scenario not based on a population genetics model, and are otherwise believed to be unusual (Sul and Eskin, 2013;Price et al., 2013). Our evaluations on real human data, which contain such loci in Tie if no significant difference using Bonferroni threshold; in last column, pairwise ties are specified and "Tie" is three-way tie.
relevant proportions if they exist, do not replicate that result. Family relatedness strongly favors LMM, an advantage that probably outweighs this potential PCA benefit in real data. Relative to LMM, the behavior of PCA fell between two extremes. When PCA performed well, there was a small number of PCs with both calibrated p-values and AUC PR near that of LMM without PCs. Conversely, PCA performed poorly when no number of PCs had either calibrated p-values or acceptably large AUC PR . There were no cases where high numbers of PCs optimized an acceptable AUC PR , or cases with miscalibrated p-values but high AUC PR . PCA performed well in the admixture simulations (without families, both trait models), real human genotypes with RC traits, and the subpopulation tree simulations (both trait models). Conversely, PCA performed poorly in the admixed family simulation (both trait models) and the real human genotypes with FES traits.
PCA assumes that genetic relatedness is restricted to a low-dimensional subspace, whereas LMM can handle high-dimensional relatedness. Thus, PCA performs well in the admixture simulation, which is explicitly low-dimensional (see Genotype simulation from the admixture model), and our subpopulation tree simulations, which are likely well approximated by a few dimensions despite the large number of subpopulations because there are few long branches. Conversely, PCA performs poorly under family structure because its kinship matrix is high-dimensional ( Figure 6-figure supplement  1). However, estimating the latent space dimensions of real datasets is challenging because estimated eigenvalues have biased distributions (Hayashi et al., 2018). Kinship matrix rank estimated using the Tracy-Widom test  did not fully predict the datasets that PCA performs well on. In contrast, estimated local kinship finds considerable cryptic family relatedness in all real human datasets and better explains why PCA performs poorly there. The trait model also influences the relative performance of PCA, so genotype-only parameters (eigenvalues or local kinship) alone cannot tell the full story. There are related tests for numbers of dimensions that consider the trait which we did not consider, including the Bayesian information criterion for the regression with PCs against the trait (Zhu and Yu, 2009). Additionally, PCA and LMM goodness of fit could be compared using the coefficient of determination generalized for LMMs (Sun et al., 2010).
PCA is at best underpowered relative to LMMs, and at worst miscalibrated regardless of the numbers of PCs included, in real human genotype tests. Among our simulations, such poor performance occurred only in the admixed family. Local kinship estimates reveal considerable family relatedness in the real datasets absent in the corresponding subpopulation tree simulations. Admixture is also absent in our tree simulations, but our simulations and theory show that admixture is well handled by PCA. Hundreds of close relative pairs have been identified in 1000 Genomes (Gazal et al., 2015;Al Khudhair et al., 2015;Fedorova et al., 2016;Schlauch et al., 2017), but their removal does not improve PCA performance sufficiently in our tests, so the larger number of more distantly related pairs are PCA's most serious obstacle in practice. Distant relatives are expected to be numerous in any large human dataset (Henn et al., 2012;Shchur and Nielsen, 2018;Loh et al., 2018). Our FES trait tests show that family relatedness is more challenging when rarer variants have larger coefficients. Overall, the high relatedness dimensions induced by family relatedness is the key challenge for PCA association in modern datasets that is readily overcome by LMM.
Our tests also found PCA robust to large numbers of PCs, far beyond the optimal choice, agreeing with previous anecdotal observations Kang et al., 2010), in contrast to using too few PCs for which there is a large performance penalty. The exception was the small sample size simulation, where only small numbers of PCs performed well. In contrast, LMM is simpler since there is no need to choose the number of PCs. However, an LMM with a large number of covariates may have conservative p-values, as observed for LMM with large numbers of PCs, which is a weakness of the score test used by the LMM we evaluated that may be overcome with other statistical tests. Simulations or post hoc evaluations remain crucial for ensuring that statistics are calibrated.
There are several variants of the PCA and LMM analyses, most designed for better modeling linkage disequilibrium (LD), that we did not evaluate directly, in which PCs are no longer exactly the top eigenvectors of the kinship matrix (if estimated with different approaches), although this is not a crucial aspect of our arguments. We do not consider the case where samples are projected onto PCs estimated from an external sample (Privé et al., 2020), which is uncommon in association studies, and whose primary effect is shrinkage, so if all samples are projected then they are all equally affected and larger regression coefficients compensate for the shrinkage, although this will no longer be the case if only a portion of the sample is projected onto the PCs of the rest of the sample. Another approach tests PCs for association against every locus in the genome in order to identify and exclude PCs that capture LD structure (which is localized) instead of ancestry (which should be present across the genome; Privé et al., 2020); a previous proposal removes LD using an autocorrelation model prior to estimating PCs . These improved PCs remain inadequate models of family relatedness, so an LMM will continue to outperform them in that setting. Similarly, the leaveone-chromosome-out (LOCO) approach for estimating kinship matrices for LMMs prevents the test locus and loci in LD with it from being modeled by the random effect as well, which is called 'proximal contamination' (Lippert et al., 2011;Yang et al., 2014). While LOCO kinship estimates vary for each chromosome, they continue to model family relatedness, thus maintaining their key advantage over PCA. The LDAK model estimates kinship instead by weighing loci taking LD into account (Speed et al., 2012). LD effects must be adjusted for, if present, so in unfiltered data we advise the previous methods be applied. However, in this work, simulated genotypes do not have LD, and the real datasets were filtered to remove LD, so here there is no proximal contamination and LD confounding is minimized if present at all, so these evaluations may be considered the ideal situation where LD effects have been adjusted successfully, and in this setting LMM outperforms PCA. Overall, these alternative PCs or kinship matrices differ from their basic counterparts by either the extent to which LD influences the estimates (which may be a confounder in a small portion of the genome, by definition) or by sampling noise, neither of which are expected to change our key conclusion.
One of the limitations of this work include relatively small sample sizes compared to modern association studies. However, our conclusions are not expected to change with larger sample sizes, as cryptic family relatedness will continue to be abundant in such data, if not increase in abundance, and thus give LMMs an advantage over PCA (Henn et al., 2012;Shchur and Nielsen, 2018;Loh et al., 2018). One reason PCA has been favored over classic LMMs is because PCA's runtime scales much better with increasing sample size. However, recent approaches not tested in this work have made LMMs more scalable and applicable to biobank-scale data (Loh et al., 2015;Zhou et al., 2018;Mbatchou et al., 2021), so one clear next step is carefully evaluating these approaches in simulations with larger sample sizes. A different benefit for including PCs were recently reported for BOLT-LMM, which does not result in greater power but rather in reduced runtime, a property that may be specific to its use of scalable algorithms such as conjugate gradient and variational Bayes (Loh et al., 2018). Many of these newer LMMs also no longer follow the infinitesimal model of the basic LMM (Loh et al., 2015;Mbatchou et al., 2021), and employ novel approximations, which are features not evaluated in this work and worthy of future study.
Another limitation of this work is ignoring rare variants, a necessity given our smaller sample sizes, where rare variant association is miscalibrated and underpowered. Using simulations mimicking the UK Biobank, recent work has found that rare variants can have a more pronounced structure than common variants, and that modeling this rare variant structure (with either PCA and LMM) may better model environment confounding, reduce inflation in association studies, and ameliorate stratification in polygenic risk scores (Zaidi and Mathieson, 2020). Better modeling rare variants and their structure is a key next step in association studies.
The largest limitation of our work is that we only considered quantitative traits. Previous evaluations involving case-control traits tended to report PCA-LMM ties or mixed results, an observation potentially confounded by the use of low-dimensional simulations without family relatedness ( Table 1). An additional concern is case-control ascertainment bias and imbalance, which appears to affect LMMs more severely, although recent work appears to solve this problem (Yang et al., 2014;Zhou et al., 2018). Future evaluations should aim to include our simulations and real datasets, to ensure that previous results were not biased in favor of PCA by not simulating family structure or larger coefficients for rare variants that are expected for diseases by various selection models.
Overall, our results lead us to recommend LMM over PCA for association studies in general. Although PCA offer flexibility and speed compared to LMM, additional work is required to ensure that PCA is adequate, including removal of close relatives (lowering sample size and wasting resources) followed by simulations or other evaluations of statistics, and even then PCA may perform poorly in terms of both type I error control and power. The large numbers of distant relatives expected of any real dataset all but ensures that PCA will perform poorly compared to LMM (Henn et al., 2012;Shchur and Nielsen, 2018;Loh et al., 2018). Our findings also suggest that related applications such as polygenic models may enjoy gains in power and accuracy by employing an LMM instead of PCA to model relatedness (Rakitsch et al., 2013;Qian et al., 2020). PCA remains indispensable across population genetics, from visualizing population structure and performing quality control to its deep connection to admixture models, but the time has come to limit its use in association testing in favor of LMM or other, richer models capable of modeling all forms of relatedness.

Materials and methods
The complex trait model and PCA and LMM approximations Let x ij ∈ {0, 1, 2} be the genotype at the biallelic locus i for individual j , which counts the number of reference alleles. Suppose there are n individuals and m loci, X = (x ij ) is their m × n genotype matrix, and y is the lengthn column vector of individual trait values. The additive linear model for a quantitative (continuous) trait is: where 1 is a lengthn vector of ones, α is the scalar intercept coefficient, β is the lengthm vector of locus coefficients, Z is a design matrix of environment effects and other covariates, η is the vector of environment coefficients, ϵ is a lengthn vector of residuals, and the superscript prime symbol ( ′ ) denotes matrix transposition. The residuals follow ϵ j ∼ Normal(0, σ 2 ϵ ) independently per individual j , for some σ 2 ϵ . The full model of Equation 1, which has a coefficient for each of the m loci, is underdetermined in current datasets where m ≫ n . The PCA and LMM models, respectively, approximate the full model fit at a single locus i : where x i is the lengthn vector of genotypes at locus i only, β i is the locus coefficient, Ur is an n × r matrix of PCs, γ r is the lengthr vector of PC coefficients, s is a lengthn vector of random effects, Φ T = (φ T jk ) is the n × n kinship matrix conditioned on the ancestral population T , and σ 2 s is a variance factor. Both models condition the regression of the focal locus i on an approximation of the total polygenic effect X ′ β with the same covariance structure, which is parameterized by the kinship matrix. Under the kinship model, genotypes are random variables obeying where p T i is the ancestral allele frequency of locus i (Malécot, 1948;Wright, 1949;Jacquard, 1970;Astle and Balding, 2009). Assuming independent loci, the covariance of the polygenic effect is which is readily modeled by the LMM random effect s , where the difference in mean is absorbed by the intercept. Alternatively, consider the eigendecomposition of the kinship matrix Φ T = UΛU ′ where U is the n × n eigenvector matrix and Λ is the n × n diagonal matrix of eigenvalues. The random effect can be written as which follows from the affine transformation property of multivariate normal distributions. Therefore, the PCA term Urγ r can be derived from the above equation under the additional assumption that the kinship matrix has approximate rank r and the coefficients γ r are fit without constraints. In contrast, the LMM uses all eigenvectors, while effectively shrinking their coefficients γ LMM as all random effects models do, although these parameters are marginalized (Astle and Balding, 2009;Janss et al., 2012;Hoffman and Dubé, 2013;Zhang and Pan, 2015). PCA has more parameters than LMM, so it may overfit more: ignoring the shared terms in Equation 2 and Equation 3, PCA fits r parameters (length of γ ), whereas LMMs fit only one ( σ 2 s ).
In practice, the kinship matrix used for PCA and LMM is estimated with variations of a method-ofmoments formula applied to standardized genotypes X S , which is derived from Equation 4: where the unknown p T i is estimated by p T i = 1 2n ∑ n j=1 x ij Kang et al., 2008;Kang et al., 2010;Yang et al., 2011;Zhou and Stephens, 2012;Yang et al., 2014;Loh et al., 2015;Sul et al., 2018;Zhou et al., 2018). However, this kinship estimator has a complex bias that differs for every individual pair, which arises due to the use of this estimated p T i (Ochoa and Storey, 2021;Ochoa and Storey, 2019). Nevertheless, in PCA and LMM these biased estimates perform as well as unbiased ones (Hou et al., 2023b).
We selected fast and robust software implementing the basic PCA and LMM models. PCA association was performed with plink2 (Chang et al., 2015). The quantitative trait association model is a linear regression with covariates, evaluated using the t-test. PCs were calculated with plink2, which equal the top eigenvectors of Equation 5 after removing loci with minor allele frequency MAF < 0.1 .
LMM association was performed using GCTA (Yang et al., 2011;Yang et al., 2014). Its kinship estimator equals Equation 5. PCs were calculated using GCTA from its kinship estimate. Association significance is evaluated with a score test. In the small simulation only, GCTA with large numbers of PCs had convergence and singularity errors in some replicates, which were treated as missing data.

Simulations
Every simulation was replicated 50 times, drawing anew all genotypes (except for real datasets) and traits. Below we use the notation f B A for the inbreeding coefficient of a subpopulation A from another subpopulation B ancestral to A . In the special case of the total inbreeding of A , f T A , T is an overall ancestral population, which is ancestral to every individual under consideration, such as the most recent common ancestor (MRCA) population.

Genotype simulation from the admixture model
The basic admixture model is as described previously (Ochoa and Storey, 2021) and is implemented in the R package bnpsd. Both Large and Family simulations have n = 1, 000 individuals, while Small has n = 100 . The number of loci is m = 100, 000 . Individuals are admixed from K = 10 intermediate subpopulations, or ancestries. Each subpopulation Su ( u ∈ {1, ..., K} ) is at coordinate u and has an inbreeding coefficient f T S u = uτ for some τ . Ancestry proportions q ju for individual j and Su arise from a random walk with spread σ on the 1D geography, and τ and σ are fit to give F ST = 0.1 and mean kinship θ T = 0.5F ST for the admixed individuals (Ochoa and Storey, 2021). Random ancestral allele frequencies p T i , subpopulation allele frequencies p Su i , individual-specific allele frequencies π ij , and genotypes x ij are drawn from this hierarchical model: where this Beta is the Balding-Nichols distribution (Balding and Nichols, 1995) with mean p T i and variance p T Fixed loci ( i where x ij = 0 for all j , or x ij = 2 for all j ) are drawn again from the model, starting from p T i , iterating until no loci are fixed. Each replicate draws a genotypes starting from p T i . As a brief aside, we prove that global ancestry proportions as covariates is equivalent in expectation to using PCs under the admixture model. Note that the latent space of X , which is the subspace to which the data is constrained by the admixture model, is given by (π ij ) , which has K dimensions (number of columns of Q = (q ju ) ), so the top K PCs span this space. Since associations include an intercept term ( 1α in Equation 2), estimated PCs are orthogonal to 1 (note Φ T 1 = 0 because X S 1 = 0 ), and the sum of rows of Q sums to one, then only K − 1 PCs plus the intercept are needed to span the latent space of this admixture model.

Genotype simulation from random admixed families
We simulated a pedigree with admixed founders, no close relative pairings, assortative mating based on a 1D geography (to preserve admixture structure), random family sizes, and arbitrary numbers of generations (20 here). This simulation is implemented in the R package simfam. Generations are drawn iteratively. Generation 1 has n = 1000 individuals from the above admixture simulation ordered by their 1D geography. Local kinship measures pedigree relatedness; in the first generation, everybody is locally unrelated and outbred. Individuals are randomly assigned sex. In the next generation, individuals are paired iteratively, removing random males from the pool of available males and pairing them with the nearest available female with local kinship < 1/4 3 (stay unpaired if there are no matches), until there are no more available males or females. Let n = 1000 be the desired population size, nm = 1 the minimum number of children per family and n f the number of families (paired parents) in the current generation, then the number of additional children (beyond the minimum) is drawn from Poisson(n/n f − nm) . Let δ be the difference between desired and current population sizes. If δ > 0 , then δ random families are incremented by 1. If δ < 0 , then |δ| random families with at least nm + 1 children are decremented by 1. If |δ| exceeds the number of families, all families are incremented or decremented as needed and the process is iterated. Children are assigned sex randomly, and are reordered by the average coordinate of their parents. Children draw alleles from their parents independently per locus. A new random pedigree is drawn for each replicate, as well as new founder genotypes from the admixture model.

Genotype simulation from a subpopulation tree model
This model draws subpopulations allele frequencies from a hierarchical model parameterized by a tree, which is also implemented in bnpsd and relies on the R package ape for general tree data structures and methods (Paradis and Schliep, 2019). The ancestral population T is the root, and each node is a subpopulation Sw indexed arbitrarily. Each edge between Sw and its parent population Pw has an inbreeding coefficient f Pw Sw . P T i are drawn from a given distribution, which is constructed to mimic each real dataset in Appendix 1. Given the allele frequencies p Pw i of the parent population, Sw 's allele frequencies are drawn from: .
Individuals j in Sw draw genotypes from its allele frequency: . Loci with MAF < 0.01 are drawn again starting from the p T i distribution, iterating until no such loci remain.

Fitting subpopulation tree to real data
We developed new methods to fit trees to real data based on unbiased kinship estimates from popkin, implemented in bnpsd. A tree with given inbreeding coefficients f Pw Sw for its edges (between subpopulation Sw and its parent Pw ) gives rise to a coancestry matrix ϑ T uv for a subpopulation pair ( Su, Sv ), and the goal is to recover these edge inbreeding coefficients from coancestry estimates. Coancestry values are total inbreeding coefficients of the MRCA population of each subpopulation pair. Therefore, we calculate f T S w for every Sw recursively from the root as follows. Nodes with parent Pw = T are already as desired. Given f T Pw , the desired f T S w is calculated via the 'additive edge' δw (Ochoa and Storey, 2021): These δw ≥ 0 because 0 ≤ f Pw Sw , f T P w ≤ 1 for every w . Edge inbreeding coefficients can be recovered from additive edges: f Pw Sw = δw/(1 − f T P w ) . Overall, coancestry values are sums of δw over common ancestor nodes, where the sum includes all w , and Iw(u, v) equals 1 if Sw is a common ancestor of Su, Sv , 0 otherwise. Note that Iw(u, v) reflects tree topology and δw edge values.
To estimate population-level coancestry, first kinship ( φ T jk ) is estimated using popkin (Ochoa and Storey, 2021). Individual coancestry ( θ T jk ) is estimated from kinship using Lastly, coancestry θ T uv between subpopulations are averages of individual coancestry values: Topology is estimated with hierarchical clustering using the weighted pair group method with arithmetic mean (Sokal and Michener, 1958) −θ T uv , which succeeds due to the monotonic relationship between node depth and coancestry (Equation 7). This algorithm recovers the true topology from the true coancestry values, and performs well for estimates from genotypes.
To estimate tree edge lengths, first δw are estimated from θ T uv and the topology using Equation 7 and non-negative least squares linear regression (Lawson and Hanson, 1974) (implemented in nnls;Mullen, 2012) to yield non-negative δw , and f Pw Sw are calculated from δw by reversing Equation 5. To account for small biases in coancestry estimation, an intercept term δ 0 is included ( I 0 (u, v) = 1 for all u, v ), and when converting δw to f Pw Sw , δ 0 is treated as an additional edge to the root, but is ignored when drawing allele frequencies from the tree.

Trait simulation
Traits are simulated from the quantitative trait model of Equation 1, with novel bias corrections for simulating the desired heritability from real data relying on the unbiased kinship estimator popkin (Ochoa and Storey, 2021). This simulation is implemented in the R package simtrait. All simulations have a fixed narrow-sense heritability of h 2 , a variance proportion due to environment effects σ 2 η , and residuals are drawn from ϵ j ∼ Normal(0, σ 2 ϵ ) with σ 2 ϵ = 1 − h 2 − σ 2 η . The number of causal loci m 1 , which determines the average coefficient size, is chosen with the heuristic formula m 1 = round(nh 2 /8) , which empirically balances power well with varying n and h 2 . The set of causal loci C is drawn anew for each replicate, from loci with MAF ≥ 0.01 to avoid rare causal variants, which are not discoverable by PCA or LMM at the sample sizes we considered.
, the effect size of locus i equals 2v T i β 2 i , its contribution of the trait variance (Park et al., 2010). Under the fixed effect sizes (FES) model, initial causal coefficients are for known p T i ; otherwise v T i is replaced by the unbiased estimator (Ochoa and Storey, 2021) where φ T is the mean kinship estimated with popkin. Each causal locus is multiplied by -1 with probability 0.5. Alternatively, under the random coefficients (RC) model, initial causal coefficients are drawn independently from β i ∼ Normal(0, 1) . For both models, the initial genetic variance is σ 2 0 = ∑ i∈C 2v T i β 2 i , replacing v T i with v T i for unknown p T i (so σ 2 0 is an unbiased estimate), so we multiply every initial β i by h σ0 to have the desired heritability. Lastly, for known p T i , the intercept coefficient is α = − ∑ i∈C 2p T i β i . When p T i are unknown, p T i should not replace p T i since that distorts the trait covariance (for the same reason the standard kinship estimator in Equation 5 is biased), which is avoided with Simulations optionally included multiple environment group effects, similarly to previous models (Zhang and Pan, 2015;Wang et al., 2022), as follows. Each independent environment i has predefined groups, and each group g has random coefficients drawn independent from η gi ∼ Normal(0, σ 2 ηi ) where σ 2 ηi is a specified variance proportion for environment i . Z has individuals along columns and environment-groups along rows, and it contains indicator variables: 1 if the individual belongs to the environment-group, 0 otherwise.
We performed trait simulations with the following variance parameters ( Table 7): high heritability used h 2 = 0.8 and no environment effects; low heritability used h 2 = 0.3 and no environment effects; lastly, environment used h 2 = 0.3, σ 2 η1 = 0.3, σ 2 η2 = 0.2 (total σ 2 η = σ 2 η1 + σ 2 η2 = 0.5 ). For real genotype datasets, the groups are the continental (environment 1) and fine-grained (environment 2) subpopulation labels given (see next subsection). For simulated genotypes, we created these labels by grouping by the index j (geographical coordinate) of each simulated individual, assigning group g = ceiling(jk i /n) where k i is the number of groups in environment i , and we selected k 1 = 5 and k 2 = 25 to mimic the number of groups in each level of 1000 Genomes ( Table 2).

Real human genotype datasets
The three datasets were processed as before (Ochoa and Storey, 2019; summarized below), except with an additional filter so loci are in approximate linkage equilibrium and rare variants are removed. All processing was performed with plink2 (Chang et al., 2015), and analysis was uniquely enabled by the R packages BEDMatrix (Grueneberg and de Los Campos, 2019) and genio. Each dataset groups individuals in a two-level hierarchy: continental and fine-grained subpopulations. Final dataset sizes are in Table 2.
We obtained the full (including non-public) Human Origins by contacting the authors and agreeing to their usage restrictions. The Pacific data (Skoglund et al., 2016) was obtained separately from the rest (Lazaridis et al., 2014;Lazaridis et al., 2016), and datasets were merged using the intersection of loci. We removed ancient individuals, and individuals from singleton and non-native subpopulations. Non-autosomal loci were removed. Our analysis of both the whole-genome sequencing (WGS) version of HGDP (Bergström et al., 2020) and the high-coverage NYGC version of 1000 Genomes (Fairley et al., 2020) was restricted to autosomal biallelic SNP loci with filter "PASS".
Since our evaluations assume uncorrelated loci, we filtered each real dataset with plink2 using parameters "--indep-pairwise 1000kb 0.3", which iteratively removes loci that have a greater than 0.3 squared correlation coefficient with another locus that is within 1000 kb, stopping until no such loci remain. Since all real datasets have numerous rare variants, while PCA and LMM are not able to detect associations involving rare variants, we removed all loci with MAF < 0.01 . Lastly, only HGDP had loci with over 10% missingness removed, as they were otherwise 17% of remaining loci (for Human Origins and 1000 Genomes they were under 1% of loci so they were not removed). Kinship matrix rank and eigenvalues were calculated from popkin kinship estimates. Eigenvalues were assigned p-values with twstats of the Eigensoft package , and kinship matrix rank was estimated as the largest number of consecutive eigenvalue from the start that all satisfy p < 0.01 (p-values did not increase monotonically). For the evaluation with close relatives removed, each dataset was filtered with plink2 with option "--king-cutoff" with cutoff 0.02209709 ( = 2 −11/2 ) for removing up to 4th degree relatives using KING-robust (Manichaikul et al., 2010), and MAF < 0.01 filter is reapplied ( Table 4).

Evaluation of performance
All approaches are evaluated using two complementary metrics: SRMSDp quantifies p-value uniformity, and AUC PR measures causal locus classification performance and reflects power while ranking miscalibrated models fairly. These measures are more robust alternatives to previous measures from the literature (Appendix 2), and are implemented in simtrait. P-values for continuous test statistics have a uniform distribution when the null hypothesis holds, a crucial assumption for type I error and FDR control (Storey, 2003;Storey and Tibshirani, 2003). We use the Signed Root Mean Square Deviation ( SRMSDp ) to measure the difference between the observed null p-value quantiles and the expected uniform quantiles: where m 0 = m − m 1 is the number of null (non-causal) loci, here i indexes null loci only, p (i) is the i th ordered null p-value, u i = (i − 0.5)/m 0 is its expectation, p median is the median observed null p-value, u median = 1 2 is its expectation, and sgn is the sign function (1 if u median ≥ p median , -1 otherwise). Thus, SRMSDp = 0 corresponds to calibrated p-values, SRMSDp > 0 indicate anti-conservative p-values, and SRMSDp < 0 are conservative p-values. The maximum SRMSDp is achieved when all p-values are zero (the limit of anti-conservative p-values), which for infinite loci approaches The same value with a negative sign occurs for all p-values of 1. Precision and recall are standard performance measures for binary classifiers that do not require calibrated p-values (Grau et al., 2015). Given the total numbers of true positives (TP), false positives (FP) and false negatives (FN) at some threshold or parameter t , precision and recall are Precision(t) = TP(t) TP(t) + FP(t) ,

Recall(t) = TP(t) TP(t) + FN(t) .
Precision and Recall trace a curve as t is varied, and the area under this curve is AUC PR . We use the R package PRROC to integrate the correct non-linear piecewise function when interpolating between points. A model obtains the maximum AUC PR = 1 if there is a t that classifies all loci perfectly. In contrast, the worst models, which classify at random, have an expected precision ( = AUC PR ) equal to the overall proportion of causal loci: m 1 /m .

Appendix 2
Comparisons between SRMSDp , AUC PR , and evaluation measures from the literature 2.1 The inflation factor λ Test statistic inflation has been used to measure model calibration (Astle and Balding, 2009;Price et al., 2010). The inflation factor λ is defined as the median χ 2 association statistic divided by theoretical median under the null hypothesis (Devlin and Roeder, 1999). To compare p-values from nonχ 2 tests (such as t-statistics), λ can be calculated from p-values using where p median is the median observed p-value (including causal loci), u median = 1 2 is its null expectation, and F is the χ 2 cumulative density function ( F −1 is the quantile function).
To compare λ and SRMSDp directly, for simplicity assume that all p-values are null. In this case, calibrated p-values give λ = 1 and SRMSDp = 0 . However, non-uniform p-values with the expected median, such as from genomic control (Devlin and Roeder, 1999), result in λ = 1 , but SRMSDp ̸ = 0 except for uniform p-values, a key flaw of λ that SRMSDp overcomes. Inflated statistics (anticonservative p-values) give λ > 1 and SRMSDp > 0 . Deflated statistics (conservative p-values) give λ < 1 and SRMSDp < 0 . Thus, λ ̸ = 1 always implies SRMSDp ̸ = 0 (where λ − 1 and SRMSDp have the same sign), but not the other way around. Overall, λ depends only on the median p-value, while SRMSDp uses the complete distribution. However, SRMSDp requires knowing which loci are null, so unlike λ it is only applicable to simulated traits.
We fit this model to λ > 1 only since it was less noisy and of greater interest, and obtained the curve shown in Figure 2-figure supplement 1 with a = 0.564 and b = 0.619 . The value λ = 1.05 , a common threshold for benign inflation (Price et al., 2010), corresponds to SRMSDp = 0.0085 according to Equation 10. Conversely, SRMSDp = 0.01 , serving as a simpler rule of thumb, corresponds to λ = 1.06 .

Type I error rate
The type I error rate is the proportion of null p-values with p ≤ t . Calibrated p-values have type I error rate near t , which may be evaluated with a binomial test. This measure may give different results for different t , for example be significantly miscalibrated only for large t (due to lack of power for smaller t ), and it requires large simulations to estimate well as it depends on the tail of the distribution. In contrast, SRMSDp uses the entire distribution so it is easier to estimate, SRMSDp = 0 guarantees calibrated type I error rates at all t , while large |SRMSDp| indicates incorrect type I errors for a range of t . Empirically, we find the expected agreement and monotonic relationship between SRMSDp and type I error rate (Figure 2-figure supplement 2).

Statistical power and comparison to AUC PR
Power is the probability that a test is declared significant when the alternative hypothesis H 1 holds. At a p-value threshold t , power equals