The impact of the COVID‐19 pandemic on school‐level scholarly outcomes and research focus of pharmacy practice faculty

The effects of the coronavirus disease 2019 (COVID‐19) pandemic on scholarly outcomes and research focus of schools of pharmacy (SOPs) have not been studied.

output, especially for tenure-track faculty with probationary periods spanning these pandemic years.

K E Y W O R D S
bibliometrics, COVID-19, pharmacy faculty, pharmacy research, pharmacy schools, scholarly communication

| INTRODUCTION
Coronavirus disease 2019 (COVID- 19) was declared a public health emergency of international concern by the World Health Organization in January 2020. 1 Major disruption started after a nationwide state of emergency was declared.Lockdown policies were thereafter led by states, 2 during which schools of pharmacy (SOPs) changed their modes of instruction to virtual/online-only or hybrid systems 3,4 to comply with their respective state-level lockdown policies while maintaining accreditation standards. 5[8][9] Due to the challenges, stress, and increased time and workload caused by the emergency changes, research and scholarship activitiesessential components of their tripartite mission, were deprioritized, paused, or suspended. 9,10[13] The pandemic affected lab-based, basic science researchers who conduct time-sensitive experiments in wet labs more than researchers from other fields. 14,15Thus, studies in Science, Technology, Engineering, Mathematics, and Medicine (STEMM) fields 12,13,16 are too heterogenous to be generalizable to pharmacy practice.[8][9] Pre-pandemic tenure and promotion scholarship benchmarks are also different across SOPs. 18Because scholarship is a key tenure and promotion requirement, 10,18 it is important to understand how COVID-19 impacted this metric within SOPs.
This study aimed to determine how COVID-19 impacted scholarly outcomes measured by the proportion of pharmacy practice faculty who published any document (scholarly activity ratio; SAR) and the number of documents published (scholarly output; SO) at the SOP level within regions of the United States.As SOP-level metrics, SAR and SO represent constituent faculties' levels of involvement or participation rate and productivity in scholarship, respectively.Taking 2018-2019 and 2020-2021 as the periods before and during the pandemic, respectively, we hypothesized that the pandemic affected SAR and SO of SOPs by geographic regions-a relevant factor for both scholarly outcomes 19,20 and state-level COVID-19 policies and outcomes. 2,17Lastly, we explored the shift in research focus during the pandemic based on author keyword co-occurrence analysis.1).These criteria assumed that funding rank correlates with research and scholarly productivity, 21 and that these were the most likely affected SOPs.The SOPs, their 4-year funding, and ranks based on total funding are available in Table S1.
The NIH-AACP document also contained the Carnegie classification of the SOPs.The institution type (private/public) and being part of an academic health center were obtained from the Pharmacy College Application Service (PharmCAS; https://www.pharmcas.org/schooldirectory/explore-and-compare/public-or-private).The geographic/ census region by state were obtained from Census.gov (https://www.census.gov/programs-surveys/economic-census/guidance-geographies/levels.html).A consistent school name was used as primary keys and data frames from the different sources were merged using Python (version 3.10.11)pandas library (version 1.5.2). 22

| Pharmacy practice faculty names
A list of pharmacy practice faculty from each school was obtained from the AACP Roster of Faculty and Professional Staff. 23Each SOP was searched separately, using the corresponding school names in the Institutional members' directory. 24The search fields were highest rank (Dean [associate, assistant, campus or interim], professor [including assistant and associate]), and discipline (pharmacy practice).For schools that returned less than 10 faculty members, "pharmacotherapy and translational research" discipline was included.The names returned from this expanded search were then cross-referenced with faculty listed on the websites of the departments/divisions that most closely correspond to pharmacy practice.Only faculty in this expanded list, found on the SOP's websites and with a Doctor of Pharmacy (PharmD) degree with or without a Doctor of Philosophy (PhD) degree, were included.

| Scopus search
The Scopus database (Scopus.com) was used to identify the publications by pharmacy practice faculty from each SOP.Scopus provides metrics comparable to Web of Science, but has a broader coverage of pharmacy journals. 25Individual faculty members' Scopus IDs (SCIDs) were first obtained.Then the Scopus advanced document search was used to identify the publications of all faculty (within each SOP) using the unique SCIDs obtained in step 1, with the following refinements: Year ("2018," "2019," "2020," "2021"), Language ("English"), and Source type ("Journals").This search was completed in November 2022.The "Analyze" function was used, and document count data for each of the 4 years and the number of Scopus-categorized document types (articles and reviews) were obtained for each SOP.The final collection (corpus) for each SOP was exported as .csvfiles, and duplicates were removed before further processing.

| Author keyword and title analysis
Author keywords represent research focus and can also be used to track and analyze research trajectories. 26,27We analyzed the evolution of research focus before and during the pandemic.The unique before and during pandemic corpora were subjected to a co-occurrence analysis using author keywords as the unit of analysis in VOSviewer (version 1.6.19). 28The occurrence threshold was initially set at 10 to identify similar words.A thesaurus was then generated to consolidate similar words (Table S2).COVID-19-related keywords ("SARS-CoV-2," "pandemic," "pandemics," and "coronavirus") were consolidated to "COVID-19."The final minimum occurrence F I G U R E 1 Inclusion and exclusion criteria.The National Institute of Health (NIH) American Association of Colleges of Pharmacy (AACP) institutional ranking documents for fiscal years 2018-2021 were assessed.A total of 117 schools of pharmacy (SOPs) were found before the inclusion and exclusion criteria were applied.
(threshold) was set at 20.The network and map data were exported, and the keywords were compared between the two periods.Keywords were designated as either "Submerged," "Emerged," or "Reemerged" based on their above-threshold occurrence only before the pandemic, only during the pandemic, or in both periods, respectively.
Re-emerged keywords were ranked according to their occurrences in each period; the change in rank (before minus during) were calculated, broader than what would be considered as either scholarship of teaching and learning or educational research. 29

| Statistical analysis
A generalized linear mixed model (GLMM) is indicated for the response variables SAR (proportion) and SO (count), respectively (Box S1).However, because model diagnostics of the GLMM showed overdispersion, the outcome measures were transformed to square and square root for SAR and SO, respectively. 30,31Overdispersion occurs in GLMMs when the variance is higher than model prediction.
If not addressed, it leads to Type I errors. 30,31These transformations resulted in normal distributions (Shapiro-Wilk Normality Test; p > 0.05); therefore, general (Gaussian) linear mixed effects modeling was employed. 30,31The lmer function in the lme4 32 R (version 4.1.3)package was used.
Our hypothesis-informed "base model" included the pandemic and region as fixed effects and SOP as the random effect.The effects of known predictors of scholarly productivity, including NIH-funding rank, SOP type (private/public), Carnegie classification (R1/nonR1), and being part of an academic health center, 20,21,33 and additional predictors were tested in that order, using model comparisons with the performance package. 34The additional predictors were related to the contribution of review documents to the outcome measures as we believe that compared with articles; reviews would not have been hampered by institutional review board (IRB) bottlenecks and access to research subjects during the pandemic.These predictors include the proportion of faculty who published reviews (faculty Review fraction; relative to all faculty whose SCIDs were found), and proportion of reviews published (documents Review fraction; relative to all documents published) for SAR and SO, respectively.In contrast to the variables in the base model, other variables were only included if they had significant effects on the response variables or if they significantly improved the model based on lower Akaike and/or Bayesian Information Criteria (AIC and BIC) and higher explanatory powers (total, and from fixed effects alone [conditional and marginal R 2 , respectively]), consistent with a stepwise model selection method.Only one of highly correlated funding rank variables were included (2018-2019 versus 2018-2021).Effect sizes were determined with the report package.The emmeans package 35 was used to calculate the estimated marginal means (EMM) and 95% confidence intervals, with pairwise comparisons where p values were adjusted with the Benjamini-Hockberg method. 36The EMM plots were generated with ggplot2. 37pha was set at 0.05 in all cases.Author keyword (research focus) analysis was presented as descriptive statistics only.

| Summary of characteristics of SOPs by regions
Fifty-five SOPs were included (Figure 1; Table S1).Table 1 shows details of the SOPs grouped by their regions.The South had the highest number of SOPs, followed by the Midwest, West, and lastly, the Northeast.The South also had the highest total funding during the 4 years of interest, while the West had the highest average funding.The median rank based on total funding was highest in the West and lowest in the South.However, the ranks overlapped considerably as shown by the wide range between the highest-ranked and lowestranked SOPs in each region.
A total of 1343 faculty members were included.The South had the most faculty, while the Northeast had the least.We found Scopus profiles/SCIDs for all faculty in the roster for 22 (40%) of the 55 SOPs.
The average percentage found by region ranged from 89.9% to 95.5%, with an overall 93.6% found across all 55 SOPs (Table 1).

| Scholarly activity ratio (SAR)
The raw SAR (±SD) before the pandemic were 0.75 ± 0.08, 0.68 ± 0.16, 0.69 ± 0.17, and 0.74 ± 0.10 for Midwest, Northeast, South, and West, respectively, and 0.75 ± 0.11, 0.62 ± 0.14, 0.67 ± 0.19, and 0.74 ± 0.17 during the pandemic.Overall, the SAR were 0.71 ± 0.14 and 0.70 ± 0.16 before and during the pandemic, respectively.The SAR base model was better without the pandemic-by-region interaction.The pandemic did not have a significant effect on SAR (p > 0.05), but region had a significant effect ( p < 0.05).None of the other predictors were significant, except faculty Review fraction, that had a positive and small effect size (Table 2).More importantly, a pairwise comparison of the estimated marginal means (EMM) based on the final model showed that the pandemic did not affect SAR in any region, and SAR was not different across regions ( p > 0.05; Figure 2A).The model without region produced similar results, except that funding rank was significant with a small effect size (Table S3).

| Scholarly output (SO)
The average raw counts (±SD) of (non-unique) documents published (SO) before the pandemic were 58.5 ± 36.1, 62.9 ± 33.2, 53.0 ± 44.5, and 50.9 ± 23.5 for Midwest, Northeast, South, and West, respectively, while the numbers during the pandemic were 74.The base model of the mixed effects model for SO was better with the pandemic-by-region interaction, which was also statistically significant ( p < 0.05) (Table 2), therefore the interaction effect was included.None of the additional hypothesized predictors had significant effects ( p > 0.05).The 4-year funding rank and the documents Review fraction had significant effects ( p < 0.05) (Table 2).More importantly, a pairwise comparison of the EMM of the SO from the model demonstrated that SO significantly increased only in the Midwest ( p = 0.0136; Figure 2B) and no significant change in any other region.Also, SO was not different across regions ( p > 0.05).This model without region produced similar results (Table S4).

| Research focus and COVID-19 document types
Before the pandemic, 2121 of 2755 unique documents (77.0%) had author keywords in the Scopus files while 2344 of 3001 unique documents (78.2%) had author keywords during the pandemic.Forty and 48 keywords met the occurrence threshold before and during the pandemic periods, respectively.Figure 3 shows the network visualization of the keywords in both periods.Table 3 demonstrates the evolution of keywords during the pandemic relative to before the pandemic (Figure 3A,B, respectively).Among the 38 keywords that "re-emerged," "obesity" was the top upward mover, while "atrial fibrillation" was the top downward mover (Table 3).(with zero, low positive, or low negative rank differences) than with top movers (Table 3).

| DISCUSSION
This study showed an overall increase in the number of documents published (SO) during versus before the pandemic.However, there was no change in the proportion of pharmacy practice faculty (SAR) in each SOP responsible for those publications.Unlike previous studies that showed greater than expected increases in scholarly output during the pandemic, 11,12 overall SO in pharmacy practice was lower than expected, and the magnitude of this pandemic effect varied across regions.
The current study is the first to report objective, bibliometric measures of scholarly outcomes in pharmacy practice due to the pandemic and based on documented faculty within respective SOPs.A recent effort in pharmacy practice was a survey that reported challenges and perceptions of the impact of the pandemic on faculty members on different aspects of their jobs, including research The models' intercepts correspond to the Midwest region before the pandemic (with the numerical variables at 0).Note that response variables (and therefore the predictors' estimates) are in the transformed scales (square and square root for SAR and SO, respectively).Predictors with bold p values had significant effects on the response variables and the effect size (Cohen's [1988] recommendations) of their estimates are denoted as either very small † , small ‡ , or medium § .The optimal model for SAR did not include pandemic X region interactions.Scholarly activity ratio (SAR) is the proportion of faculty members from the school of pharmacy (SOP) who published at least 1 document within each period (before or during the pandemic).Scholarly output (SO) is the number of documents published by faculty from the SOPs.Faculty Review fraction is the fraction of faculty that published reviews, relative to all faculty whose Scopus IDs were found.Documents Review fraction is the fraction of review documents published relative to all documents published.σ 2 is the residual variance (neither explained by the fixed nor the random effects).τ 00 is the variance of the random intercepts.The model's total explanatory power is the conditional R 2 while the part related to the fixed effects alone is the marginal R 2 .Number of observations = 110 (55 SOPs Â 2 periods [before and during the pandemic]).Abbreviations: CI, confidence interval; ICC, intraclass correlation coefficient.
productivity. 9Studies in other fields assessed the pandemic effect based either on surveys or publication data from pre-print servers during some or all the months of 2020 versus 2019 or prior. 11,12First, we analyzed publication records of pharmacy practice faculty within their respective SOPs, rather than self-reports or publication records from selected, field-relevant journals.We also chose a 2-year effect period because the median lag time between submission and online publication dates in pharmacy practice journals is 138 days with an interquartile range of 79-217 days, 38 and the annual publication rate of faculty is less than 1. 33 A longer effect period would be needed later in the future to map the full temporal dynamics of COVID-19 impacts on these outcomes.
Despite previous reports that a small minority of faculty are responsible for the majority of pharmacy practice publications, 21,33,39 to the best of our knowledge, the SAR metric has never been described before.Along with total and/or average SO, SAR is also a relevant, valid, and easy-to-calculate metric to compare schools, departments, or other groupings of faculty.This study is the first to T A B L E 3 Evolution of author-keywords and relationship with "COVIDÀ19."report multi-year SAR at the SOP level; therefore, there is no existing data to compare with this outcome.AACP workforce data shows a slight (<1%) decrease in the number of full-time faculty from 2019 to 2020, with a sharp (13.3%) rise in 2021. 40Our roster data collection in 2021 may or may not have captured this sharp rise depending on how likely new faculty members join AACP and opt into the roster listing.These factors would impact the SAR denominator (total number of faculty).On the other hand, a previous study reported a subinflationary increase in the number of authors per document due to a survival strategy of faculty within an SOP publishing together. 39The stress of the pandemic may have stoked this strategy, thus increasing the numerator (number of scholarly-active faculty) in the SAR.While an interplay of these factors could have neutralized the pandemic effect on the SAR, a simpler explanation is that the consistently scholarly-active faculty who were responsible for publications were resilient and remained active despite the pandemic.
The slight increase in scholarly output (SO) was mostly due to an increase within the Midwest (Figure 2B).Except for having all SOPs representing the region being public schools, no characteristics stood out to explain this region-specific effect (Table 1).However, a plausible explanation is the stringency of COVID-19 lockdown measures in the different regions, and the Midwest had the lowest Oxford COVID-19 Government Response Tracker (OxCGRT) lockdown stringency index, at least until May 2021 (page 23). 41The overall increase in SO was an 8.9% (biennial) increase.Urry and colleagues 20 also used an AACP roster-based approach to study trends in tenure-track pharmacy practice faculty publications, where they reported a 2.5-fold increase in output from 2010 to 2019.Their results represented a 24.9% average biennial increase.This suggests that the 8.9% increase in the current study falls short of the historical trend and expected increase.Thus, the pandemic was associated with a decrease in the expected (biennial) growth in SO.This widespread decrease in output may be important, especially for tenure-track faculty with a higher expectation of SO. 20,42 Like the NIH instituted COVID-19-related flexibilities for grants (eligibility, funding, and report timeline extensions), 43 these results make a case for SO benchmark and tenure-clock adjustments for faculty with probationary periods spanning the pandemic years.
The keyword analysis showed a clear evolution of research focus (Table 3).However, while COVID-19 was the top new research focus, it did not dominate the field as most top-moving keywords were not linked with COVID-19 as the stable ones.While keywords such as "older adults," "critical care," and "mortality" had link strengths of 2-3 with COVID-19, it is surprising that among the well-known risk factors for severe disease and death with COVID-19, including "obesity," "diabetes," and "hypertension,"  3), which was also an important COVID-19-exacerbating factor, 44,45 there was a dearth of COVID-19 research in the context of obesity.
There are a few limitations in this study.First, we assumed that the faculty roster was static during the index years and that faculty were affiliated with one institution and remained in academia throughout the period.This limitation was why we restricted the Note: The threshold of inclusion of an author-keyword was 20 occurrences (including consolidations like SARS-CoV-2, pandemic etc. to COVID-19; see thesaurus in Table S2)."Submerged" keywords met the threshold before (2018-2019), but not during the pandemic (2020-2021); "Emerged" keywords met the threshold during but not before the pandemic, and "Re-emerged" met the threshold before and during the pandemic.Superscripts represent clustering with COVID-19 † and being considered an education-related keyword ‡ .Keywords with a value in the "COVID-19 Link-strength" column are the only ones that co-occurred with COVID-19 in 1 or more documents.Positive (+) rank change refers to movement up the rank and vice versa for negative (À) rank change.
index years to 4 years.Second, being pharmacy practice faculty was based on self-selection of discipline on the roster.However, other disciplines often housed within pharmacy practice departments may have selected their discipline based on their home departments.Third, our outcome measures were limited to articles and reviews.While these peer-reviewed document types constitute the bulk of scholarly works, faculty also publish non-peer-reviewed letters-to-the-editor, editorials, and commentaries. 21,48,49Other scholarly activities include conference abstracts, podium presentations, and book chapters.
Fourth, our SOP selection assumed that more funding predicts more output (Table 2) and, in turn, a higher likelihood and magnitude of pandemic disruption.However, the SOPs' NIH funding may not necessarily reflect funding received by pharmacy practice faculty; the most recent 2015-2019 data showed that pharmacists represented only 10% of principal investigators of NIH's R01 grants to SOPs. 50ditionally, a case could be made for less-funded schools impacting the results because we used a mixed effects model that included the random effect of individual SOPs.Finally, our study assumes that the rate-limiting step for these outcomes lies with the faculty; however, the effects of the pandemic on key pharmacy practice journals' publication processes, capacities, and outputs could have played a yet unknown role.

| CONCLUSION
Pharmacy practice research pivoted toward COVID-19, but these efforts were predominantly in the context of pharmacy education, while salient clinical contexts were inadequately addressed.Scholarlyactive faculty remained resilient in publishing during the pandemic, but their output was lower than expected based on historical trends.
These results make a case for adjusting tenure and promotion evaluation benchmarks and timelines, particularly about scholarly output and especially for tenure-track faculty whose probationary periods spanned these pandemic years in these SOPs.Future studies will address the pandemic effect on faculty at the individual level, based on gender and rank.

2 | METHODS 2 . 1 |
Protocol, data sources, and SOP inclusion and exclusion criteria The High Point University Institutional Review Board (IRB) reviewed the study and considered it exempt.The inclusion criterion for the SOPs was rankings in the National Institute of Health (NIH) Institutional rankings during all four fiscal years (2018-2021).These ranking documents were obtained from the American Association of Colleges of Pharmacy (AACP) (https://www.aacp.org/research/fundedresearch-grant-institutional-rankings).The rankings were based on only NIH grant funding each year (e.g., October 2017 to September 2018 for the 2018 fiscal year).Exclusion criterion was ranking outside the top 50 during all 4 years (i.e., not ranked within the top 50 in at least one of the 4 years) (Figure

A
Python custom function was written to extract the number of faculty who published (≥1 document) during each period and the total number of documents published by all faculty within each SOP.A similar function was used to extract data for only review document types.The extracted data were cross-validated with data from the Scopus "Analyze" function, and the results obtained with Microsoft Excel ® (Microsoft Corporation, Redmond, WA) from five randomly selected SOPs.This Excel analysis was done independently by the second author.The following outcome measures were calculated for each SOP: a. Scholarly activity ratio (SAR): The proportion of faculty members who published at least 1 document during each of the periods (2018-2019 and 2020-2021).The denominator was the total number of faculty members from the roster whose Scopus profiles/IDs were found.b.Scholarly output (SO): The total number of documents published during each of the periods (each document counted once, regardless of the number of co-authors within each SOP corpus).
and the top upward (most positive) and downward (most negative) movers were determined.All documents with (consolidated) COVID-19 in their author keywords or titles (if missing author keywords) were categorized as COVID-19 documents.Both authors independently examined the COVID-19 documents by their titles and/or abstracts and further classified them as pharmacy education-related documents (or not).Consensus was reached on the few documents marked questionable by either authors or classified as education-related by only one author.The working definition of a pharmacy education-related document was one that deals with pharmacy schools, faculty, or pharmacy learners (students, residents, fellows, or pharmacists) within pharmacy schools or other training settings.Of note, this definition is 3 ± 50.4,58.6 ± 27.0, 53.4 ± 44.6, and 54.3 ± 27.0.Overall, the average SO before and during the pandemic were 55.4 ± 36.3 and 60.4 ± 40.8, respectively.The total number of (unique) documents published were 2755 and 3001 before and during the pandemic, respectively, which is an increase of 246 documents (8.9%) over the period before the pandemic.
Scholarly activity and (B) scholarly output in different regions before and during the pandemic.The figures show the estimated marginal means (EMM) and 95% confidence intervals of the linear mixed effect models: Scholarly Activity Ratio $ Pandemic + Region + Funding-rank + Faculty Review fraction + (1jSOP) and Scholarly Output $ Pandemic X Region + Funding-rank + Documents Review fraction + (1jSOP), for A and B, respectively.The asterisk (*) represents pairs in which the pairwise comparison of all levels of Region and Pandemic are significantly different (p < 0.05).Before and during pandemic periods are years 2018-2019 and 2020-2021, respectively.Note that the EMMs are based on all the predictors in the model stated above (also in Table 2), not just on the predictors in the plot.In addition, the response variables are on transformed scales (squared or square root, in A and B, respectively) for scholarly activity ratio (proportion of faculty who published ≥1 documents) and scholarly output (the count of documents published by faculty in a school of pharmacy), in A and B, respectively.F I G U R E 3 Author keyword co-occurrence.Documents published (A) before (2018-2019) and (B) during the pandemic (2020-2021) were subjected to co-occurrence analysis in VOSviewer ® .The occurrence threshold was set to 20 in both cases.The different colors represent clusters of keywords that are most closely related.The sizes of the circles around the keywords represent the total occurrence of the keyword and the thickness of the connecting lines (links) represent the link strength of keyword pairs.See Figure S1 for a spotlight on COVID-19 and linked keywords from Figure 3B.
Characteristics of included schools of pharmacy (SOPs) by regions.
Note: School of pharmacy (SOP) type and affiliation with Academic Health Center data were obtained from the Pharmacy College Application Service (PharmCAS; https://www.pharmcas.org/school-directory/explore-and-compare/public-or-private).Funding data and Carnegie classifications were obtained from the National Institute of Health (NIH) American Association of Colleges of Pharmacy (AACP) Institutional Ranking document for the fiscal years 2018, 2019 (before pandemic), 2020, and 2021 (during pandemic).Regions are determined by the grouping of states in which the SOP is located, according to the census regions from Census.gov.Carnegie classifications are based on 2021 classifications.Because of the low counts, all other classifications (R2, M2, and all Special Focus Four Year) were merged into non-R1.Percentages are either within all 55 SOPs † (rows) or within each region ‡ (columns).education-related documents.All COVID-19 documents published in the American Journal of Pharmaceutical Education and Currents in Pharmacy Teaching and Learning met our education-related document criteria.COVID-19 was linked predominantly with stable keywords T A B L E 2 Linear mixed effects model summaries for scholarly activity ratio (SAR) and scholarly output (SO).