Tracking and Assessing Undergraduate Research Campus-wide: Demographics, Academic Success, and Postgraduation Plans

: Evaluation must occur at the university level to understand the full impact of undergraduate research (UR). UR assessment is often only completed at the individual program level because of limited technology, time, and/or resources. At our large research institution, we have been documenting a wide variety of research experiences annually since the 2009–2010 academic year through an online portal. With our institutional research team and campus partners, we created interactive dashboards that display involvement in UR by semester and academic year. Here we compile data on students involved in UR compared to the university population as a whole. Consistent trends from this yearly data have shown that non-STEM (science, technology, engineering, and mathematics) students, transfer students, and part-time students are less involved in research. However, underrepresented and first-generation involvement tends to trend consistently with the university population, likely because of a wide variety of focused programming. Despite many interventions aimed at engaging students in their first three years, data show that researchers remain mostly seniors. Students are also tracked to graduation and beyond, providing a unique evaluation of UR. Grade point averages and graduation rates tend to be higher for student researchers. Time to degree is similar between researchers and nonresearchers. Students are tracked into graduate school as well and on average have an almost 50% increase in matriculation compared to nonresearchers. There are still gaps in this university-level knowledge, but this portal helps clarify campus-wide involvement and opportunities for enhancement, while serving as a comparison data set and a model system for other universities.


Background
In laboratories, research stations, and libraries, faculty and students are collaborating to create knowledge and models that advance their fields; these partnerships are the hallmark of traditional undergraduate research (UR) activity.While it can be hard to determine how many faculty and students are participating on any campus, much research has been done to discover the impact of these Schneider, Kuperman, Watts, Barulich, and Campbell Journal of the Scholarship of Teaching and Learning, Vol.21, No. 1, April 2021.josotl.indiana.eduactivities on undergraduates.There is a rich body of literature documenting the benefits of student participation in UR, including learning gains in hard and soft skills and improvement on general student success metrics.
Involvement in UR has been shown to expand students' skills across a variety of measures, including observing and collecting data, acquiring information independently, analyzing literature critically, and communicating results (e.g., Kardash, 2000;Bauer & Bennet, 2003;Junge, Quinones, Kakietek, Teodorescu, & Marsteller, 2010;Lopatto, 2007).Studies have also shown that UR students report an increased tolerance for obstacles and improved ability to work independently (Bauer & Bennett, 2003;Lopatto, 2007).Put more broadly, UR has been shown to increase students' confidence in these skill sets (e.g., Sadler, Burgin, McKinney, & Ponjuan, 2010).However, these skills and attitude changes are not the only benefits of research experiences.
UR students who participate in structured programs have also been shown to develop strong interpersonal networks with peers and mentors (i.e., faculty, graduate and other undergraduate students) that benefit them as individuals and as developing scholars.These networks help make UR program participants, particularly those from underrepresented minority groups, more likely to apply to graduate school and more competitive in the graduate school admissions process (Linn, Palmer, Baranger, Gerard, & Stone, 2015;Ovink & Veazey, 2011;Thompson, Conaway, & Dolan, 2016).Additionally, structured research programs have been shown to improve student success metrics such as retention, persistence toward graduation, and grade point averages (GPAs;NSSE, 2007;Nagda, Gregerman, Jonides, von Hippel, & Lerner, 1998;Schneider, Bickel, A., & Morrison-Shetlar, 2015;Schneider, Tripp, Nair, Straney, & Lancey, in press;Sell, Naginey, & Stanton, 2018).Several studies have shown that these effects are more pronounced among underrepresented student populations (e.g., Linn et al., 2015;Schneider et al., in press).
Less well documented in the literature is just how many undergraduates participate in UR activities nationally and at individual institutions (Blockus, 2012;Webber, Fechheimer, & Kleiber 2012;Wilson et al., 2012).Since UR is such a valuable experience, it is necessary to understand university-level participation.Getting the "count" of student participation is difficult for most institutions because of the lack of a centralized office, staff time, technical capacity, and convenient data sets (e.g., faculty annual reports, work-study employee data; Blockus, 2012;Wilson et al., 2012).These issues make it especially difficult to track involvement at large universities.There are a wide variety of approaches, but no "silver bullet" solution that works for all institutions.
One commonly used tracking tool to get a campus-wide assessment of UR activities is the National Survey on Student Engagement (NSSE), which tracks "research with faculty" as an educational high impact practice (HIP) on both the first-year and senior surveys.In 2019, national data showed that 5% of first-year students participated in research with a faculty member, while 22% of graduating seniors reported having participated (NSSE, 2019).However, this self-reported measurement is taken from a very broad statement that asks students if they "worked with a faculty member on a research project," which respondents may not interpret as requiring an original contribution to the field (Wilson et al., 2012) and may include classroom projects.Other campuses, for example, have used enrollment in UR courses to track campus-wide involvement, which is a valuable measure (e.g., Fechheimer, Webber, & Kleiber, 2011; Webber et al., 2012).Campus-specific surveys are also a common tool (e.g., Berkes, 2008).While allowing for consistency among tracked experiences, these approaches can exclude participants or count some students more than once.
In Schneider et al. (2016), we reviewed a model for overcoming one of the central challenges that many campuses face, getting a unique count, which allowed us to see the unique count of students involved in UR annually (i.e., see Blockus, 2012) and went beyond this to look at a wide variety of student involvement data in one centralized format.Many models document students multiple times annually.For example, when a student participates in a campus-wide poster symposium one spring, Schneider, Kuperman, Watts, Barulich, and Campbell Journal of the Scholarship of Teaching and Learning, Vol.21, No. 1, April 2021.josotl.indiana.eduwhile enrolled in honors thesis credit that semester, most campuses might have to count the student twice.Our model allows us to count each individual student once per year, or semester, as needed.This creates clearer, more reliable demographic and enrollment trends in our data and prevents duplication.

Centralizing UR Tracking
At the University of Central Florida (UCF), obtaining an accurate count of those involved in research was a challenge.UCF is a recently designated Hispanic-Serving Institution (HSI) with a Carnegie Classification of very high research activity.This metropolitan campus has a mix of traditional first time in college (FTIC), transfer, and online-only students.
The UCF Office of Undergraduate Research (OUR) and our Institutional Knowledge Management (IKM) office began a collaboration in 2013 to collect and share information on UR activity in a database and interactive dashboard.This dashboard tracks student and faculty involvement in four broad buckets of research activity: (1) structured research programs (e.g., honors theses, McNair Scholars); (2) research professional-development opportunities (e.g., campus showcase, travel funding); (3) independent research credit completed with a faculty member; (4) paid opportunities from external research-focused grants (Schneider et. al., 2016).This approach creates a highly reliable data set but does have limitations.Those who "volunteer" as research assistants or use federal work-study funds are not counted.This is communicated to faculty and department administrators, who are encouraged to use a general "Directed Independent Research" course for students not in documented programs.
The resulting dashboard allows campus stakeholders to draw comparisons between the population of tracked UR students and the university's larger undergraduate population, which facilitates strategic program planning at the department, college, and campus levels.Between 1,500 and 1,850 students are documented through the dashboard annually.After creating the first database in 2013, OUR focused on first-generation, underrepresented, and transfer students and found that that the percentage of students involved in UR who were first generation very closely matched the percentage of first-generation students in the general student body, but that transfer students were not involved at the same rates as FTIC students (Schneider et al., 2016).However, at that time the data provided just a snapshot of several semesters.Years later, OUR continues to document student involvement demographics and additionally includes elements of student success through the dashboard.
Here we share longer term trends in our campus-wide data set.This article outlines further population-level findings, focusing on several key student demographics and student success variables.Specifically, we ask several questions about UR in a broader context: 1. What are the consistent trends in demographics of UR students compared to the campus as a whole?How can these trends support strategic planning?This includes demographics such enrollment status, transfer status, and ethnicity.2. How do student success variables track for UR students compared to undergraduate students as a whole?Here we look at graduation rate, movement into graduate school, and a few other student success indicators that are often highly valued by institutions.

Method
Collecting data on student research involvement requires campus-wide collaboration.This includes a strong partnership between OUR and IR.Additionally, OUR partners with campus stakeholders who The number of programs and opportunities tracked each year varies.In Schneider et al. (2016), 19 unique programs were documented.In the 2018-2019 academic year, however, only 14 different programs or courses were tracked.This variation is linked to internal and external funding.Inclusion criteria for this database are broad, as we seek to document all students engaged in research and scholarship creation outside of the traditional classroom when possible.Valuable partners for this project include our Honors College, the sponsored research office, STEM (Science, Technology, Engineering, and Mathematics) initiatives, and other grant-funded program offices (e.g., McNair Scholars).However, it should be noted that classroom UR experiences (CUREs) are not documented through this portal; UCF courses designated as research intensive (RI) are tracked separately through a more recent campus initiative to document a variety of HIP courses.
OUR collects data from our partners for each full year (summer, fall, and spring terms) of involvement at the beginning of the following summer term.Thus, one drawback of this model is data often lag by a year or more before they are widely available to campus administrators and staff.The process begins at the end of each spring semester, when OUR provides campus partners with a guide for cataloging data.The data set requires six inputs for each unique entry: year start, yearend, student campus ID, faculty campus ID, research term (semester), and the unique research program code.Compiled data are provided by our campus partners to OUR to organize and load to a portal provided by IKM via a custom-built process in PeopleSoft Campus Solutions (UCF's software that houses student information).

Data Validation and Processing
When all the data are uploaded, OUR and IKM partner to validate each entry through an in-depth cleaning process.IKM checks for inconsistencies, including research programs that have not entered any data for that year (i.e., missing inputs), faculty who are not associated with a college and/or department (or are not faculty, but are instead staff), and students who were not enrolled as undergraduates that semester (including recent graduates, or those enrolled in graduate classes).These inconsistencies are communicated to OUR via error validation processing reports.OUR reconciles all errors by fixing or deleting entries according to strict protocols and corrects errors in PeopleSoft (e.g., students not enrolled in a summer term who are enrolled as undergraduates the following fall are included in the data set).
Once the data are cleaned, IKM extracts the data from PeopleSoft to a data warehouse, which combines the data with housing, student enrollment, and demographic information.This process provides student enrollment information, student and faculty demographics, faculty primary department, and faculty tenure status.IKM then appends this final data set to the historical data set.This dashboard is one of several HIP dashboards provided by the university.After the UR dashboard was completed, others were added, including capstone, study abroad, internships, learning communities, and HIP-designated courses.

Dashboards
When the process is finalized, the dashboard built with SAS Visual Analytics (VA) is updated to display the new data.The VA dashboard is accessible by a select group of users on campus and contains the following six separate sections that display data from the past five academic years.Most sections can be filtered by college, department or major, program, and term.Currently, UCF has the following dashboards available to the internal community: Student involvement.This section contains the total number of students who participated in UR (see Figure 1).It shows the unique number of students involved in the past five academic years.Many students are involved in multiple programs, but this count shows each student only once, allowing for a clear picture of total involvement.However, the data can also be sorted by program.Enrollment data.This section includes academic level (e.g., 1st-year, senior), student type (e.g., transfer, postbaccalaureate), full-time/part-time, and college comparisons to all UCF students for the specified academic term (e.g., fall 2018).
Performance.This section contains comparisons of the average time to degree and the average GPA at graduation compared to all UCF students over the past 5 years.The tab also includes UR students' 4-, 5-, and 6-year graduation rates.Finally, the percentage of bachelor's degrees awarded to UR students is also displayed in this section.
Postgraduation plans.This section displays the percentage of students who continued their education after receiving a bachelor's degree (compared to all UCF students).Additional details include their highest postgraduate enrollment career (second degree seeking undergraduate, graduate, or doctoral program), their enrollment time (in years) in the new program after completing their UCF bachelor's degree, and the institution they enrolled at (broken out by in state, out of state, and Ivy League).Postgraduation plan data is sourced from the National Student Clearinghouse StudentTracker database.This database includes enrollment and degree information for 99% of all students in public and private U.S. colleges and universities.
Faculty involvement.Although no data from this dashboard are included here, it is relevant to mention that this section includes faculty involvement and the average number of students per faculty mentor.It also shows faculty demographic details including race, gender, tenure status, and faculty rank.

Additional Resources
In addition to the VA dashboard, OUR is also able to run more detailed reports through a specialized data and information portal to support campus operations.From this portal, OUR can pull the following: (1) faculty capacity, to look at how many faculty per department are mentoring students, (2) individual faculty mentor activity reports provided to faculty upon request, (3) detailed report of research involvement for a specific UR program or academic department, and (4) email addresses of individuals who are listed as faculty mentors in UR to be used for recruitment and communications.

Current Study
Using the above-described dashboard, we could take a close look at all student researchers.Our dashboard reports semester-level data for most cases.However, for this purpose, academic year data were examined to understand involvement over the past 5 years and beyond.Here we focus on (1) general student involvement, compiled from the student demographic and enrollment data dashboards described above, and (2) student success indicators, compiled from the performance and postgraduation plan dashboards.

Findings and Discussion
Through the UCF database we explored trends in participation, including demographics and student success indicators, and compared UR populations (i.e., students involved in UR) and the UCF undergraduate population as a whole (the university population includes UR populations).

Student Involvement
STEM.Students who were involved in UR were more likely to be in STEM majors than the undergraduate population as a whole (Table 1).For STEM we included physical and life sciences but not the social sciences.This is a consistent trend over all 5 years: On average, twice as many UR students were STEM majors compared to the UCF population as a whole (UR: 49.42% vs. UCF: 23.46%).One should note, however, that over 50% of students involved in UR at UCF are not enrolled in the traditional STEM degree programs (i.e., they are in the social sciences, humanities, arts, and business).Therefore, although UR involvement is skewed toward STEM, student research is very common beyond the STEM disciplines.Our results show a similar trend to other studies of undergraduate student participation in UR in STEM and non-STEM fields (e.g., Berkes, 2008;Fechheimer et al., 2011).Note.STEM = Science, Technology, Engineering, and Mathematics; UCF = University of Central Florida and refers to the total undergraduate population; UR = undergraduate research and refers to the population of students engaged in UR.
a Students who transferred from our state college system with an associate of arts or sciences degree (A.A. or A.S.).b This includes anyone with over 90 credit hours, and many students come in with credits and/or stay an additional year.
One interesting note in student involvement is how the data ebb and flow as a result of programs being funded by soft money, such as federal grants.For example, participation in programs funded by National Science Foundation grants tracked in the database decreased by 28% between the 2015-2016 and 2019-2020 academic years.
Transfers.Transfer students who matriculate to UCF with an associate of arts or sciences (A.A. or A.S.) degree from our statewide college system make up almost half of our student body and are less involved in UR compared to the FTIC population (Table 1).There are likely several reasons for this trend.On our campus, the transfer population typically reported working part-or full-time at much higher rates than our incoming FTIC students, in our incoming student survey (Lancey, 2020).For example, in 2019-2020, 21% of traditional FTIC students planned to work part-time and 3.1% full-time.In contrast, 30.6% and 20.6% of transfer students planned to work part-time and full-time, respectively.Additionally, transfer students have less time to learn about opportunities and often struggle with the transition from a college to university setting, further limiting their time (known as transfer shock).
Underrepresented students: First generation and minorities.For students who self-report as first generation at our institution, the percentage involved in research is typically equivalent to that of the total population (see Table 1).We see a similar trend with our Hispanic students (see Table 2) but a slightly larger gap between the percentage of Black students in the general population and the percentage engaged in UR (Table 2), which highlights an area for future focus.It is unclear what barriers may exist for this population that do not seem to be present for Hispanic or first-generation students.
UCF has many outreach programs focused on promoting UR to diverse populations, including through external funding and internal inclusive introduction to research programs.These programs reduce the barriers to research, and evidence indicates that they are producing more equitable opportunities on our campus.We see this in the parity between first-generation and Hispanic student involvement compared to the general student population.Berkes (2008) looked at ethnicity in the University of California, Berkeley UR populations, comparing STEM and non-STEM majors, rather than the student body as a whole.Her report does document that Latino/Latina students and African American students participate in UR at lower rates than other underrepresented groups.Note.UCF = University of Central Florida and refers to the total undergraduate population; UR = undergraduate research and refers to the population of students engaged in UR. a Native American, Pacific Islander, multiracial.b White, Asian, international, not specified.
Student Success Indicators GPA.The UR population's cumulative GPA at graduation was only slightly higher than that of the UCF undergraduate population.This varied by year, but on average was 0.2 points above the GPA of other graduating students (Table 3).This trend appears consistent with Fechheimer et al., 2011, which found that GPA was slightly higher for UR students, and that GPA increased the longer students were involved in research.We have not looked at this data regarding how many "times" students appeared in our database, but that would warrant further investigation to see if extended involvement impacted GPA (and other factors).
Graduation data: Rates and time to degree.The average years to degree completion show a very small difference between the UCF and the UR population (Table 3).However, the FTIC 4-year graduation rate, a key metric for many institutions, was consistently higher for UR students (Table 4).We only have 4 years of this data.Our first tracked cohort started in 2011-2012.UR students have not graduated in fewer semesters then the university undergraduate population.However, conducting research does not extend students' time to degree.We could not find any other articles reporting on this, but it is an important result, and it would be interesting to know if other campuses see similar trends.
We were only able to look at graduation rates for FTIC cohorts because of the complication of other groups of students matriculating in at different times.UR has been considered an effective tool to increase retention and graduation rates (e.g., Nagda et al., 1998;Locks & Gregerman, 2008;Schneider et al., in press).However, this data set is the first of its kind to look at institution-wide involvement at this scale.
Postgraduation plans.Not surprisingly, students who engaged in UR continued their education at a higher rate compared to the university undergraduate population, although matriculation into postgraduate education does not always happen immediately (see Table 3).These rates include all forms of postgraduation education, including students seeking a second bachelor's degree (i.e., not just graduate and professional degrees).Several studies have documented that students engaged in research feel more prepared and ready for graduate education, often making plans to attend graduate school (e.g., Lopatto, 2004Lopatto, , 2007;;Carpi, Ronan, Falcone., & Lents, 2007).Yet tracking of these students is rare.Note.FTIC = First time in college; UCF = University of Central Florida and refers to the total undergraduate population; UR = undergraduate research and refers to the population of students engaged in UR.

Strategic planning.
The dashboard provides the campus with clear information necessary to make plans to close consistent gaps in student involvement.Thus, these data have been important for our own campus strategic planning.Additionally, trend data can be used for internal and external funding proposals.
One example that demonstrates the dashboard's utility focuses on bringing awareness to the transfer student engagement gap (Table 1).With a goal of reducing the gap, OUR continues to work with our transfer institutes to increase the pipeline and encourages new transfer students to connect early with the office.Additionally, new programming for transfer students has been developed through

Conclusions
University-level data is necessary to understand the full impact of UR.However, we are not sure how UCF compares to other universities because this type of university-level data does not exist beyond a few isolated case studies (e.g., Berke, 2008;Fechheimer et al., 2011).It would be powerful to compare the full results of our single institution with other peer institutions, to better understand our data set's meaning.We have set forth some benchmarks for comparing and contrasting in the future.Additionally, we have outlined areas of concern as new programming and structures are developed to lower the barriers to student involvement.
This work adds to a growing body of literature assessing the benefits of engagement in research including traditional student success metrics (e.g., graduation rates and GPA) and matriculation rates into graduate school.In state universities this is often linked to performancebased metrics that are linked to funding.For example, in Florida, state universities are held to performance-based funding metrics, including "percent of bachelor's graduates employed (earning $25,000+) or continuing their education" (State University System of Florida, Board of Governors, 2019).This database can show what some student success indicators look like at the university level when numerous programs and opportunities are combined.University-level data highlight the strategic importance of sustained funding for UR programming.To meet strategic and performancebased metrics, data can help drive support and resources to grow internal programming and sustain programs initially developed with external funding.
It is important to note that our data include experiences across the spectrum of student involvement.For example, the data include research assistants paid on grants who had menial tasks (i.e., maintaining stocks, data entry) and, on the other end of the spectrum, students who developed high-level honors theses or published in peer-reviewed publications.This difference in experience makes the data noisy when looking at student success metrics and matriculation in graduate school.
In conclusion, exploring the benefits and value of UR requires campuses to more systemically address how to "count" participating students.This count data allow an institution to benchmark their successes, find gaps in student involvement, and track basic student success indicators.

Figure 1 .
Figure 1.Example screenshot of student involvement dashboard.Student demographics.This section includes race, gender, age, and first-generation status counts and percentages for UR students compared to all UCF students for the specified academic term (e.g., fall 2018; Figure2).