Using database technology to improve STEM student retention : A total quality management approach to early alert and intervention

Article history: Received October 1, 2011 Received in Revised form November, 10, 2011 Accepted 10 November 2011 Available online 11 November 2011 Students at risk of dropping out of Science, Technology, Engineering, and Mathematics (STEM) programs often display signs that indicate they are at risk. A need exists to identify at risk STEM students early and to develop and implement effective intervention strategies that utilize the Total Quality Management (TQM) approach. Most of all, a database system is needed to track this early intervention process, if retention rates are to be improved. To address this need at a small community college in North Carolina, a system was developed and underwent a pilot study in Fall 2009 and Spring 2010. The two pilot groups were compared to the two control groups to identify differences in retention, course credit completion rates, and grade point averages (GPA). The first pilot group displayed no significant differences, while the second pilot group displayed significant differences in most of the areas analyzed in the study, indicating a database system can be used to improve STEM student retention. While the second of the two pilot groups displayed promising results, managerial and logistical issues, such as less than optimal instructor involvement, impeded success were identified. This paper will describe the design, implementation, and the preliminary results of this study and outlines the need for further research that confirms these preliminary findings. © 2012 Growing Science Ltd. All rights reserved.


Introduction
Academic institutions are working hard to retain at risk students in STEM programs.In order for the United States to effectively compete on a global scale, academic institutions need to attract and retain Science, Technology, Engineering, and Mathematics (STEM) students until they graduate and enter the workforce.As of 2004, there were more than 200 federal education programs established and about $2.8 billion was spent in an effort to increase the number of graduates from STEM programs of study (GAO, 2005).This federal investment has resulted in significant research in STEM areas.Recent studies such as Mahoney (2010), Phalke and Lysecky (2010), Subotnik, et al. (2010) and others focus on STEM programs in high schools and colleges and explore areas such as student attitudes and interest in STEM.
There is also significant research in early intervention of at risk students (van Schalkwyk, 2010).Researchers such as Yorke (2001) have presented evidence of the usefulness of early intervention.Most of this research has centered on early intervention among first year students (van Schalkwyk, 2010), but little research explores the use of technology for the Early Alert and management of intervention strategies that keep STEM students in classes and programs of study.A literature search of the Academic Search Premier database for the words "Early Alert" and "STEM" did not identify any related research.Furthermore, other searches for similar terms such as "early intervention" and "STEM" did not reveal any relevant research.One of the challenges academic institutions face is identifying at risk students early, implementing effective intervention strategies, and ensuring those strategies that do work are applied again to other students.Furthermore, students at risk of dropping out of STEM programs often display signs that indicate they are at risk.These signs are first observed by instructors that interact with students in or out of the classroom.These signs may be poor attendance or grades, changes in behavior, difficulty in purchasing course materials, tardiness, and other issues.Unfortunately, instructors often ignore these signs, opting instead to focus on preparing and teaching their courses.As a result, STEM instructors are not equipped to act as counselors nor do they have the time to develop and implement intervention strategies that reduce dropout rates.Therefore, at risk students may not be identified and helped until they have either failed or dropped out of a course or even worse, withdrawn from their program of study.
Furthermore, a need exists to develop and implement effective intervention strategies early in the student's program of study, since studies have shown that student problems that are identified and corrected early, improve a student's chances of success and leads to improvements in retention (Carter, 2007).Also, improving student retention goes beyond addressing outcomes and looks at creating academic cultures where students feel they belong (Zepke & Leach, 2007).As Zepke and Leach point out, if academic institutions are to have a positive effect on retention rates, they must realize outside influences placed on students affect retention in many ways.Developing intervention models that take this understanding into consideration should enable academic institutions to concentrate on areas that actually contribute to retention.Their findings emphasize the need for a system that allows faculty, advisors, and support units to create a culture where students' needs are addressed.Above all, a database system is needed to track and manage this process, to codify intervention strategies that work, and to track student progress throughout a program of study.This database system should help improve retention rates, since it serves as a central point of identification, tracking, and management of students identified at risk of dropping courses and programs.
This design approach views student retention through a TQM approach, where the student's progress through the STEM program is considered the process that is managed and continuously improved.The use of a database system that tracks and manages at risk students early in their academic programs will assist in identifying intervention strategies, tracking student progress from identification until graduation, and allowing follow-up alert submissions to create a continuous improvement cycle needed for TQM success.In order to design and develop such a system that allows for a continuous evaluation and intervention of students in STEM programs or even any type of program, a community college in North Carolina hired a consultant to develop this multiuser system.The goal of this system is to improve retention rates in STEM programs by aggregating Early Alerts and follow-ups, tracking student issues and their intervention strategies, and identifying strategies that lead to improved retention creating a continuous improvement cycle.This TQM approach will allow for the management of student progress in STEM programs from beginning to end of a student's program of study, through the use of a centralized database system.This paper will describe the design, implementation, and the preliminary results of this pilot study.

Background
In Spring 2009, a community college in eastern North Carolina (approximately 1,800 FTE) began to explore the development of an Early Alert System in order to improve retention rates at the College.From the school's Retention Committee, a task force was established to research relevant processes and models.The group identified a small set of guiding criteria for selecting the system.First, the system requires a team-oriented approach that assists students in overcoming academic barriers to success.Initially, the group felt that it was necessary to distinguish academic from nonacademic issues, and to utilize the system to provide support for academic issues.This was in part driven by the lack of trained counselors at the College.All students at the College must take a firstyear-experience (FYE) course within their first six credit hours.The FYE instructor of the section in which a reported student is enrolled would serve as the lead support for the student.The FYE instructor may or may not submit an Early Alert notice on an individual student, yet the FYE instructor would still serve in a coordinating role.Depending on the individual student's circumstances, the FYE instructor may also be assisted by someone from the Tutoring Center, an Academic Counselor, a Financial Aid officer, depending on study needs.If the student is beyond the first semester and has entered an academic program, the student's advisor then assumes the role of lead support for the alert.
The second criterion established by the Early Alert task force was that the system cannot be complicated.It must be user-friendly because faculty are under mounting pressure (e.g.heavy teaching loads, increasing pressure to collect and analyze learning outcomes data, and no increase in salary in four years), and in order to secure their engagement for this essentially volunteer activity, the system must be perceived as a useful tool, and not an impediment on productivity.The third criterion was that the system must be accessible to a school of limited means.There is a significant need for data management solutions (e.g.Early Alert) that are cost-effective and sustainable for colleges with low FTE generation.Data management solutions that require annual license fees and maintenance contracts, that offer remote hosting, and that are relatively sophisticated in their range of functions are by and large out of reach for smaller schools with limited budgets.There is a distinct need for a system that is effective, efficient, and can be hosted locally.
Following review of a number of models in use by other schools, the community college determined that in-house design and implementation would best achieve the criteria that the group had established.A faculty member at a large university in North Carolina, who has experience in developing such software, was consulted with regarding the design and development of a system that would meet the needs of the community college.The College decided to administer a pilot Early Alert program in the Fall of 2009 and Spring of 2010.The task force, in working with the College's IT department, established the parameters and design of the Early Alert software.The designer would have the software ready for deployment by August 2009.A pilot group and a control group of students were also identified for the Fall semester.
The Early Alert software offers users (i.e.faculty and staff) demographic information on a student, a list of courses that the student is enrolled in for that semester, a space to identify the nature of the Alert, and space for the support group to input data on intervention and progress toward resolution of any issues.A list of common Early Alert triggers was identified by the task force (Fig. 1).If a faculty member chooses to issue an Alert, the student record is called up in the software (by name or student identification number).The student's relevant demographic and enrollment data is prepopulated and the faculty member selects the record for the appropriate section in which that student is enrolled.The faculty member checks a box next to any of the applicable triggers and adds relevant comments in an adjoining text box.For each alert, an email is sent to the administrator of the Early Alert program, while a carbon copy is sent to the faculty member submitting the alert.
Once a pilot group had been identified, those instructors who had students from the pilot group in any section they were teaching that semester received a request to participate in the pilot project.The pilot instructors were then offered professional development in using the Early Alert software.By the end of the first week of the semester all pilot instructors were required to submit Early Alert reports on any pilot students were in their classes.In the first pilot semester (Fall 2009), instructors did not receive a request to submit Early Alert reports until the fourth week of the semester.It was realized that this was too late (i.e.students could be "lost" by that point) and the reporting period was moved up to the first week.The purpose for this reporting was to gather data on how well the pilot instructors had been informed and trained in the use of the software.For the purposes of the pilot, if a student had no issues, the faculty member could check a box indicating "no intervention needed at this time".

Fig. 1. Early Alert Triggers
If the need for intervention arose, the Early Alert coordinator would analyze the alert(s) submitted for a specific student, make a determination regarding intervention, and then contact the relevant individuals for support, including the student's FYE instructor or program advisor.The Early Alert coordinator would also make an attempt to contact the student and inform him or her that the College was providing support in order to help the student succeed academically.
In the initial planning of this project, the timeframe for the pilot was two academic semesters (i.e.Fall 2009 and Spring 2010).By the end of Spring 2010, it was evident that a number of barriers to successful expansion to a college-wide Early Alert system persisted.The major challenges that manifested themselves were: 1) Instructor participation; and 2) Response by the Early Alert support team.Regarding instructor participation, by the fourth week of Spring Semester 2010, all pilot instructors were required to submit reports on any pilot students in their classes.By the fifth week of the semester, only 50% of pilot instructors had done so.The majority of non-participants were identified as part-time faculty.This raised a separate issue of how to effectively encourage their engagement in this process.The other concern that emerged after the two-semester pilot was that the response cycle was not fluid.The purpose of the Early Alert system is to provide a support net to catch students before it is too late.Data on response times and communication consistency among the support staff demonstrated that the net still had holes large enough for students to slip through.

Planning and design approach
The initial version of the Early Alert system was designed and developed based on the requirements established by the Early Alert Task Force.Significant input was obtained from future users of the system before the development of the system.In order to reduce costs and allow for rapid development, Microsoft ® Access was chosen as the database management system.Since Access was already available for all faculty and staff, no additional software or hardware requirements were needed.The Early Alert system has been modified several times since the initial version.The most recent version is version 3.5.This version contains many new security features and reports that the previous versions did not.Furthermore, this version added additional data integrity constraints to reduce data input errors and to simplify the alert priority selection process.

Materials
Text books, etc.

Class
Poor Preparation Excessive absences The system was divided into three separate database files.The first of the files is the back-end database, which contains only the tables used to store the Early Alert data and lookup fields used by the system.The lookup fields are used by the system to reduce user input errors and to reduce the time it takes to submit Alerts.Prior to the start of the semester the initial course data for each student is loaded into the system's tables.Preloading the data reduces the time needed by instructors to submit Early Alerts for students in their classes, since the alert submission form is pre-populated with the initial data.This approach also reduces input errors by eliminating the need for instructors to manually enter student names, IDs, and other required data for each alert they submit.Fig. 2 shows the core data tables and their relationships the system uses to store and organize the alert data.The second Access file contains all of the user interface objects and other database objects that make up the faculty version of the system.The faculty version of the system provides access to the alert submission features and other features faculty need in order to access the system.
The alert intervention portion of the system is not available to faculty in this version.The third Access file (advisors' version) contains the intervention portion of the system and all of the available features of the system.Therefore, all of the Access objects (forms, reports, queries, etc.) except for the tables are in this file.Since advisors manage the intervention strategies for all submitted Early Alerts, the file was loaded on the advisors' computers.Fig. 3 displays the main menu for the advisors' version of the system.
The back-end Access file containing the tables was placed on a network server in a folder with write permissions for all users.Links were then created between the back-end file and the faculty and advisor files placed on other computers throughout the college.The Linked Table Manager in Access was used to create the links.Separating the tables from the other objects and creating the links to the tables from the other Access files allows multiuser access to the data stored in the system and prevents record locking errors during multi-access usage.This approach also allows for centralized backup and storage of Early Alert data, since all of the data is found in one file.Once the initial course and student data is entered and the semester begins, faculty throughout the college are able to submit Early Alerts using the alert submission form displayed in Fig. 4. The form is a network enabled input form that allows an instructor to choose from a series of problems a student may be encountering by selecting the appropriate check box and filling in the comment box associated with the selected problem.Since the intent of the system is to alert advisors and support offices of student difficulties as early as possible, faculty were encouraged to submit their Alerts as soon as they observe any student difficulties.Furthermore, they are able to submit follow-ups when students continue to encounter difficulties in a particular class.

Limitations
The three primary limitations to this study were the scope, communication, and technology access.
The following section describes these limitations and their potential impact on the study.

Scope
The Pilot Study took place during the Fall of 2009 and Spring of 2010.The purpose of selecting a pilot group was to ensure the process was applied to a small, select group of students, so the focus remained on managing and revising process and system as opposed to monitoring a large population of students.The group chosen for the pilot was comprised of GOT-D students (i.e.General Occupational Technology -Diploma) who were enrolled in their first semester at the college in at least two so-called gateway courses.Gateway courses are defined as those developmental or General Education courses that have the highest attrition rates.GOT-D are those students whose intent is to enter a Health Science program (e.g.Nursing, Radiography, Respiratory Therapy, etc…), but have not yet completed the prerequisite courses.Therefore, for both semesters that the pilot was administered, the pilot group was small (40 participants in Fall 2009 and 26 in Spring 2010).While this was an efficient number for the pilot, it represented a small percentage of the overall student population at the College.Therefore, this group did not provide a realistic view of Early Alert traffic that would be experienced in a full-scale operation.Likewise, the number of faculty and staff involved in the pilot was a subset of those who would participate in a College-wide implementation, again providing a limited perspective of logistical requirements.

Communication
An effective and efficient communication plan is critical for success in Early Alert implementation.
The chain of communication begins with a faculty member submitting a report, and ultimately includes the Early Alert coordinator, a network of support, and the student.Multiple feedback loops are necessary to ensure that a case is not neglected and that resolution is achieved.Through the course of the pilot implementation, a number of communication issues arose.Some of the challenges included: Faculty not submitting alerts in a timely manner (i.e.submitting a report three weeks into the semester may be too late); Clear roles, responsibilities, and expectations of the Early Alert coordinator must be established (e.g. who notifies the student that an alert has been submitted; who continues to monitor support until resolution is achieved); Communication with the College community at-large (e.g. who provides faculty and staff professional development; who communicates policies and procedures of the Early Alert process).

Technological access
Part-time faculty comprises a significant percentage of those participating in the pilot and teaching across all programs at the college.As such, part-time faculty are an integral component of the Early Alert system; without their input, a large portion of those students who are struggling academically go unnoticed.However, the Instructional Technologies Department believed that the most secure way to access the software is via the College's intranet, meaning that only those full-time faculty and staff who work full time on campus could access the software.Conversely, part-time faculty who wished to submit an Early Alert report could only do so if they came to campus.For those Distance Learning instructors who could not come to campus, the only viable solution was to submit an alert by emailing the Early Alert coordinator.If the software precluded nearly half of all instructors at the College, it significantly jeopardizes the success of the program, and the students who should be receiving its services.

Results
The hypothesis for the study was that: By utilizing specialized Early Alert software to monitor and develop response plans for students experiencing academic challenges, the College retention rate would increase.Of the 40 students randomly chosen during the fall 2009 semester to participate in the pilot study, 28 (70.0%)enrolled in courses in the spring 2010.The comparison group of 40 students resulted in 31 (77.5%)enrolled students during the same semester as the pilot group.The small difference in retention between the pilot group and the comparison group is not significant (Chi-Square test, P-Value=.4459,α=.05).The comparison of the grade point average of the two groups was also performed during the study.Similar to the retention comparison, the difference between the grade point averages of the two groups was not significant (T-Test, P-Value=.6707,α=.05) (EA Report FA09, 2010).Table 1 displays the grade point averages of the two groups and their standard deviations.The number of course successful completions was also compared between the Early Alert pilot group and the comparison group.Successful completion was defined as achieving a grade of A, B, or C (EA Report FA09, 2010).The number of withdrawals was also compared between the two groups.Here as well, the results show there is no significant difference between the two groups at the end of the semester for successes (Chi-Square test, P-Value=.7804,α=.05) or for withdrawals (P-Value=.6060,α=.05) (EA Report FA09).See Table 2 below.
The EA Report SP10 (2010) outlined much better results than the previous semester.The Spring 2010 pilot group contained 26 students, while the control group contained 59 students.The success rate for the pilot group was 80%, while the success rate was for the control group was 61%.The percentage of credits completed in comparison to the credit attempts produced similar positive results for the pilot group with an 82.4% credit completion rate, while the control group had a 66% credit completion rate.Furthermore, the 15 withdrawals for the pilot group were significantly smaller than the 52 withdrawals of the control group.Also, the 2.51 average GPA for the pilot group was higher than the 2.27 average GPA for the control group.These above results outlined in the EA Report SP10 indicate the Early Alert tool is helping retention, GPA, and course credit completion.
Just as the Fall 2009 pilot identified issues that required attention, the Spring 2010 pilot generated similar issues that need attention.One of these issues is the need for more instructor involvement in the Early Alert initiative (EA Report SP10, 2010).For example, the EA Report SP10 identified significant differences in reporting by instructors for a particular student, although the student had withdrawn from the courses.Another issue identified in the report was the poor communication and follow-up with students who received an Early Alert.Some students that could not be reached immediately after the initiation of the Early Alert were never contacted during the semester and as a result no intervention took place.These issues were logistical or administrative issues and were not attributed to the design of the Early Alert software.

Conclusions and recommendations
Although the results of the Fall 2009 pilot did not produce significant differences between the pilot group and the control group, the Spring 2010 pilot produced promising results, since significant differences appeared between the pilot and control group.Despite the managerial issues identified in the Spring 2010 pilot, there were notable differences in retention, credit completion, and GPA between the groups, indicating the Early Alert software can lead to improvements in retention among students to include STEM students.Since STEM students are faced with challenging coursework and are likely to experience the same types of issues as other students in different programs of study, the use of a database system that can initiate and track Early Alerts for students identified as at risk provides a promising solution to a common problem students and academic institutions face.
Future improvements to the and more administrative and faculty support for the Early Alert Process is likely to produce more positive results than identified in the Spring 2010 pilot.The Early Alert software is only one tool in the process, but was able to produce promising results since it was designed using a TQM continuous improvement approach where student problems were identified and reported early, intervention strategies performed, and follow-ups on those intervention strategies were performed.The intervention strategies that proved to be successful can then be reused or improved upon to add more value to the intervention process and ultimately improve retention among students.
It is important to keep in mind that student retention requires a proactive approach that identifies student problems as early as possible and then tracks the steps and procedures that immediately take place after the initial Early Alert.Furthermore, since this critical problem and intervention data is centralized in one system, mining of this critical data is likely to lead to a better understanding of the problems students encounter and what works to retain students that are at risk.Most of all, the design of a process where student problems are identified, addressed, tracked, and changes to processes and policies that can help reduce these problems are made is essential to improving student retention.The results of these pilot studies have shown that an information system is required to manage this process that is expected to lead to continuous improvement in academic processes and strategies that lead to improvement in student retention.
The Early Alert pilot studies identified areas of improvement and the need for more research on the use of database systems to manage student problems and the strategies employed to address them.Therefore, more research is needed to determine if systems similar to the Early Alert system used in these studies can produce similar or better results.Although the Spring 2010 results produced positive results, studies that last several years are likely to produce more accurate assessments of this approach to STEM and other student retention.Furthermore, similar studies that utilize larger populations are needed.

Fig. 2 .Fig. 4 .
Fig. 2. Partial view of the core tables Fig. 3. Main menu of the advisors' version