Moving Beyond Smile Sheets: A Case Study on the Evaluation and Moving Beyond Smile Sheets: A Case Study on the Evaluation and Iterative Improvement of an Online Faculty Development Program Iterative Improvement of an Online Faculty Development Program

Institutions of higher education are struggling to meet the growing demand for online courses and programs, partly because many faculty lack experience teaching online. The eCampus Quality Instruction Program (eQIP) is an online faculty development program developed to train faculty to design and teach fully online courses. The purpose of this article is to describe the eQIP (one institution’s multipronged approach to online faculty development), with a specific focus on how the overall success of the program is evaluated using surveys, analytics, and social network analysis. Reflections and implications for improving practice are discussed


Introduction
As online enrollments grew over the last decade, Boise State University was confronted with a common problem in higher education: What is the best way to design and develop highquality online courses and support faculty as they teach?High-quality online courses begin and end with high quality faculty (Dunlap, 2005;Wilson, Ludwig-Hardman, Thornam, & Dunlap, 2004); most faculty, though, have never taken nor taught a course online before (see Bacow, Bowen, Guthrie, Lack, & Long, 2012;Tipple, 2010;Wolf, 2013).To succeed in the online classroom, faculty must possess, among other things, pedagogical, facilitative, instructional, social, managerial, assessment, and technical competencies (Baran, Correia, & Thompson, 2011).Thus, the challenge institutions face is how to develop these competencies in faculty so that they are able to develop and teach high quality online courses.Boise State created the eCampus Quality Instruction Program (eQIP), an online faculty development program, to address this challenge.The purpose of this article is to describe the eQIP and the multipronged approach used to evaluate the eQIP.

Literature Review
Institutions across the country have used multiple approaches to address the problem of how to develop and teach high quality online courses.On one end of the continuum are universities that essentially leave it up to faculty to decide what courses they teach online and how they develop them.On the other end of the continuum are institutions, often with a forprofit business model, that have adopted a centralized "enterprise" model in which all online course development and training takes place (Herman, 2012;Lowenthal & White, 2009).Most institutions lie somewhere in between these two extremes, offering some type of support for online course development and teaching.For instance, the University of Colorado Denver takes a decentralized approach in which faculty, for the most part, decide what they want to teach online, but may be supported by CU Online (a centralized unit for faculty support) when needed; in addition to just-in-time instructional design support, CU Online also offers regular workshops and annual events to support faculty developing online courses (Lowenthal & Thomas, 2010b).On the other hand, faculty at the University of Central Florida complete a cMOOC (connectivist Mass Open Online Course) that prepares them to teach online (Moskal, Thompson, & Futch, 2015).Institutions like the University of the District of Columbia fully support faculty to develop online courses and complete a Quality Matters Standards and Peer Review for online courses to improve the quality of their online courses (Britto, Ford, & Wise, 2014; see Meyer, 2013, p.95-96 for a list of faculty development programs).There are typically three components common to models of support for online learning: (1) using instructional designers to design or support faculty as they design courses, (2) training faculty to teach online, and (3) using some type of quality control system to evaluate and in turn improve the quality of online courses (e.g., Quality Matters or the Online Consortium's 5 Pillars quality framework).
Building upon what others have already done, Boise State implemented a program called the eCampus Quality Instruction Program (eQIP) to improve the design, development, and delivery of online courses.The following paper describes the program Boise State developed and the mixed methods approach taken to evaluate and improve the program over time.

Background: The eCampus Quality Instruction Program (eQIP)
Boise State University began offering courses online in 1989.Over the years, mirroring national trends, enrollments grew.Due to this growth, as well as a desire to develop more online programs, the eCampus Center created the eCampus Quality Instruction Program (eQIP) in the fall of 2012 to improve how courses are designed and taught online at Boise State.The eQIP consists of three distinct components: • Course Design: A course design seminar; • Quality Assurance: A Quality Matters Peer Review, and • Teacher Training: A teaching online seminar.
Each component is described briefly in the following paragraphs.

Course Design Seminar
To teach faculty how to design online courses and to assist them in the development of these courses, eCampus developed a 12-week online course design seminar called the eCampus Course Design and Development Seminar or eCD2S for short.Most faculty do not have the experience or skillset to design online courses (e.g.Bailey & Card, 2009;Baran, Correia, & Thompson, 2011;Marek, 2009;Moar, 2006;Schrum, Burbank, Engle, Chambers, & Glassett, 2005); faculty are content experts, many of whom have limited coursework or structured experiences in teaching and learning online (e.g., Sellani & Harrington, 2002).The eCD2S is designed specifically for faculty with no previous experience with online course design.The seminar has been iteratively improved over time (see Chen, 2014).The seminar currently consists of two distinct phases: Over time, it became clear that even after faculty completed the seminar, many desired and benefited from additional support when designing future online courses.As a result, a shorter, condensed version of the course design seminar called the eCampus Course Development Phase or eCD was created.The eCD was designed for faculty who have completed the 12-week course design seminar (i.e., eCD2S) at Boise State.The eCD essentially focuses only on the development phase (i.e., Phase 2) of the original 12-week seminar.In the eCD, during 8 weeks, faculty work one-on-one with an instructional designer, while collaborating with colleagues, to develop a turnkey-ready online course (which we refer to as a "Master Course").Collaborating with colleagues and instructional designers helps improve the quality of the online courses faculty design, even with faculty who already have experience designing online courses (Lowenthal & White, 2009).

Quality Matters Peer Review
After faculty develop an online course (in either the eCD2S or the eCD) and teach it for one semester, the online course is peer reviewed using the Quality Matters' framework.Quality Matters (QM) is a nationally recognized quality assurance program used to improve the design of online courses (MarylandOnline, 2011).Following QM's peer-review process, each course developed in the course design seminars is peer-reviewed by three trained QM faculty reviewers (Carter-Cram, 2014; see also MarylandOnline, 2011, for more on the QM peer-review model).After a course is peer reviewed, the faculty member who originally developed the course updates the course based upon the feedback received through the QM peer-review process.The updated course then becomes a "master" online course-that is, one that can be reused and taught by any Boise State faculty (see Lowenthal & White, 2009, for more on master courses and centralized models of course development).

Teaching Online Seminar
Teaching online is different than teaching face-to-face (Stone & Perumean-Chaney, 2011;Xu & Morris, 2007).Without proper training, experienced face-to-face faculty often struggle when moving to an online environment (Natriello, 2005).While some universities offer a faceto-face workshop on how to teach online, Boise State decided that it would be more authentic and convenient to train faculty to teach online in a fully online course.The teaching online seminar, called the eCampus Teaching Online Seminar or eTOS, is a 6-week online seminar facilitated by full-time faculty (Carter-Cram, 2014)-not instructional designers.In this seminar, faculty learn how to teach online, what it feels like to be a part of an online faculty community (Brooks, 2010;Schrum et al., 2005), how to transfer evidence-based pedagogical practices (Bailey & Card, 2009), and finally what it is like to be an online student (Fein & Logan, 2003).This seminar was designed both for faculty who have never taught online as well as faculty with years of experience teaching online (see Table 2).
While the eQIP has been offered for a few years now, this case study focuses on the faculty and evaluation strategies used during 2014.In 2014, 51 faculty took part in the course design seminars, and 65 faculty took part in the teaching online seminars (see Table 1).

Method
Meyer (2013) reviewed a series of published literature in online faculty development and criticized seven methodological flaws of the selected research.One of the flaws was that outcome measures of past faculty development research overly relies on faculty persons' "honesty and self-understanding" (p.104) of the training.Unlike many faculty development programs that simply depend on a short post-workshop survey-sometimes pejoratively referred to as a "smile" sheet-we treat eQIP seminars more like for-credit online courses than snapshot workshops and therefore use a mixed-method approach of collecting data to find out what is working and what is not working in the program (Creswell, 2008;Creswell & Plano, 2007;Greene, 2007).Following standard practices of program evaluation (Stufflebeam & Shinkfield, 2007), empirical data were collected from participants in the course design seminar (i.e.eCD2S) and the teaching online seminar (i.e., eTOS) to evaluate the success of the eQIP and to make informed improvements of the program.This study was both evaluative and exploratory in nature.The following questions guided our inquiry: 1. How did faculty participate in the cCD2S and eTOS? 2. What were faculty perceptions of the cCD2S and eTOS? 3. How did faculty skills and dispositions change over time? 4. What concerns did faculty have with eCD2S and eTOS?
More specifically, at regular checkpoints, data from surveys (e.g., module surveys, entrance and exit surveys), learning analytics, and social network analysis was collected and analyzed to better understand faculty participation, perceptions, skills and dispositions, concerns, and thus, the overall effectiveness of the eQIP (see Table 2 for an overview of the data collected).The quantitative data collected from the surveys was analyzed using descriptive statistics to reveal faculty participation, perceptions, dispositions, and concerns; the qualitative data were coded in Nvivo 10 using an open-coding method; looking for emergent themes, insights, feedback, commonalities, and differences (Merriam, 1998).Using multiple sources of data facilitated triangulation and meaning-making in the study (Lowenthal & Leech, 2009).In addition to data triangulation that was designed and collected by the first author, a methodological triangulation was conducted in turn.The synthesized findings were given feedback from the third, fourth, and fifth authors who operated the eQIP to obtain practitioner and insider perspectives.Finally, the second author joined the research team and served as the external member checker who critically reviewed and evaluated the validity and reliability of interpretations.Direct quotes supporting a finding were identified and labeled using the format of seminar name, term, and the data source.

Result and Discussion
The results and our reflections are presented in the following section, beginning with an analysis of faculty participation in the program, then faculty perceptions of the program, followed by how faculty skills and dispositions changed over time, and ending with faculty concerns about the program.

Faculty participation
Faculty are busier than ever before (Lucas & Murry, 2011).However, simply putting faculty development online does not magically make it easier for faculty to participate.Research suggests that scheduling conflicts and overall lack of interest prevent online faculty from engaging in faculty development (Dailey-Hebert, Mandernach, Donnelli-Sallee, & Norris, 2014).Given faculty's competing priorities and limited time, we contend that it is important for institutions, and specifically faculty developers, to analyze how much time faculty are spending in online faculty development activities as well as which parts are taking the most (or least) time.

User behavior
One way to investigate faculty participation in online faculty development is to analyze the analytics in the learning management system (e.g., Vu, Cao, & Cepero, 2014).Two common ways to analyze participation using analytics are through looking at hits per page or time logged into the system.In 2014, the average page hits per user in the course design seminar ranged from 70 to 257 hits each week or an average of 132.3 hits across all 6 weeks (see Table 3 and 4).When looking at the clicks per week, the first three weeks had more page views than the last three weeks (of the design phase).This change is likely due to the content during these weeks; weeks 1-3 focused on designing the "skeleton" of an online course, whereas weeks 4-6 focused on preparing faculty to develop the course site by outlining corresponding learning activities and assessments.The change in page views, however, simply could be the result of time; in other words, faculty might simply lose some interest after three weeks into an online faculty development seminar, or it could be the result of what else is going on at that time in the semester.Additional research needs to be conducted to uncover what might be causing this change in page views.Future research like this could help inform faculty developers of the optimal length of an online faculty development seminar given the standard ebb and flow (and therefore faculty workload) each semester.
The course design seminar is designed to take faculty 8 hours each week to complete.But this seminar is often many faculties' first experience in an online learning environment.Even though the time learners spend in any online course is bound to fluctuate some from week-toweek, we strive for these seminars to follow as accurately as possible the time estimates we make to faculty when they begin.Therefore, we regularly check the time faculty report they spend in these seminars with the time they actually spent logged into Blackboard and then make changes when needed.For instance, during the spring of 2014, faculty in the course design seminar reported spending 7 to 10 hours a week on the seminar.As a result, we simplified the seminar a little by removing some assignments.Then in the following summer and fall, we were happy to find that faculty reported spending an average of 5 to 9 hours a week in the seminar.
We regularly conduct the same type of analysis on faculty participation in the teaching online seminar (eTOS) as we do with the course design seminars.Faculty are expected to spend 6 hours each week in the 6-week teaching online seminar.Page hits for this seminar ranged between 112 and 268 each week.Similar to the frequency of page hits in the course design seminar, the average page hits decreased over time.However, whereas time logged into Blackboard fluctuated week-to-week in the course design seminar, time logged in the teaching online seminar gradually decreased over time (see Table 4).As before, more data is needed to understand this gradual decline over time.While findings like these can leave one with more questions than answers, analysis like this needs to become a regular practice in online education and especially in online faculty development where faculty continue to try to do more with less.

Discussion participation
Another way to investigate faculty participation in online faculty development is to analyze online discussions (Blignaut & Trollip, 2003).Analyzing the online discussions revealed that the facilitators in the course design seminar (who are instructional designers) posted 25% -50% of the discussion posts, while the facilitators in the teaching online seminar (who are fulltime faculty) contributed only 8%-30% of the posts (see Table 5).This finding was not a surprise; the teaching online seminar is designed intentionally to be less facilitator-driven than the course design seminar because the it focuses more on online teaching strategies (Carter-Cram & Black, 2014), which among other things, allows faculty to leverage their prior experience as educators more than designing online courses from scratch.Further, an intentional goal in the teaching online seminar is to help create a community of learners who might assist faculty throughout their online teaching careers.Our faculty new to online education tend to recognize the importance of student-tostudent interaction and, specifically, the role online discussions play in predominantly asynchronous online courses, but they struggle with managing how to facilitate and engage in online discussions properly and how they use these discussions in their own online courses.For instance, some faculty reported how useful the discussions were in these online seminars because they allowed insight into what others were dealing with.One respondent said, "I found many were similar to my own issues or ideas" (eCD2S Spring, module survey), whereas another said that the discussions "allowed me to ask questions, explore my hypotheses and inclinations and see what others have to say, and therefore permitted the faculty member to "enjoy the collaborative nature of the discussions as well" (eTOS Spring, module survey).
At the same time, others reported that they "struggled to find meaning in the discussion boards" (eCD2S Fall, exit survey), and "think this is in part because I am an independent learner by nature and DB, in general, aren't really my thing (I personally don't feel mandatory DB posts aid in my learning of the material, but I do appreciate their role in an online course environment)" (eTOS Summer, module survey).
Analyzing online discussions and specific learners' perceptions of how their instructors use online discussions is important to the success of any discussion-based online course (Ramsay, Aman, & Pursel, 2014).Frankly, it takes time and commitment for faculty and students to become comfortable and literate with the unique style of reading and writing required to participate effectively in electronically mediated environments (Dunlap, Bose, Lowenthal, York, Atkinson, & Murtagh, 2016;Vaughan & Garrison, 2005).

Faculty Perception
The success of an online faculty development program in an institution where faculty are not forced to complete professional development depends heavily on faculty having a positive experience and finding value in the program.Therefore, from the inception of eQIP, we wanted to know whether faculty were satisfied, feeling a sense of community, and learning new skills and dispositions in the program.

Faculty Satisfaction
It is common practice to administer an end-of-course evaluation at the end of a workshop or course to determine learner satisfaction (Lowenthal, Bauer, & Chen, 2015).However, the problem with waiting until the end of the course or workshop to get learner feedback is that once the course is over, it is too late to make any changes and possibly improve the experience for the cohort learners (Dobrovolny & Lowenthal, 2011).Therefore, after each module in the course design and teaching online seminars, faculty are asked to rate their satisfaction on a 5-point Likert scale (i.e., Please rate your level of satisfaction with this module/seminar).
In 2014, faculty reported overall that they were satisfied with the course design and teaching online seminars (see Table 6).Faculty responses in the course design seminar ranged from 3.55 to 4.78 and from 3.67 to 4.73 in the teaching online seminar.Collecting data after each module enabled us to assess the high and low points of each seminar as well as overall trends.For instance, a closer look at the teaching online seminar weekly survey results revealed that faculty satisfaction gradually increased.Also, while Week 3 in the course design seminar in the spring was low regarding faculty satisfaction, in other semesters, faculty reported being satisfied with this week, which suggests that there is some value in not making quick changes to a workshop simply because of one bad week.Instead, faculty developers should try to collect data over time and make informed decisions based on more than one cohort of learners whenever possible.While we found it helpful to get a sense of a learner's overall satisfaction, doing so ultimately did not help identify specifically what in a course or workshop faculty are satisfied or dissatisfied about.Therefore, we also looked to specific responses from midterm and exit surveys to get a better sense of what satisfied our learners.In the course design seminar, faculty reported extremely high satisfaction with the instructional designers (who were the facilitators).However, not all the faculty were satisfied with the layout of the course.But perhaps most interesting is that when faculty were asked whether the seminar was meeting their learning expectations, the average satisfaction score decreased from the midterm to the end of the seminar, which is when they started working one-on-one with their instructional designers to develop their own courses.This decline could be due to the change in the structure of the seminar, from one that involved interacting with fellow faculty during the first phase to one where the faculty were mostly working alone (with the guidance of an instructional designer) on their own courses.It could also be that faculty expected the instructional designers or the development experience overall to be something different than it was.Like all online learning experiences, managing the expectations of faculty taking part in future seminars might help increase their perceived learning during the second half of the seminar.Due to the short duration and already high workload in the teaching online seminar, we decided to ask the previous questions only to faculty in the eTOS exit survey.While we lose some of the value of the week-to-week data, we recognize that there can come a time where learners are surveyed too often.At the end of the teaching online seminar, faculty reported that they were satisfied with every item but the course layout (see Table 8).Both online seminars are conducted in Blackboard; Blackboard offers faculty the ability to create as many folders as they like.While the folders were created with what appeared to be a clear structure and numbering system to guide the learners, a few learners reported that it was not as clear as we suspected.This data, coupled with some experiments using a screen reader on the seminar, prompted the eQIP team to restructure the layout of the teaching online seminar by flattening the entire structure and therefore having fewer nested folders.We will continue to collect data to see if this alternation changes faculty perceptions of the layout of the seminars over time.

Perception of faculty learning community
Research has demonstrated multiple benefits of establishing a community of learners in online courses (Abdelmalak, 2015;Boettcher & Conard, 2004;Dzubinski, 2014).Therefore, when developing the eQIP, the eQIP team wanted to find a way to create an online faculty learning community to both model the power of online community as well as to engage faculty further.Community-building activities-influenced in part by the theory and practices of the Community of Inquiry (CoI) (see Bond, 2011;Dunlap & Lowenthal, 2014;Garrison, Anderson, & Archer, 2000;McElrath, & McDowell, 2008;Murdock, & Williams, 2011;Pickett, 2010;Stavredes, 2011;Vesely, Bloom, & Sherlock, 2007;Wilcoxon, 2011)-were intentionally incorporated into each of these seminars.For instance, a weekly "Dear Jen, Kim, and Patrick" letter in the teaching online seminar (see Figure 1) was posted regularly throughout the teaching online seminar to model and encourage self-disclosure and a sense of shared connection while also leveraging faculty participants' prior experiences teaching.
Community is difficult to measure but it begins and ends with interacting with others.Therefore, we created weekly social network diagrams using SNAPP (Social Networks Adapting Pedagogical Practice, http://www.snappvis.org) to better understand participants' interactions in the seminar.While student-to-student or student-to-teacher interactions are not unequivocal proof of a learning community, they are a necessary building block to establishing and maintaining a community of online learners.Visualizing these interactions (by SNAPP) enabled the facilitators to diagnose and monitor participation overtime (see also Dawson, Bakaharia, & Heathcote, 2010;McCormick, 2013).These diagrams revealed, for example, that in the teaching online seminar, interactions were facilitator-centric (see dark gray node in Figure 7) during the first week, but changed to participant-centered over time and eventually began to form what appears (when looking at the interactions) as a community of learners (see Figure 8).Supporting our theory, a participant in the teaching online seminar noted in the spring of 2014 that "not only do I feel happy and excited about my teaching as I exit eTOS, I feel a connection to many more professionals at BSU" (eTOS Spring, exit survey).Establishing a learning community takes time (Dzubinski, 2014); unfortunately, though, it does not always work as planned (Olson & McCracken, 2015).For instance, in the course design seminar, some faculty complained about using the discussion board as a place to "show and tell" their design documents-a strategy encouraged by some in the field (see Lowenthal & Thomas, 2010a).However, despite resistance from some faculty, other faculty, as evident in the quotes below, clearly developed a sense of community over the seminars (largely through their interactions in the discussion forums) and in turn considered adding some community building activities in their own online courses: • "The sense of community is very important to me and I believe I gained some ground on how to promote them in my class" (eCD2S Spring, module survey).• "The idea of 'community' in an online course is something that I think most of us will have to learn and adjust to in a 'trial by fire' sort of way" (eCD2S Spring, module survey).• "The Discussion Board was a really helpful way to build community and share ideas about learning objectives" (eCD2S Fall, module survey).• "I read through how the others were approaching this task made me flash on [course designs] I hadn't thought of to try myself "(eCD2S Fall, module survey).

Changes in faculty attitudes and disposition
The eQIP is designed to teach faculty how to design high-quality, fully online courses from scratch as well as how to teach effectively in this new environment.Hence, we were interested in looking at faculty attitudes, knowledge, and dispositions at the end of each seminar, as well as how they might have changed over time while participating in the online seminars.For the course design seminar, we created a survey with a series of quantitative, Likert scale questions as well as qualitative, open-ended questions to measure faculty attitudes and dispositions.The survey was administered three times in the course design seminar (i.e., beginning, middle, and end of the seminar) and two times in the teaching online seminar (i.e., the beginning and end).The results from these surveys are summarized in Table 9 and discussed along with open-ended feedback in the following sections.

Faculty confidence
We were interested in how faculty confidence in designing and teaching online changed over time.We hoped that the eQIP would transform participants' view of online learning from content-oriented toward student-oriented (Vries, Grift, &, Jansen, 2014) as well as leave them with a more realistic expectation of online teaching.Table 9 shows that faculty confidence in identifying and describing target online student characteristics, analyzing differences between the online learning environment and the on-campus classroom, and designing online assessments linked to course learning objectives and activities increased over time in both seminars.Interestingly, participants in these online seminars expressed different attitudes in questions 4, 5, and 6.For instance, after faculty participated in the teaching online seminar, they anticipated lower course evaluation scores when teaching online courses; a decrease of -1.52 from the beginning to the end of the seminar (see Question 4 in Table 9).This result could be due to the fact that faculty participants read the assigned textbook about online teaching, realized the demands of highly interactive online courses in action, and read research on teaching evaluations decreasing when teaching online compared to teaching face-to-face courses (see Lowenthal et al., 2015).Then when asked if they expected the workload of designing and teaching to be higher in online courses than in face-to-face courses, faculty responses increased over the duration of both seminars-meaning that while they expected the workload to be higher, this expectation grew by the end of both seminars.However, faculty responses regarding expected workload in the teaching online seminar barely increased from the beginning to the end of the seminar, whereas faculty response in the course design seminar increased 0.49 by the end of the seminar (see Question 5 in Table 9).Finally, when asked about how difficult it will be to motivate online students, faculty responses in the teaching seminar decreased some (suggesting that they might be gaining confidence in motivating online learners), whereas faculty responses in the course design seminar increased over time (+0.62).While it is normal to lose confidence once one learns more about something (e.g., teacher preparation: students often report being more unsure about teaching after they have student taught), a goal of the eQIP is to develop our learners' confidence.Therefore, we will continue to find ways to improve these aspects of our seminars over time.

Perceived changes in online course development
Teaching is an individual activity; as such, teachers at all levels of education often feel isolated (McQuiggan, 2011).Some faculty thrive on this autonomy, while others do not.Instructional design, on the other hand (at least in large departments), is often a collaborative endeavor; therefore, we intentionally developed the course design seminar to help faculty see course design not as an individual intellectual activity (as they naturally might) but as an open collaborative activity.Open-ended feedback revealed how faculty sensed their growth.For instance, one faculty member reported how the seminar activities provided an opportunity to "think about the different scales (course vs. unit/module) of objectives.It helps to be forced to understand and articulate how they are connected with one another" (eCD2S Fall, module survey).Another reported how figuring "out how I would write directions for an online activity was a challenge and eye opener" (eCD2S Fall, module survey).Faculty also reported how these seminars helped them rethink how they teach other courses-a goal for nearly any faculty developers.For instance, one faculty member reported how she or he now saw the potential to "redesign course assessment/evaluation in my other courses, and I'm already telling colleagues outside of this seminar about it.Good stuff!" (eCD2S Fall, module survey).

Perceived changes in online teaching
Faculty who have never taught online before often hold certain assumptions about teaching online.These can range from "teaching online entails more work than teaching face-toface" to other opinions such as "students cheat more online."The teaching online seminar is designed not only to teach faculty how to teach online but to challenge common misconceptions faculty might hold.Participants reported changes in their perception regarding the ability to effectively communicate online.For instance, one participant found "orientation videos to be personal, informative, and a great way to begin an online course," and that he or she "enjoyed watching the samples and getting ideas for [her] own" (eTOS Spring, module survey).Others discovered some of the affordances of a LMS.For instance, one participant pointed out how "the possibilities for reflection in a safe and confidential modality is important.It's different than the discussion board because the public aspect of that really forces you to be on your guard" (eTOS Spring, module survey).Finally, others specifically identified how their perceptions of online teaching changed but remained realistic.For instance, one participant stated that "I know teaching my first online class will take a lot of time; I appreciate the reminders and [facilitator]'s comment to not freak me out" (eTOS Summer, module survey).Perceived changes like these are something we hope to see (e.g.Scagnoli, Buki, & Johnson, 2009) from participants as a result of taking part in the eQIP.

Faculty Concerns
Exit survey results suggest that overall faculty considered the eQIP a valuable faculty development experience.However, our inquiry also uncovered concerns faculty had with the eQIP as well as online teaching in general.

Individualized vs. grouped training
Boise State, like many institutions, is interested in creating new online courses and programs to meet student demand as efficiently and inexpensively as possible.The eQIP was designed to train faculty to be able not only to design online courses (that they or others would teach) but to learn how to teach any online course (whether one they designed or not).Some faculty, though, had concerns with this model and felt that it was too standardized.Regarding the course design seminar, some faculty reported concerns with standardized course development.One faculty member explained that a standardized model of course development "takes longer to work within someone else's templates than your own" (eCD2S Summer, midterm survey).Some faculty explained that the eQIP-grouped "standardized" training approach does not work well for faculty, many of which work better individually (e.g., with a dedicated instructional designer) than in groups.One faculty participant captured this sentiment when he or she suggested that eCampus should embrace "the idea that faculty learners are very different from graduate students and that a personalized, one-on-one training defined by faculty priorities and focusing on personalized interaction would be much more effective" (eTOS Spring, exit survey).While we recognize that some faculty would prefer a one-on-one approach, there are not enough resources to accommodate such an approach.We are, however, continually reminding faculty that they are the subject matter experts, that these are their courses, and that they can change the course template when needed.

Novice vs. veteran faculty participants
The target audiences of the eQIP are novice online instructors who have little or no prior experience designing/developing online courses and/or teaching online at Boise State.However, more and more faculty with prior experience as either online students or online instructors at prior institutions sign up for the eQIP; they sign up because their specific department expects all new online faculty to go through the eQIP.For instance, one faculty participant reported: Not all faculty who take part in the eQIP with prior online teaching experience are upset.A number of them are excited to learn new ways of designing and/or teaching online courses.But the majority of these faculty do want an advanced version of the eQIP as captured in the following quotes: I would encourage you to consider breaking eTOS into two seminars: (1) with standard topics for instructors who have never taught online before, much like this seminar; (2) built around grouping experienced faculty with shared priorities (collect a ranked set of priorities) into small task groups to address those priorities, allowing for extensive oneon-one interaction (for example one hour per week via Google Hangouts) with instructional design consultants who have expertise in that area (eTOS Spring, exit survey).
Maybe there should be a third option of course development where teachers have taught online before, but want to radically alter the design and look of their online course (eCD2S Spring, module survey).
One idea here might be to create a diversion plan -a set of assignments for faculty who have never taught online, and a different role set for faculty who have -so that the latter are being pushed both to review what they have (rather than plan it) and also to offer reflections on what works and not for those who have not.In other words, faculty clearly have very varied backgrounds and one size does not fit all (eTOS Spring, module survey).
Feedback like this argues for a more flexible and individualized and/or customized version of the eQIP (eCD2S Review Meeting, August 8, 2014).This complaint is a common problem with many online courses, not just online faculty development.The eQIP team continues to consider ways to change the eQIP by incorporating individualized learning designs such as worked examples (eCD2S Review Meeting, June 4, 2014), quest-based learning and digital badges (Haskell, 2013) or competency-based learning (Sally & Louis, 2014).

Technology as prerequisite vs. just-in-time technical training
Faculty are subject-matter-experts-not necessarily technology experts (Kukulska-Hulme, 2012).At Boise State a tension exists, as it does at many other universities, about how many technology skills faculty should be expected to have (or develop) to teach online and where they should go to get help when they need it.The eQIP currently does not focus much on developing basic technical skills in faculty (e.g., uploading a document to Blackboard).Instead, the eCampus takes the stance that all faculty who wish to teach online (and participate in the eQIP) should have basic technical competences before beginning the eQIP.However, in practice, faculty do not always pay attention to this prerequisite; neither do they understand what basic technical skills are in the first place.Thus, we find a reoccurring concern with some faculty is about whether being proficient with basic technology should be a prerequisite or whether we should provide just-in-time technical training.For instance, one faculty participant suggested that there needs "to be better support during the heavy tech portions of the class."Along the same lines, another person pointed out the need for a "specific contact person [to] be identified to help troubleshoot issues" (eTOS Fall, exit survey).Aware of this continuing concern, the eQIP facilitators have tried to provide more-than-expected just-in-time technical training (such as tip sheets and tutorial videos) to compensate for the gaps in faculty technical skills.And while some faculty have praised this flexibility and support-for example, one reported how he or she "enjoyed specific skill set instruction" (Camtasia, Google Hangouts, tech skills, etc.) (eTOS Summer, exit survey)-the tension still remains about whether it is fair to expect faculty to have basic technical skills before teaching online for the first time.

Experiencing learning as an online student
Each of the eQIP seminars is conducted fully online, which allows faculty to experience firsthand what it is like to be an online student and to experience some of the common complications students face (e. g., technical difficulties with Blackboard LMS, using e-textbooks and multimedia, academic honesty, accessibility, copyright issues, student interaction, and time management [see Mason et al., 2010]).Faculty regularly report that this hidden curriculum is a valuable experience.For instance, faculty often recognize how a busy life can get in the way of completing an online course.One faculty person said, "I have more sympathy and empathy with my online students now.I get to this class after family and work responsibilities; I am trying to do enough to get by" (eTOS Spring, exit survey).Another group found themselves questioning whether they really should have deadlines over the weekends if they do not want to work over the weekends.For instance, one faculty participant in the eCD2S commented: "I am curious as to why there are Saturday deadlines and I have been considering whether I want those in my course" (eCD2S Fall, module survey).Additionally, faculty often experience technical problems for the first time from a student's perspective as captured in this quote: "I know this is more of a Bb issue but now I understand when my students say -'Oh I didn't see that.' " (eTOS Fall, module survey).This authentic experience, though, takes time.Some faculty reported that this authentic experience could be done in less time or with less work or that their participation in the seminar should not be graded.This concern about being treated too much like an online student was addressed in ways in the course design seminar by explaining to faculty that by experiencing discussions, assignments, quizzes, and group tools from a student's view, they would be in a better position to resolve students' issues during teaching.The eTOS facilitators, however, tried to make participants' comments and concerns teachable moments.For instance, after receiving a complaint asking "Why am I 'losing points' on a self-assessment for not doing an optional assignment?That doesn't seem fair!" (eTOS Spring, module survey), facilitators apologized for the error, which was simply an oversight, and explained that errors like this sometimes happen when adjustments are made to a course from semester to semester and that faculty should strive to double check each course when making even minor adjustments (eTOS Review Meeting, October 22, 2014).Such modeling was effective because, as one faculty member reflected in his journal entry: "Don't be afraid to admit a mistake…… I'm somewhat dyslexic and sometimes things just get confused.I tell my students that if you find something that seems odd, just tell me because it probably is and I'll fix it" (eTOS Spring, journal activity).

Workload and time competition
Time is a sensitive issue with faculty (Lowenthal, Wray, Bates, Switzer, & Stevens, 2013).One of the challenges the eQIP faces is competing with limited faculty time.For instance, even though the teaching online seminar only takes 6 hours a week for 6 weeks (the shortest of the seminars), this is still 6 hours a week that faculty have to take away from teaching, scholarship, or service.Despite our best efforts, some participants reported that tasks in seminars were "busy work."For instance, one faculty reported in the teaching online seminar "I simply, realistically, literally(!) did not have enough time to complete while at the same time maintain my teaching, research, and personal life!!! (Wheww!)" (eTOS Summer, exit survey).This tension gets even worse with the course design seminar that takes even more time.Boise State faculty, like online faculty elsewhere, regularly report that designing an online course takes more time than preparing for a face-to-face course (see also Bento, 2011).However, some qualitative data suggest that many of these participants begin to change their perception of "busy work" by the end of the seminars.A common response from faculty looks like this: "The material in this course was overwhelming many times.At the end I can see that the structure was appropriate" (eCD2S Spring, exit survey).This change in perception is even more common in the online seminars because early in the course design seminar faculty are required to complete multiple worksheets as part of the instructional design process that they are not used to completing when designing face-to-face courses.It is usually not until the fifth or sixth week that faculty report that they "feel like things are starting to come together in a way that will allow me to best utilize my time with the design consultant" (eCD2S Spring, module survey).We have found that faculty who sign up for the course design seminar to design a course that they are signed up to teach the following semester do better than those who have a few semesters before they actually teach the course they are developing.We also began adding time estimates for each activity in the LMS for both seminars to help faculty manage their own time.However, providing accurate time estimations for all learners is nearly impossible; therefore, some faculty who worked slower than others actually were bothered when time estimates did not match the time it took them to complete a project.

Conclusion
A successful online faculty development program must include pedagogical support, technology support, and design and development support (Baran & Correia, 2014) that overcome obstacles about time, expertise, and motivation of faculty (Henning, 2012).This study addressed the implementation of an online faculty development program at a mid-size, metropolitan research university.We investigated faculty participation and perceptions by user pattern, discussion participation, satisfaction, learning community, attitude and disposition, as well as their concerns.The results of our inquiry suggest that while there are still ways we can improve each seminar, overall, the eQIP is preparing Boise State faculty to design, develop, and teach online courses.In 2014, in fact, 51 online courses were developed as a result of the eQIP, 100 faculty members completed both seminars, and 38 courses went through QMPR.With the faculty intent to change teaching practice, eQIP impacts both online and face-to-face teaching.In the words of one faculty participant: This course is a worthy endeavor for both online and face to face teachers.I will use the materials that I made for the online course in face-to-face classes as well.I plan to incorporate online components into my face-to-face classes in the future (eCD2S Spring, exit survey).

Future improvement based on lessons learned
Online faculty development is a rapidly changing field and its offerings require continuous evolutions (Meyer & Murrell, 2014).Based on the findings of this study, the eCampus Center should continue improving the seminars to be as engaging as possible.Our continued data collection efforts reveal the following four potential areas of improvement.

1.
Provide individualized seminar processes Several faculty comments indicated that having all participants go through the faculty development seminars at the same speed failed to serve faculty from various disciplinary backgrounds.A more directional and customized faculty development process may better meet faculty's needs.For example, a discipline-based repository for online course will be helpful and flexible for faculty who are not beginners in online education.A faculty development program like the eQIP does not have to be 100% online to be authentic, relevant, motivating, engaging, and useful for faculty (Sorinola, Thistlethwaite, Davies, & Peile, 2015).

2.
Empower faculty ownership of the seminars Experienced faculty should be identified as soon as possible and encouraged to become mentors to novices in the seminar (Xu & Morris, 2007).Seminar topics should be chosen from faculty's teaching experience to promote faculty ownership of the seminars.

3.
Condense course content In spring 2014, the eTOS team invited a new facilitator to lead the seminar.His external perspective helped reorganize and remove any redundant information.This significantly lowered faculty's workload in the teaching online seminar.

4.
Provide tenure-related incentives The eCampus Center should negotiate at the University level to provide faculty supports and recognitions in online teaching.For example, to include the eQIP participation as one element in faculty promotion, allow online course developers to count development time toward semester teaching loads, and create online teaching fellow programs to recognize faculty contribution to online education.

Limitations
The purpose of this study was to describe one institution's effort to develop high-quality online courses and high-quality online teachers.Therefore, as a case study within a highly-individualized context, the results of this study offer insights to peer institutions with similar backgrounds, but may not be generalizable to a larger population (Stake, 1995;Yin, 2009).But even as a case study, the results could have been strengthened by including additional data such as a faculty focus group; a design of pre-and post-seminar faculty survey, participants' retention rates, and student outcomes in the high-quality courses developed, and faculty end-of-course evaluations teaching these courses.In the future, concerted and coordinated research efforts across peer institutions may advance the study of faculty development beyond the level of sharing best practices in the field (Meyer, 2013).
1. Phase 1−Design Phase (Weeks 1-5): A group of faculty, facilitated by instructional designers, work online in a Learning Management System to learn the skills necessary for online course design; 2. Phase 2−Development Phase (Week 6-12): The faculty work one-on-one with instructional designers to develop their online courses.

Figure 2 .
Figure 2. The Social Network Diagram of Interaction in Teaching Online Seminar (Summer) Dear Jen, Kim, and Patrick: I took your suggestion and had my online students fill out a midterm survey about the course.When I looked at the survey results, I felt very discouraged.Students had some good things to say about the course, but they also had lots of complaints about the workload, the course organization, and the assignments.They even said they felt like the discussion board prompts were boring and didn't inspire meaningful conversation.What should I do now?Should I try to change everything they don't like about the course?Or should I just stick with what I have planned

Table 1
Number of eQIP participants in 2014

Table 2
Overview of Data Collection

Table 5
Facilitator Post Ratio of Seminars

Table 6
Faculty Satisfaction with the Course Design and Teaching Online Seminars

Table 7
Faculty Perceptions of the Course Design SeminarSpring, Summer, and Fall 2014

Table 8
Faculty Perceptions of the Teaching Online Seminar During 2014

Table 9
Faculty Confidence Designing and Teaching Online Courses During eQIP Seminars in 2014 am beginning to think I am in the wrong course....I have been trained in course development at the graduate level and have participated in accreditation self-studies.What I really needed was the skills in the technology of online learning, but you can't take that class until you are teaching online for Boise and you can't teach online for Boise until you take this course.(eCD2S Spring, module survey).