Student Experience Survey
The Teaching Evaluation Working Group (TEWG) has been working since Fall 2019 on designing a new system for evaluating teaching at OU. The goal is to create a culture that encourages development of teaching skills and rewarding the use of evidence-based effective teaching practices. As part of this effort, we have developed a new student survey, the Student Experience Survey (SES).
The Purposes of the Annual Evaluation of Teaching
- To provide actionable feedback to instructors to help them understand and improve the effectiveness of their teaching methods, materials, and activities.
- To assess instructors for purposes of evaluation, including but not limited to tenure, promotion, annual faculty evaluations, teaching awards, continuing a teaching contract, etc.
The Student Experience Survey
Feedback from students is crucial to the development and continual improvement of pedagogy, materials, and content in a course. TEWG has created a new kind of mechanism for gathering student feedback, called the Student Experience Survey, or SES. The SES provides students with an avenue of expression to faculty where they may articulate their experience in classes, including their impressions of what went well and what did not.
The SES was designed with both of the annual evaluation of teaching purposes in mind, as well as to provide students with a mechanism to reflect on their experiences in the class and offer feedback to faculty.
Many of the questions are specific and ask about aspects of the course that instructors have direct control over. Students are given the opportunity to respond in text boxes in even greater detail in each section of the survey, if they so choose. (Purpose 1)
Questions are designed primarily to ask students about their experiences in class, rather than to assess the quality of the instructor or instruction. This leaves it to those with expertise in the subject area and in pedagogy to draw inferences about teaching effectiveness and quality, where appropriate. (Purposes 1 & 2)
Open text boxes provide students the opportunity to provide more detailed feedback. These text boxes are framed by the previous questions and the text box prompts so as to maximize relevant and useful responses. (Purpose 2)
The TEWG believes that student feedback is important and should play an indirect role in administrative assessment of teaching. Student feedback is an element of such assessment, but we encourage departments and administrators to find ways of taking such feedback into account that does not treat it as a direct assessment of teaching in and of itself.
We will be working with pilot units to suggest ways to do this in the context of their system for evaluating teaching. (Purpose 2)
We also think that whatever instruments are used by administration for evaluation purposes should be as free from the effects of implicit (and explicit) bias as possible. The questions in the SES were crafted with concerns about implicit bias at the forefront, based on up-to-date research on the prevalence and mitigation of such bias. (Purpose 2
Problems with the STE’s current role in annual faculty evaluation of teaching
- STE data provides little in the way of helpful, actionable feedback to instructors.
- Reams of data show that STE ratings are subject to bias of all sorts.
- Numerical scores are subject to all sorts of misuse.
- Traditional STEs are questionable at best as a measure of student learning.
Models and motivations
We used insights from work and discussions at other institutions or organizations, including:
- University of Oregon https://provost.uoregon.edu/revising-uos-teaching-evaluations
- Teaching Quality Framework at the University of Colorado and other affiliates https://www.colorado.edu/teaching-quality-framework/about-tqf
- AAU report “Aligning Practice to Policies: Changing the Culture to Recognize and Reward Teaching at Research Universities” https://doi.org/10.1187/cbe.17-02-0032
- National Academies: National Dialog on Transforming STEM Teaching Evaluation in Higher Education https://www.nationalacademies.org/event/01-14-2021/national-dialogue-on-transforming-stem-teaching-evaluation-in-higher-education
- The TEWG collected survey data that include how faculty responded to the old STE system. (We plan to collect data that compares faculty responses to the new SES.)
Evidence that we are on the right track
Two articles that appeared in early 2021 clearly articulate the reasons to change how we evaluate teaching and make recommendations similar to the actions TEWG has undertaken (citations below).
I. A recent meta-analysis of over 100 surveys on bias in teaching surveys produced a list of suggestions for how to use student input fairly and constructively. The OU-Student Experience Survey aligns strongly with each of these suggestions.
- Contextualize evaluations as perceptions of student learning, not as a measure of actual teaching. Although they are commonly called “student evaluations of teaching,” SETs do not actually evaluate teaching. Instead, student evaluations represent a student’s perception or experiences in a course (Linse, 2017; Abrami, 2001; Arreola, 2004). Students should not, and arguably cannot, evaluate teaching. A more accurate name for these surveys would be student experience questionnaires or student perceptions of learning. When properly contextualized as feedback on experience, rather than evaluating teaching, these assessments can provide useful feedback for faculty and administrators.
- Be proactive about increasing the validity of the assessment by improving response rates. Administrators should be cautious with assessments based on too few responses.
- Administrators should interpret the results of student ratings with caution. Student evaluations are not designed to be used as a comparative metric across faculty (Franklin, 2001); rather, their purpose is to gather information about how students perceived a faculty member teaching a certain course.
- Restrict or eliminate the use of qualitative comments. Across all the studies in our sample, the clearest evidence of gender bias is in qualitative comments… Instead of asking for general “comments,” assessments should direct students to provide feedback on certain experiences with the course, as this may reduce irrelevant and mean comments.
- Administrators must not rely on student evaluations as the sole method of assessing teaching.
- Produce more research in interventions to reduce bias.
- While there are multiple dozens of articles establishing bias in student evaluations of teaching, there are very few articles that test interventions to mitigate bias. What little research exists yields some promising leads. For instance, reducing the size of the scale can mitigate gender bias (Rivera & Tilcsik, 2019). Another study that uses a randomized control trial finds that making students aware of biases can mitigate the gender gap in SETs (Peterson et al., 2019), though the evidence here is somewhat contradictory (Key & Ardoin, 2019), and anecdotally may induce a backlash effect.
II. A recent article summarizing the literature on STEs from 1990-2020 had the following abstract:
This paper analyses the current research regarding student evaluations of courses and teaching. The article argues that student evaluations are influenced by racist, sexist and homophobic prejudices, and are biased against discipline and subject area. This paper’s findings are relevant to policymakers and academics as student evaluations are undertaken in over 16,000 higher education institutions at the end of each teaching period. The article’s purpose is to demonstrate to the higher education sector that the data informing student surveys is flawed and prejudiced against those being assessed. Evaluations have been shown to be heavily influenced by student demographics, the teaching academic’s culture and identity, and other aspects not associated with course quality or teaching effectiveness. Evaluations also include increasingly abusive comments which are mostly directed towards women and those from marginalised groups, and subsequently make student surveys a growing cause of stress and anxiety for these academics. Yet, student evaluations are used as a measure of performance and play a role in hiring, firing and promotional decisions. Student evaluations are openly prejudiced against the sector’s most underrepresented academics and they contribute to further marginalising the same groups universities declare to protect, value and are aiming to increase in their workforces.
In Summer 2020, the TEWG ran a pilot project in more than 500 course sections. The TEWG then used the feedback from both instructors and students to modify the SES questions. In Fall 2020 and early Spring 2021, we gathered more feedback and developed the next draft. The next pilot phase is currently taking place in Spring 2021. During this pilot, some departments across campus have opted to use the SES as their primary end-of-course survey. The pilot departments will work with TEWG to develop their strategy for appropriately using the SES in their annual evaluation process. The pilot departments will then share their experiences with other academic units. OU plans to implement the SES campus-wide in Fall 2021.