LENS (new SEI)

SEI Committee Membership expanding section

Jennifer Klein (Biology), Chair
Adam Driscoll (Sociology and Criminal Justice), Recorder
Alysa Remsburg (Environmental Studies), Member
Alder Yu (Biology), Member
Tesia Marshik (Psychology), Member
Nabamita Dutta (Economics), Member
Bryan Kopp (English, CATL), Member
Samuel Ott, Student Senate Representative

Consultants: Kristin Koepke (CATL)

Summary of the SEI Committee’s Work expanding section

In response to concerns that have been raised in both local and national arenas regarding the validity of student evaluations of instruction (SEIs) and the problematic use of SEIs in personnel decisions, the University of Wisconsin-La Crosse (UWL) Faculty Senate Executive Committee (SEC) established an ad-hoc committee to investigate the use of SEIs on UWL’s campus.  ​

Executive Summary of 2019-2020 Report to faculty Senate

In 2018-2020, the UWL SEI Committee completed (1) an in-depth literature review of SEI tools, focusing on validity and bias, and (2) an investigation of SEI use at UWL. The major findings were described in the 2019-2020 report and are summarized here:

  • Multiple recent high-quality meta-analyses have shown that SEI scores generally fail to correlate with student learning.
  • Numerous studies indicate that SEI scores are highly sensitive to implicit bias against instructors of color and female instructors. Using SEI scores in personnel decisions could be a discriminatory practice at UWL.
  • Many UWL instructors report stress and anxiety due to both the use of SEIs in personnel decisions and in receiving low scores and/or negative student comments.
  • Most UWL instructors (>70%) do not find SEIs useful for teaching development.
  • Female instructors and instructors of color are unfairly burdened with striving to achieve higher SEI scores by devoting excessive time to altering their teaching or presentation, or in dealing with anxiety related to low SEI scores.
  • Resources devoted to improving student perceptions of learning could detract instructors and the institution from higher impact practices.
  • UWL students are dissatisfied with the UWL SEI because they feel their voice is not heard on high priority items.

Recommendations to faculty Senate regarding SEI methodology and the use of SEIs in personnel decisions

The SEI Committee has made the following recommendations to Faculty Senate to mitigate bias in the SEI tool and to lessen the impact of biases in personnel decisions:

  • Evaluation of teaching effectiveness should include student feedback, but only as a part of a holistic evaluation that includes peer observations, review of teaching materials, instructor development activities, and instructor self-reflection.
  • Questions on SEIs should focus on student experiences and the instrument should be framed as an opportunity for student feedback, and not a formal rating of instructor effectiveness.
  • Numerical SEI scores should never be used to compare instructors to each other or to a department average (ranked lists are particularly problematic). As part of a holistic assessment, numerical scores can be used to document patterns for an individual instructor member over time.
  • Evaluators (all instructors) should be trained in using student ratings as part of a holistic teaching evaluation. Students should also be trained in rating their experiences in a First Year Seminar.

Charge to test and pilot an alternative SEI tool

In 2020-2021 the committee was tasked with determining what a productive student feedback instrument might look like that is less discriminatory, less stressful for instructors, and helps instructors to improve their teaching. The committee searched for existing instruments that adhered to the above recommendations and found just a few. The most fully developed tool was created by Dr. Zinta Byrne, an organizational psychologist at Colorado State University (CSU) and was implemented at CSU for the first time in 2018-2019. We’ve adapted that tool for UWL in collaboration with Dr. Byrne and her colleagues.

Here, we share with you a vision for a revised SEI tool that we will now refer to as the UW-La Crosse Learning Environment Survey (UWL-LENS).

Learning Environment Survey (LENS) expanding section

Focus on student experience of learning environment

Students aren’t qualified to formally evaluate their own learning or the effectiveness of their instructor, as they haven’t achieved mastery of the discipline, nor are they experienced teachers. However, students are uniquely situated to describe the course learning environment, which includes, but is not limited to, instructors, peers, course activities, inclusivity of activities and discussions, online platforms, and other technology. The UW System Regent Policy Document 20-2: Student Evaluation of Instruction suggests that in interpreting SEIs “emphasis may to good effect be placed on description of what happens in the classroom.” Knowledge of what happens in the classroom, as the students experience it, is a critical aspect of teaching development and evaluation.

The student learning environment shapes student success, both in terms of academic achievement and the general sense of fulfillment that accompanies learning. With knowledge of how students perceive key aspects of their learning environment, it is possible for instructors to reshape the environment to promote student success. The UWL-LENS therefore focuses on the student’s experience of the learning environment and strictly avoids questions that students are not qualified to respond to.

Based on Research in undergraduate education

The UWL-LENS survey items are based on the Seven Principles For ​Good Practice in Undergraduate Education, originally described by Chickering and Gamson in 1987 (AAHE bulletin 3 (1987): 7):

  • Encourages contact between students and instructors
  • Develops reciprocity and cooperation among students
  • Encourages active learning
  • Gives prompt feedback
  • Emphasizes time on task
  • Communicates high expectations
  • Respects diverse talents and ways of learning

These principles of effective teaching have been applied to modern teaching modes, including online education (New directions for teaching and learning 75 (1999): 81; The technology source 30.5 (2001): 50). The LENS questions were created by considering the student experience associated with each principle of effective teaching. For example, effective teaching includes providing prompt feedback. A student can be reasonably asked whether the feedback they received (from the instructor or peers or elsewhere) was useful and timely enough to monitor their own learning and make progress. This contrasts a typical SEI, which might ask students to rate the quality of an instructor’s feedback, which a student isn’t qualified to evaluate. The shift in focus from evaluating the instructor to describing the learning environment is a subtle, but important distinction.

Facilitates evidence-based improvement in teaching

Students are generally eager to provide feedback and desire that instructors will use this feedback to improve the course experience. Since LENS questions are aligned with established principles of effective teaching, survey items can be mapped to resources for instructor development. Student responses are reported as frequency distributions (and ample comments), which allows the instructor to monitor the percentage of students who experience the learning environment in a particular way. When used in a process of reflection, purposeful change, and re-assessment, LENS data can be used as evidence of teaching development in formal teaching evaluations. Of course, instructors drive their own development, choosing focus areas. The LENS will promote movement away from evaluating instructors using a potentially biased and statistically mistreated number and toward a holistic evaluation that focuses on teaching development.

The LENS survey was designed to be given both mid-semester and at the end-of-the-semester so that instructors can respond to student feedback at a point in the semester where changes could be introduced.

Colorado State University has carefully mapped the LENS questions to their framework for effective teaching developed by their center for teaching and learning (https://tilt.colostate.edu/ProDev/TEF). CATL is working to similarly align LENS survey items with their teaching framework.

Reduced bias against instructors

Student evaluations of instruction often include questions that trigger student biases related to the instructor’s gender, race, nationality, and age. Numerous studies have indicated that SEI scores more sensitively reflect these biases than the quality of instruction. The LENS redirects student focus from the instructor to the learning environment to lessen the probability of triggering implicit bias against the instructor. A preliminary pilot of LENS questions at Colorado State University generally supports its effectiveness, but a larger study is needed to clearly establish whether these questions trigger the same biases as the traditional SEI tool at UW-La Crosse. It is important to note that no tool will be free of bias, as implicit bias can occur for survey items seemingly unrelated to gender norms or stereotypes.

The UWL LENS asks no direct questions about the instructor.  All questions are framed to prompt the student to reflect on their experience of the learning environment, which the instructor plays a large role in shaping.

No instructor comparisons

No matter the instrument used to collect student feedback, it is a best practice to avoid comparing instructor scores for the purpose of formal evaluation​s used in personnel decisions, as no tool will be entirely free of bias.

The UWL-LENS was designed so that it doesn't make sense to ascertain an aggregate numerical 'score' that could be used as the basis for comparison among instructors. ​The UWL-LENS presents students with just a few categorical options rather than a numerical Likert-scale. Frequency distributions and student comments are reported. By replacing the Likert-scale with categorical options (that don't map sensibly to numbers), we’ve decreased the probability that an aggregate score could be derived from multiple survey items.  ​

For example, the LENS report would communicate that 42% of students felt feedback ‘occurs early enough to make progress,’ rather than communicating that students rated instructor feedback as an average of 2.8 on a 5-point Likert scale. The current SEI survey uses Likert scales to make it easy to average scores from multiple survey items. Note that this is an inappropriate mathematical manipulation of categorical data (like averaging the responses ‘blue’, ‘green,’ and ‘red’). The resulting aggregate score made for easy comparisons among instructors, but the practice is both statistically and ethically inappropriate. For the LENS, student responses for every question must be considered independently of other questions, which lends itself to focused teaching development.

It would be practical for a Chair or Dean to focus on a couple of key questions on instructor LENS reports for quick insight into how students experience an instructor's classroom. Departments and colleges could choose which LENS items would be particularly helpful for this purpose.

The UW System Regent Policy Document 20-2 encourages system institutions to allow a wide variety of methods for using SEI results for teaching improvement and requires each institution to develop “systematic and firm procedure(s)” guiding the use of SEI results for personnel decisions (e.g. tenure or promotion). It would be reasonable for a subset of LENS results to be included in promotion, retention, and tenure materials, perhaps within the context of teaching development.

Richer data for personnel decisions

The UWL-LENS does not ask students to evaluate the instructor, and shouldn't be used directly for evaluations related to personnel decisions. Instead, the LENS produces a rich dataset that could be used by instructors to reshape the learning environment to maximize student success. ​

A component of a formal teaching evaluation might focus on how an instructor has reflected on student responses, made efforts to reshape the learning environment for the success of all students, and measured the impact of these changes. Evidence of teaching development based on LENS data will result in a more accurate and useful evaluation than the SEI score. ​

Some LENS items might be useful to departments and promotion committees for insight into an instructor’s efficacy in particular domains of teaching. It is valuable to monitor an instructor’s progress along a domain of teaching, particularly if they’ve indicated they have focused development efforts in that area.  It might be reasonable for an instructor to choose one focus area per year and document progress within that domain.

Summary

In summary, the LENS was developed to (1) gather student perspectives and experiences related to their learning environment so that feedback can be used to enhance their course experience, (2) redirect students from evaluating the instructor to describing their learning environment to avoid triggering implicit biases, and (3) create a reliable and informative tool for instructor development.

Learning Environment Survey Items (Spring 2021)

Spring 2021 UWL-LENS Pilot expanding section

All UWL instructors are invited to participate in the Spring 2021 pilot of the LENS.  The research aims of the pilot are summarized below:

Research aims

Triggered Bias​

  • Characterize the extent to which both the existing SEI and the LENS questions trigger implicit bias​. Students must complete both the SEI and LENS.

Instructor Stress

  • Evaluate instructor self-reported stress prompted by LENS reports in comparison to SEI reports. Stress might be related to student comments, scores, or perceptions of how data might be used in personnel decisions.

Teaching Development​

  • Evaluate instructor perceptions of the LENS usefulness in teaching development in comparison to the SEI.

Student Voice​

  • Evaluate whether the LENS facilitates student feedback on high priority items for students​.

What can Participants expect?

As a participant in the pilot study, we will email your students (in the courses you indicate) the LENS in the same week that students complete the SEI. Students are encouraged to take both surveys. Instructors are encouraged to incentivize student participation and to allocate class time to this activity.

The Spring 2021 student survey will include both LENS survey items and some items related to whether students felt the LENS adequately provided space to address their observations and concerns about the learning environment.  The LENS takes, on average, 10 minutes for a student to complete. 

Participating instructors will receive LENS feedback after the end of the semester. At that point, instructors will be surveyed to measure the perceived usefulness of the LENS in teaching development, the stress prompted by reading LENS reports, perceptions of bias in the LENS instrument, and issues with any particular LENS items.

At a later point, department chairs and deans will be surveyed for their perceptions of the LENS usefulness in personnel decisions.

Your data, which will include demographic information and student feedback for your courses is sensitive information, and is protected by our IRB protocol. Please read our protocol here and feel free to email jklein@uwlax.edu with questions about how we will protect your information. When you register, we will ask you some questions about your understanding of the risks and benefits of participating and ask for your informed consent.

In summary, participating instructors will be expected to do the following:

  • Register your courses using the link below. During registration you will be asked demographic questions that you may choose not to answer.
  • Respond to our request for student email addresses.
  • Allow us to email your students during the last two weeks of the semester to request they complete the LENS. We provide students with a link to a Qualtrics survey for your course.
  • Complete a survey about your perceptions of the LENS after we send you LENS reports for your courses.

We can’t overstate the importance of high instructor participation in this pilot study. Student feedback plays a large role in our professional development and career trajectory. It is worth the attention of every instructor. To participate, please register below.  We will follow up with you at multiple points in the semester so that you know what to expect.

LINK TO QUALTRICS SURVEY