IDP NEWSLETTER ARCHIVE: April 1984

The Evaluation of College Teaching

Editor's Note: On February 13, 1984, the University of Oklahoma Faculty Senate "approved" a committee report on the evaluation of teaching and has forwarded the report to the provost with the request that it be sent on to the colleges and departments for their review and discussion. Because of the potential significance of the evaluation procedures recommended in this report, this issue of Notes on Teaching is being devoted to a condensed version of this report.
Interested readers may request a copy of the full report (37 pages) by contacting Dee Fink at the Office of Instructional Services.

Academic units should also be aware that Prof. Nim Razook, who was chairman of the Faculty Senate committee, and Dee Fink would be pleased to be available in person to discuss the ideas and recommendations of the report with any faculty group wanting to review their own procedures for the evaluation of teaching.

There are few tasks at a university more important than the evaluation of teaching. Without it, professors themselves are unable to determine the direction of needed improvement, and they thereby become vulnerable to the process of stagnation. Without it, academic units are unable to identify and encourage professors who truly are effective with their students in the classroom.
Yet despite the importance of evaluation, academic organizations find themselves still struggling to find a satisfactory approach to this activity. This concern led the Faculty Senate at the University of Oklahoma to constitute a committee in February 1983 with the charge of reviewing the methods currently used to evaluate teaching on this campus and, if necessary, to propose an alternative system of collecting information that would provide a better base for personnel decisions and for the improvement of teaching.

After studying the problem at length, the committee eventually came to the conclusion that better evaluation would require two fundamental adjustments. The first was to establish the capability of examining multiple dimensions of teaching, something more than just what the teacher does in the classroom. The second was to deal with the need for multiple sources of information, something more than an exclusive reliance on student evaluations of teachers.

The following report describes the reasons for believing that these two adjustments are fundamental to effective evaluation. Some guidelines are also presented for academic units that wish to establish a set of procedures that take into account the multiple dimensions of teaching and multiple sources of information.

The Nature of Teaching

Before evaluating teaching, one must develop a clear concept of what is to be evaluated. For purposes of evaluation, teaching can be defined as"helping someone else learn something."
To advance this definition one step further, good teaching can be considered"being effective in the process of helping someone else learn something significant." The two added elements of effectiveness and significance both seem necessary to warrant the label of "good teaching."
The act of teaching can also be viewed as an interactive process that involves a teacher and students. This interaction occurs within a context or environment that can influence the success of that interaction. This is illustrated diagrammatically in Figure 1.

This definition of good teaching and the interactive character of teaching have a number of implications for evaluation.

(1) For purposes of evaluation, the primary purpose of teaching is to generate as much significant learning as possible. Students and teachers may bring additional purposes to the classroom, but for purposes of evaluation, the main concern is the amount of significant learning generated.

(2) The teacher is an important but indirect factor in the process of learning. This is simply a recognition of the fact that it is the student who does the learning; the teacher's role is to help the student in whatever ways possible.

(3) In higher education, the teacher has primary responsibility for key decisions about a course. These decisions include such things as determining the scope of a course, identifying the educational goals, selecting reading materials, constructing tests, and assigning grades.

(4) The quality of teachers' classroom behavior also has a major effect on students' reaction to the course on a day-to-day basis. This refers to such characteristics of teachers as develop with students, and the degree to which they are organized and prepared for class on a regular basis.

(5) Teaching takes place within several kinds of contexts, all of which can have a significant influence on the quality of the teaching and the learning. Examples include the following:
(a) Physical-the characteristics of a classroom and the time at which a course is scheduled.
(b) Social-the relationship between the teacher and the students is an interactive one. Students can inspire or discourage the teacher and vice versa.
(c) Institutional-the attitude and actions of the department and the larger institution. Do their attitudes and actions encourage or discourage good teaching?
(d) Personal-The situation of the teacher's non-professional life. Has there been an illness, divorce, or financial problems?
In summary, teaching can be viewed as an interactive process that takes place within several types of contexts for the purpose of generating as much significant learning as possible.

The Nature of Evaluation

The type of evaluation appropriate for use in higher education is four dimensional. It calls for an examination of the input, the process, the product, and the context of an event or action. When this general framework is applied to the specific situation of college courses it results in the five items identified in Figure 2.

The first dimension of teaching evaluation is the teacher's input, his or her knowledge of the subject matter. A teacher who is up-to-date in his or her field and has undertaken the required research and preparation for a class provides the input necessary for significant learning. It follows that the breakdown of this input component diminishes the learning processes.

The second dimension of evaluation, the "process," involves two separate activities in college teaching: course decisions and classroom behavior. When a professor teaches a course, he or she makes decisions about the scope of the subject matter to be covered, the teaching strategy to be used, the grading system course policies, etc. In these activities, teachers need to give consideration to a variety of factors (the nature of the curriculum, the characteristics of the students, etc.) and design the course accordingly. Also part of the "process" of teaching, but quite different in nature, is what a professor does in the classroom. Once the basic course decisions have been made and the professor steps into the classroom, he or she must engage all available communication and interaction skills to deliver lectures, lead discussions, ask questions, motivate students, and generate interest.

The third dimension is concerned with the "product." In college teaching, this is amount and type of learning that occurs in a given course.

The fourth dimension is context. In college teaching, there are several types of contexts that affect the quality of a given case of teaching: physical (e.g., the characteristics of the classroom), social (e.g., the nature of the students), institutional (e.g., the support given to teaching), and personal (e.g., other events in the life of the teacher).

What then are the questions that have to be answered in order to make confident and valid judgments about the quality of teaching? The five general questions and related sub-points shown below seem applicable to all classroom teaching in a university setting. The manner in which answers are found to these questions will vary from department to department and from college to college, but the questions themselves are inherent in the nature of teaching and in the nature of evaluation.

I. Does the teacher have adequate and up-to-date knowledge of the subject matter?
_ Academic and/or practical experience
_ Efforts to improve

II. How good were the teacher's decisions about the course?
_ Goals
_ Teaching strategy
_ Reading/laboratory/homework assignments
_ Testing
_ Course grading

III. How well did the teacher's classroom behavior promote good learning?
_ Organization and clarity
_ Enthusiasm
_ Interaction with the class as a whole
_ Relationships with individual students

IV. How good were the educational results of the course?
_ Amount of learning
_ Significance of what was learned
_ Attitude towards learning more about the subject

V. How much was the quality of the teaching and learning influenced by contextual factors?
_ Physical context
_ Social context
_ Institutional context
_ Personal context

Evaluating the Quality of Teaching

The quality of teaching, therefore, can be conceptualized as consisting of five components: the teacher's knowledge of the course subject matter, the teacher's course decisions, the teacher's classroom behavior, the amount of significant learning, and the influence of contextual factors. In order to effectively evaluate any particular instance of teaching, one must engage in the task of collecting and analyzing information about each one of these components.

However, the Faculty Senate committee also concluded that no single source of information was adequate for assessing all five components of teaching­multiple sources of information are not only advisable but are in fact necessary. Therefore the committee decided to investigate different information sources and to assess each one to determine its relative value for answering questions about these five components.

The committee then constructed a two-dimensional table to relate the components of teaching to the several sources of information. This table identifies six basic sources of information: course materials, three types of students, the teacher, the teacher's peers, administrators, and instructional consultants. The table can be used to decide which source or combination of sources would be best for examining each component of teaching.

However, the committee concluded that a separate chart should be used for each of the three evaluation situations common in academic settings: annual personnel decisions, periodic personnel decisions (e.g., promotion and tenure), and faculty self-improvement. These three kinds of evaluations have some degree of similarity, but the differences are sufficient to warrant separate consideration.

Figure 3 shows the two-dimensional table with the recommended sources of information for annual personnel decisions.

Annual Decisions
Each academic unit, through its executive committee, must evaluate each faculty member in the unit on an annual basis. The committee must discern how well each faculty member taught that year compared with other faculty members in the unit. For annual decisions about the quality of that faculty member's teaching, the committee suggests the use of five of eight information sources, including course materials, present students, the individual teacher, peers, and applicable administrators.

We exclude the use of senior students (e.g., exit surveys) and alumni sources on practical grounds. It would be difficult, if not impossible, for an executive committee to collect information annually from these students or former students and to use that information in evaluating each and every unit faculty member. We additionally recommend not using instructional consultants for personnel decisions, annual or periodic. These consultants are generally adamant about restricting their role to that of aiding teaching improvement.

The following summarizes the Faculty Senate committee's conclusions on sources of information for the evaluation of teaching for annual personnel decisions:

(1) Teacher's Knowledge: Teacher's comments (personal experience, formal training, professional reading, research, conferences, continuing education), peers and administrators, and course materials will be used.

(2) Course Decisions: Course materials (course syllabi, textbooks or textbook list, handouts, exams) will be the primary source. These may be supplemented with student evaluations, teacher's comments, and peer and administrator comment.

(3) Classroom Behavior: Student evaluations (evaluations or summaries) will be the primary source. These may be supplemented with teacher's comments, and peer and administrator comments (classroom visitation, informal comments, etc.).

(4) Learning: Selected course materials (graded exams, papers) will be the primary source. Student evaluations might also be used with such questions as: Did the student learn the subject matter? Did the student achieve the stated goals of the class? Additional sources could include teacher's comments of peers and administrators.

(5) Contextual Factors: Present students, the individual teacher, and comments from peers and administrators will be used.

Periodic Decisions

All university faculty and administrators realize the importance of periodic personnel decisions. In such cases, evaluators must decide whether a faculty member's teaching during the applicable period was sufficient to warrant academic tenure, a promotion, or a teaching award. When academic units assess teaching as part of a tenure or promotion decision, they must call upon all possible sources for credible information. This committee, therefore, suggests that academic units, when evaluating teaching for periodic decisions, expand their inquiry to consider all practical sources. These sources include course materials, present students, senior students and alumni, the individual teacher, peers, and administrators. We, again, exclude the use of instructional consultants for personnel decisions.

The following summarizes the committee's recommendations regarding sources of information used for evaluation of teaching for periodic personnel decisions.

(1) Teacher's Knowledge: Teacher's comments (personal experience, formal training, professional reading, research, conferences, continuing education), peers and administrators, and course materials will be used as primary sources. They may be supplemented with comments from alumni students and outside peer evaluators.

(2) Course Decisions: Course materials (course syllabi, textbooks or textbook lists, handouts, exams) will be used as the primary source. They may be supplemented with student evaluations, exit/alumni surveys, teacher's comments, and peer and administrator comments
.
(3) Classroom Behavior: Student evaluations, (evaluations or summaries) will be the primary source, supplemented with exit/alumni surveys, teacher's comments, and peer and administrator comments.

(4) Learning: Selected course materials (graded exams and papers) will be the primary source. Student evaluations (with such questions as: Did the student learn the subject matter? Did the student achieve the stated goals of the class?), exit/alumni surveys (affective learning, professional enhancement, etc.), teacher's comments, and comments of peers and administrators will also be used.

(5) Contextual Factors: Present students, teacher's comments, and comments from peers and administrators will be used.

Self-Improvement

The University and each academic unit bear the responsibility of helping each faculty member develop his or her professional skills. All teaching faculty should be interested in what they can do to improve their teaching. To that end, both the university and the academic unit can provide resources and information for faculty.
The committee suggests the university faculty consider several readily available sources that provide credible information on teaching improvement. These include present students, peers, administrators, and instructional consultants. Faculty members stand to gain significantly from the guidance and support provided by OU's Instructional Services. It is important that academic units inform their faculty about the availability of this support function.

Conclusions

This committee's aim was to develop a system for the evaluation of teaching that goes beyond the current emphasis on student evaluations of teaching and to recommend specific procedures for use by academic units and by faculty members themselves.

It is the committee's hope that all academic units will find it possible to review the procedures currently used to evaluate teaching and to decide whether or not improvements are needed. The specific recommendations in this report are offered as a basis for discussion, not as a package that must be adopted or rejected in toto. However, the underlying principles of multiple dimensions of teaching and multiple sources of information are seen as applicable to all evaluation situations. The question then becomes one of how these principles can best be applied to the teaching context of a particular department or college. If the principles are effectively applied, the evaluation of teaching may still not be perfect in all cases, but the committee believes it is likely to be significantly improved.