READY icon 2013.png
Remodeling Session - SI2013
.....The 3Rs for NCEES Observers....

Click here for the Interactive Agenda.

Click here for the full presentation.

You may also access materials at:


Session Overview/Purpose:

To ensure that all observers are on the same page in their ability to identify and rate observations with consistency, Understanding Rater-to-Rater Reliability will help observers take the time to view teacher practice together and then discuss what each person observed. Two of the techniques used to develop rater-to-rater reliability are the use of video observation and providing evidence for marks made by each observer.

Session Description:

During the facilitated training, groups of observers will use a video to view and rank teachers on chosen elements and then participate in discussions about the evidence they collected as well as why teachers were rated at a certain level on the scale. As a practice for continued improvement, teams of observers will use collaborative discussion to grow their rater-to-rater reliability. To do this, teams of observers, will view a teacher’s lesson with the intent to look for evidences of a specific element. The observers mark the evidences and then discuss what they saw together. Again, they talk about the evidences they collect and the rationale they use to assign the score. The intent of using the common lesson is for rater-to-rater reliability only and is not intended to give feedback to the teacher.

Learning Outcomes:

By the end of Understanding Rater-to-Rater Reliability, participants will be able to:
  • Identify and rate observations with consistency through the use of a video observation
  • Rank teachers on chosen elements
  • Participate in discussions about the evidence they collected during the observation and the rationale they use to assign a score

Session Objectives:

Participants will:
  • Explore the North Carolina Educator Evaluation System wiki to gain three new resources:
  • Identify and locate the standards and elements from the North Carolina Teaching Standards
  • Watch a lesson and complete the NCEES rubric for an observation
  • Share the marks and evidence collected during the observation
  • Review the "summary data" collected from the observers
  • Compare and defend the evidence that supports the marks with their colleagues to determine rater-to-rater reliability and consistency
  • Access, review and discuss the sample 'fillable' rubric created for this observation