An Investigation of the Comparability of Commission-Approved Teaching Performance Assessment Models

Final Report – Volume I: Technical Report

Prepared
for:
California Commission on Teacher Credentialing
1900 Capitol Avenue
Sacramento, CA 95811
Attn: Michael Taylor
Editors: Andrea L. Sinclair
Arthur Thacker
Date: December 31, 2019

Cover page showing the report title, HumRRO branding, and a classroom photo.

Acknowledgments page header with HumRRO logo and report text.

Executive Summary page header with the report title and opening summary text.

Executive Summary continuation with claim bullets and overview text.

Executive Summary continuation with Activity 1 and Activity 2 summary text.

Executive Summary continuation with Activity 4 and Activity 5 summary text.

Executive Summary continuation with Activity 6 and Activity 7 summary text.

Executive Summary continuation with summary and recommendations by claim.

Executive Summary continuation with claim 3 and claim 4 recommendations.

Executive Summary continuation with claim 6 and claim 7 recommendations.

Executive Summary continuation with practical implications and future research.

Executive Summary continuation with future research bullets.

Executive Summary conclusion text.

Table of contents page showing chapters and page numbers.

Table of contents continuation showing later chapters and references.

List of tables page in the table of contents.

List of tables continuation and list of figures page.

List of figures continuation page.

List of figures continuation and introduction heading.

Introduction

California’s Commission on Teacher Credentialing (Commission) requires all programs of preliminary multiple and single subject teacher preparation to use a Commission‐approved Teaching Performance Assessment (TPA) as one of the program completion requirements for prospective teacher candidates. In conformance with applicable California statute, multiple TPA models are allowed across the state as a program completion requirement for prospective teacher candidates. There are three TPA models approved by the Commission. They are:

the FAST (Fresno Assessment of Student Teachers), owned and operated by Fresno State; and
the CalTPA (California Teaching Performance Assessment), originally developed by Educational Testing Service (ETS) and owned by the Commission, revised by a Design Team with an operational contractor of the Evaluation Systems group of Pearson; and
the edTPA, owned by Stanford University, with an operational contractor of the Evaluation Systems group of Pearson.

Each TPA model must meet the Commission’s Assessment Design Standards (adopted December 2015) and measure the Commission-adopted Teaching Performance Expectations (adopted June 2016). The Assessment Design Standards (ADS) describe the design requirements for all TPA models, as set forth by the Commission. The Teaching Performance Expectations (TPEs) describe the performance standards for beginning teachers. There are six TPE “domains” (e.g., TPE domain 1: Engaging and Supporting all Students in Learning) and each domain includes six to eight descriptors, referred to as “elements,” which describe the knowledge, skills, and abilities (KSAs) required for beginning teachers.

Although each TPA model is Commission-approved, they differ in several important ways (e.g., design of candidate tasks, scoring rubrics, teaching performance elements measured by tasks). These inter-model differences raise questions regarding the comparability of results obtained by teacher candidates completing the various TPAs. Consequently, the Commission contracted with the Human Resources Research Organization (HumRRO) to conduct an external, independent investigation of the comparability of the three TPA models. The investigation occurred between June 2017 and December 2019.

Given the Commission’s adoption of the ADS in December 2015 and the TPEs in June 2016, each of the models underwent revisions. The FAST and CalTPA models have required more extensive revisions than edTPA, constituting the need for pilot testing in 2016–17 and field testing in 2017–18. Thus, Year 1 (2017–18) of the HumRRO investigation focused on the models as they were being revised (i.e., field tested for CalTPA and FAST) and Year 2 (2018–19) focused on the operational models (i.e., post-field test).

https://www.ctc.ca.gov/docs/default-source/educator-prep/tpa-files/tpa-assessment-design-standards.pdf
https://www.ctc.ca.gov/docs/default-source/educator-prep/standards/adopted-tpes-2016.pdf

Introduction page with the opening report text and footnote links.

Technical Approach

Our goal was to investigate the comparability of the three TPAs. Numerous techniques can be used to compare assessments. For example, an equating study is a rigorous technique to derive a function to map a score from one test onto the scale of another test, so that any given score has the same meaning regardless of which test was administered. However, equating requires that the tests being compared have either some common items, or that some people take both tests. Unfortunately, the TPA models do not share items nor examinees. Thus, the objective of this investigation was to compare the three TPAs on key aspects of test design, implementation, scoring, and reporting to create a body of evidence and thereby triangulate—that is, capture from different angles—whether the models are indeed comparable. The goal was to accumulate as much evidence as possible (i.e., a “body of evidence”) to evaluate the comparability of the three TPA models.

This investigation adopted a “Theory of Action approach” (Kane, 2006; 2013) to identify the claims that need to be substantiated to “assure that the Commission-approved TPA models are sufficiently comparable [emphasis added] that they are equitably assessing candidates working toward a California preliminary multiple or single subject teaching credential” (Request for Proposal, p. 5). This investigation was guided by a technical advisory committee (TAC) comprised of model sponsors and independent assessment experts. During the first TAC meeting the attendees engaged in a discussion of the meaning of “sufficiently comparable.” This discussion resulted in the following guidance: “comparable does not mean that the models are equal in how they measure the KSAs required by the TPEs, but that all models equitably identify TPE-ready professionals.” To assure that this ultimate objective is attained the following claims must be substantiated:

Claim 1: The TPA models are sufficiently comparable in their representation of the Commission’s Assessment Design Standards (ADS) and in their assessment and weighting of the Commission-adopted Teaching Performance Expectations (TPEs).
Claim 2: The guidance and supports (e.g., guide/manual/handbook and other resources) provided by model sponsors to candidates and teacher preparation faculty are sufficiently clear and detailed to ensure that the model is implemented as designed and intended.
Claim 3: The scoring rubrics for each TPA model are sufficiently clear and detailed to ensure that trained scorers can accurately and consistently score candidate submissions.
Claim 4: For each TPA model, there is a comparable, comprehensive process to select, train, and establish calibration of the assessors who score candidate submissions.
Claim 5: The standard-setting procedures used for each TPA model are sufficiently comparable and rigorous to ensure that the respective passing standards for each model accurately and consistently identify candidates possessing the requisite knowledge, skills, and abilities required to effectively teach the content area(s) authorized by the credential.
Claim 6: The model sponsor for each TPA model conducts statistical analyses to identify differential effects in relation to candidates’ race, ethnicity, language, gender or disability. Any differences are documented, and processes implemented to eliminate sources of construct-irrelevant variance.
Claim 7: For each TPA model, the score reports (candidate-level and program-level) provide similar information about candidate outcomes and include clear guidance on how candidate score information should be used.
Claim 8: The rubrics and score reports provide diagnostic information on candidates and on programs such that the strengths and weaknesses of each can be identified.

Seven distinct activities (studies) were designed to investigate these claims. The TAC provided guidance on the design, implementation, and interpretation of results for the seven activities. An overview of each activity and its results is presented next.

https://humrro.sharepoint.com/sites/Resource-Center/Shared%20Documents/Forms/AllItems.aspx?viewid=19a6c4bb%2D35f0%2D4527%2D80a8%2D5c332ed506b6&id=%2Fsites%2FResource%2DCenter%2FShared%20Documents%2FBrownbags%2FBB%20Schedule

Technical Approach page with claim list and TAC note.

Purpose of the Current Report

Over the course of this 2.5-year investigation, HumRRO submitted intermittent progress reports, a Year 1 preliminary report (Sinclair & Thacker, 2018), and this culminating document, the Year 2 final report. In the chapters that follow, we present the findings from the following seven activities that were completed for this comparability investigation:

Activity 1: Evaluation and Comparison of Evidence across TPA Models for Adherence to Assessment Design Standards (Chapter 1)
Activity 2: Content Validity Comparability Analysis (Chapter 2)
Activity 3: Comparison of Stakeholder Input across TPA Models (Chapter 3)
Activity 4: Scoring Review—Comparison of Scoring Rubrics, Scorer Training, and Score Reports across TPA Models (Chapter 4)
Activity 5: Comparison of Standard Setting across TPA Models (Chapter 5)
Activity 6: Statistical Analysis and Comparison of Score Data across TPA Models (Chapter 6)
Activity 7: Comparison of TPA Models to a Common Criterion Measure (Chapter 7)

Purpose of the Current Report page with activity list.

Chapter 1: Evaluation and Comparison of Evidence across TPA Models for Adherence to Assessment Design Standards (Activity 1)

Emily Dickinson, Andrea Sinclair & Justin Paulsen

Introduction

Activity 1 is as an overarching investigation, via a documentation review, of the eight claims identified in the Introduction. Activity 1 most directly addresses Claim 1, “The TPA models are sufficiently comparable in their representation of the Commission’s Assessment Design Standards . . .” Activity 1 involved a comprehensive review and comparison of the documents and materials developed by each model sponsor. This evidence was reviewed, and evaluations were made regarding the strength of evidence for adherence to the Assessment Design Standards (ADS) and The Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014; hereafter referred to as the Joint Standards). The latter evaluation was conducted to ensure that the documentation and materials for each TPA model also satisfy industry-wide principles for test design and development.

This same activity was conducted in Year 1 (2017-2018) of the Comparability Study. Because the models were undergoing revisions in 2017-2018, particularly for FAST and CalTPA, the findings presented in the Year 1 report (Sinclair & Thacker, 2018) were presented as preliminary. The revised models became operational in 2018-2019. Thus, Activity 1 was updated in Year 2 of the Comparability Study (2018-2019) and the findings presented herein represent the final evaluation for Activity 1 of the Comparability Study.

Method

Year 1 Process (2017-2018)

Using the ADS as the guiding framework, the HumRRO project team created a list of documentation and materials to request from each of the model sponsors to allow us to make determinations about the strength of evidence for adherence to the ADS. Each model sponsor was emailed an Excel spreadsheet that included the list of requested materials. They were asked to use the spreadsheet to identify the specific documentation that addressed each element of our request; a column was provided next to the request in which the model sponsor was asked to enter the names of the documents (or weblinks where the information could be found) that addressed each request; the spreadsheet included a column for the model sponsor to enter any additional comments or explanations about the documentation they provided. A column was also provided to allow for explanations of why particular documentation may not be available at this time. The model sponsors were given the option to upload the requested materials to HumRRO’s secure ftp site or to use their own secure site for posting any secure materials.

One key project staff member was assigned to review the documentation and materials provided by each model sponsor. That person then mapped the documentation and materials onto each of the ADS to which the materials were relevant. This mapping was then shared with each of the model sponsors to ensure that (a) we had accurately captured the information they provided and (b) to give the model sponsors the opportunity to provide any additional information that may be relevant to addressing the ADS. After receiving feedback from the model sponsors and any additional information that was provided with that feedback, the key staff member assigned to each TPA model proceeded to review the materials and make evaluations about the strength of evidence for each ADS. This same process was repeated for the relevant test design and development Standards from the Joint Standards.

Chapter 1 introduction and method opening.

Year 2 Process (2018-2019)

The Activity 1 chapter from the Year 1 report was shared with the model sponsors. Model representatives were asked to review the evaluations about the strength of evidence for each ADS and Joint Standard (along with the supporting rationale for the rating) and provide any updated and/or additional documentation/evidence for Year 2.

Ratings

Evaluations in both Year 1 and Year 2 were made using the rating scale shown in Table 1.1. All ratings were supported by a corresponding rationale. All key staff members responsible for evaluating each model were trained and calibrated on the rating scale. The HumRRO project director then conducted a cross-check on all ratings and rationales; any discrepancies were discussed with the rater to reach a consensus.

Table 1.1. Rating Scale for Strength of Evidence

Table 1.1. Rating Scale for Strength of Evidence
Rating Level	Description of Rating Level
1	No evidence of the Standard/element found in the documentation provided.
2	Little evidence of the Standard/element found in the documentation; less than half of the Standard/element covered in the documentation and/or evidence of key aspects of the Standard/element could not be found.
3	Some evidence of the Standard/element found in the documentation; approximately half of the Standard/element covered in the documentation including some key aspects of the Standard/element.
4	Evidence in the documentation mostly covers the Standard/element; more than half of the Standard/element covered in the documentation, including key aspects of the Standard/element.
5	Evidence in the documentation fully covers all aspects of the Standard/element.

Activity 1 method and rating scale continuation.

Download the PDF

This PDF is provided as a print-only reference document and may not be fully accessible to users of assistive technology. For more information, please visit Title II Rule on Digital Accessibility page.