Inaugural Annual Report on the Commission Approved Teaching and Administrator Performance Assessments

February 2022
3A
Information
Educator Preparation Committee

Executive Summary: This agenda item provides information and score results for the past three years of implementation of Commission-approved performance assessments—the California Teaching Performance Assessment (CalTPA), the edTPA, Fresno Assessment for Student Teachers (FAST) and the California Administrator Performance Assessment (CalAPA).

Recommended Action: For information only

Presenters: Mike Taylor, Zoltan Sarda, and Gay Roby, Consultants, Professional Services Division, Nicole Merino, Director of Early Learning and Teacher Performance Assessment, Stanford Center for Assessment, Learning, and Equity (SCALE), Ricci Ulrich, Coordinator, Fresno Assessment of Student Teachers (FAST), and Juliet Wahleithner, Assistant Director of Teacher Education, Fresno State University

Strategic Plan Goal
I. Educator Quality
b) Develop, maintain, and promote high quality authentic, consistent educator assessments and examinations that support development and certification of educators who have demonstrated the capacity to be effective practitioners.

Introduction

This agenda item provides information and score results for the past three years (2018-2021) of operational implementation of Commission approved performance assessments including the California Teaching Performance Assessment (CalTPA), the edTPA, Fresno Assessment for Student Teachers (FAST), and the California Administrator Performance Assessment (CalAPA).

The item is organized into five parts:

• Part One: Background and Overview of the Teaching Performance Assessments (CalTPA, edTPA, and FAST)
• Part Two: Teacher Candidate Score Results and Analysis
• Part Three: Background and Overview of the Administrator Performance Assessment (CalAPA)
• Part Four: Administrative Services Candidate Score Results and Analysis
• Part Five: Performance Assessment Implementation Information and Next Steps

Part One: Background and Overview of the Teaching Performance Assessments

California Teaching Performance Assessment (CalTPA)

CalTPA Background

At its December 2015 meeting, the Commission adopted revised Teaching Performance Assessment Design Standards and directed staff to develop a Request for Proposals to identify a technical contractor to support Commission staff and an appointed design team of 21 California educators, to redevelop the CalTPA. Evaluation Systems group of Pearson (ES) was selected in February 2016 to serve as the technical contractor to support the CalTPA redevelopment project.

The Commission redeveloped the CalTPA for several key reasons, including alignment with the expectations of the state-adopted Common Core State Standards and the Next Generation Science Standards, as well as the needs of the Commission’s Accreditation Data System for an outcomes-based set of quality indicators. To assure scoring reliability, the CalTPA was designed to be centrally managed and scored and required the use of content-specific assessors. The redeveloped CalTPA became operational in fall 2018. For purposes of standard setting, all candidates who had complete cycle submissions submitted and scored at the provisional passing standard of 15 points for Cycle 1, and 17 points for Cycle 2 through April 25, 2019. In June 2019 the current passing standards for the CalTPA were approved by the Commission.

CalTPA passing standards are:

Cycle 1 (8 rubrics): A score of 19 points with one rubric score of one (1) allowed.
Cycle 2 (9 rubrics): A score of 21 points with one rubric score of one (1) allowed.

Since its inception in 2018, a total of 30,097 candidates have passed the redeveloped CalTPA (represented in the 2018-19, 2019-20, and 2020-21 program years) in order to be recommended for the Preliminary Teaching Credential.

CalTPA Overview

The CalTPA measures candidate competency in the California Teaching Performance Expectations (TPEs). The TPEs identify the knowledge, skills, and abilities that a teacher should demonstrate at the beginning of a teaching career. The CalTPA features two instructional cycles that general education teacher candidates complete during their preliminary preparation program. Both cycles require candidates to engage in a recursive four-step process of Plan, Teach & Assess, Reflect, and Apply (PTRA), and includes video components.

Instructional Cycle 1: Learning About Students and Planning Instruction

Instructional Cycle 1 requires a complete teaching cycle (PTRA) for one content-specific lesson that candidates develop and teach within a school placement. Cycle 1 focuses on developing one content-specific lesson for a class and for 3 focus students (English learner, student with an IEP/504 plan, and a student who has experienced trauma in or outside of school). The lesson plan is based on what the candidate learns about the students’ diverse assets and learning needs, including their prior knowledge, interests, and developmental considerations.

Candidates demonstrate how they select appropriate content-specific learning goal(s) and ELD goal(s), determine what they expect students to know and be able to do, and show how they assess student learning by checking for understanding during instruction.

In the teaching videos, candidates demonstrate how they establish a positive and safe learning environment, affirm and validate students’ cultural and linguistic backgrounds, provide social and emotional supports through positive interactions with students, and use resources and materials to promote content-specific age and/or developmentally appropriate higher order thinking skills. Candidates complete Cycle 1 by reflecting on their practice and determining how they will apply what they have learned about their students’ needs to future instruction.

Cycle 1 has 8 analytic rubrics and requires a score of 19 points to pass. The fee for Cycle 1 is $150.00.

Instructional Cycle 2: Assessment Driven Instruction

Cycle 2 also requires a complete teaching cycle (PTRA). This cycle emphasizes the interaction between standards, assessment, and instructional decision making. Candidates plan a series of content specific lessons (3 to 5) and 3 types of assessments for a class of students. In the video clips, candidates demonstrate (a) how they assess student learning and development of academic language, (b) how students use educational technology, (c) how they provide students with content-specific feedback, (d) how they assess student learning and use of age and/or developmentally appropriate higher-order thinking skills, and (e) students engaged in self-assessment.

Candidates demonstrate, based on the series of assessments, how they make informed decisions about which curricular goals or strategies need to be strengthened, how instructional strategies can be adapted or expanded depending on what students are understanding or misunderstanding during instruction, and reflect on how a particular sequence of instruction has been successful. If the candidate determines through multiple assessments that students need further content instruction, they reteach. If the students met the content-specific learning goals, the candidate provides an extension lesson. The re-teach or extension activity is submitted in a fifth 5-minute video clip. In this cycle, candidates provide feedback to their students about their content-specific academic performance(s) based on classroom informal assessment(s), student self-assessment, and formal assessment results.

Cycle 2 has 9 rubrics and requires a score of 21 points to pass. The fee for Cycle 2 is $150.00.

For further information about key features of the California Teaching Performance Assessment, please see the August 2018 commission agenda item, 2C: Update on the Redevelopment of the California Teaching Performance Assessment (CalTPA), pages 11-13.

CalTPA Analytic Rubric Scoring Process

Once a candidate submits their assessment evidence via Pearson’s electronic platform, trained and calibrated assessors score candidate submissions, returning a score report within three weeks of the submission window’s closing. Each of the CalTPA analytic rubrics consist of a five-point scale. With 8 rubrics, Cycle 1 has a possible score total of 40 points, while Cycle 2, with 9 rubrics has a possible score total of 45 points. Each rubric has an essential question, outlining the specific teaching performance expectations that will be assessed in that rubric. The five levels of each rubric provide qualitative descriptions of performance based on the TPEs measured in the cycle. Should a candidate provide exactly the evidence required in the essential question, they receive a score of “3.” Providing additional evidence, as outlined in levels 4 and 5, allows the candidate to improve their score, while providing off-target or incomplete evidence results in a lower score of 1 or 2. Appendix C provides rubric descriptions.

Condition Codes

Submissions that have not followed the instructions provided in the Assessment Guides are assigned a condition code for the appropriate rubric(s). Each condition code describes the aspect of the submission requirements that was not followed according to the outlined procedures. Condition codes are described publicly on the CalTPA Assessments Policy web page (CalTPA Assessment Policies). If a candidate receives a condition code, the candidate must address and correct the error, and then resubmit the cycle in order to obtain scores. Programs are provided data on condition codes for their candidates with each reporting date, so that support and/or alterations in program delivery can be offered, with the goal of eliminating condition codes and increasing candidate performance.

Assessor Recruitment, Training, and Support

Starting in fall 2017, Evaluation Systems began recruiting multiple subject and single subject, content specific assessors and identified lead assessors for the field test scoring process. Assessors must meet two eligibility requirements: first, they must be a California education professional in one or more of the following capacities: University/program educator providing instruction to TK–12 teacher candidates within a Commission-accredited teacher preparation program, field supervisor, mentor or cooperating teacher, TK–12 teacher, TK–12 administrator (e.g., principal, assistant principal), or National Board-Certified Teacher (NBCT). Second, they must have expertise in the content area assigned to score in one or more of the following ways: hold a current California Clear Multiple Subject or Single Subject Teaching Credential, or added authorization, in the content area, have university teaching experience in the content area, or hold a degree in the content area.

Assessors are divided into two groups: one group is trained to score Cycle 1 and a second group to score Cycle 2. Trainings include a mix of online and in-person experiences. Lead assessors are identified from the pool of assessors. Lead assessors participate in selecting marker papers, facilitated by Commission and ES staff, and are trained to support the facilitation of the ongoing assessor trainings. To support assessor participation, trainings were held in southern and northern California prior to COVID restrictions. During the pandemic, assessor training temporarily moved to an online approach.

Prior to the synchronous training, assessors are required to attend an online orientation facilitated by Commission and ES staff that provides an overview of the instructional cycle, evidence to be collected, associated analytic rubrics, and the Commission’s Using the Tools of Cultural Proficiency to Address Implicit Bias training. Assessors then attend a two-day synchronous training where lead assessors facilitate a review of five submissions representing a range of performance. Assessors are required to independently score a calibration submission and meet the calibration criteria. Once notified they have met the calibration criteria, assessors begin scoring candidate submissions using Pearson’s secure online scoring system. Assessors who do not meet the calibration requirement are provided coaching by lead assessors and given the opportunity to score an additional calibration submission.

CalTPA submissions are scored, as needed, by up to three assessors. Scoring quality is monitored on an ongoing basis, including use of the following metrics to monitor ongoing assessor calibration:

• Inter-rater reliability — agreement rates between assessors on double-scored submissions.
• Validity submissions — pre-scored submissions are sent to each assessor. Assessors are not aware that they are scoring a validity submission (i.e., blind scoring) and their performance is monitored on an ongoing basis. Assessors who do not meet the established agreement rates are flagged for additional review.
• Backreading — supervisors and lead assessors monitor and read-behind assessors as submissions are scored. Supervisors and lead assessors intervene and remediate assessors on any areas identified.

Table 1 outlines the current number of assessors scoring Multiple Subject and Single Subject submissions. ES is in the process of recruiting and training additional assessors for spring 2022 in anticipation of increased submission numbers.

A teacher candidate is required to pass both Cycles 1 (19 points of 40) and 2 (21 points of 45). Passing the performance assessment, required coursework, and clinical practice qualifies the candidate for the preliminary teaching credential.

CalTPA Program and Candidate Supports

Commission-approved Program Standard 5 for Preliminary Multiple Subject and Single Subject Credential Program Standards outlines the required candidate supports programs are required to build into their program design and implement on an ongoing basis. Accreditation activities evaluate the degree to which an institution is aligned to the standards, assigning additional timelines should the alignment be less than required. Some examples of the requirements include making assessment materials accessible to candidates, appropriate practicum placement opportunities, supporting candidates in reviewing and resubmitting evidence, and using data to improve program design and implementation.

To further support candidates and programs, Commission and ES staff have developed a series of outreach opportunities that occur at regularly scheduled intervals. These opportunities include candidate and program office hours, quarterly program coordinator meetings, a variety of webinars focused on best practices in supporting candidates through completion of the CalTPA, and an annual performance assessment implementation conference designed for programs to share evidence-based practices.

For more detail, please see the table outlining candidate and program supports offered by Commission staff and ES in Appendix A.

edTPA

edTPA Background

edTPA was created for the profession by the profession, with input from subject-specific design teams made up of teachers and teacher educators in a process led by Stanford University. The edTPA is owned by Stanford University and engages the Evaluation Systems group of Pearson as its operational partner. Stanford University exclusively owns all the intellectual property rights and trademark for edTPA and is responsible for all edTPA development including candidate handbooks, scoring rubrics and the scorer training design, scorer training curriculum, and materials as well as support materials for programs, faculty, and candidates. SCALE also recruits, reviews, trains, and endorses National Academy consultants who act as support providers within the edTPA community. Each contributor plays an important role in the design, development, and ongoing operations of edTPA. edTPA was initially approved by the Commission for use in California as a Teaching Performance Assessment in 2014 and again in 2016, having met the requirements of the revised (2015) Teaching Performance Assessment Design Standards (ADS), including meeting the requirements to measure candidate competency of the revised (2016) California Teaching Performance Expectations (TPE). The edTPA TPEs identify the knowledge, skills, and abilities that a teacher should demonstrate at the beginning of a teaching career.

edTPA Overview

edTPA is a subject-specific performance assessment that evaluates a common set of teaching principles and teaching behaviors as well as pedagogical strategies that are focused on specific content learning outcomes for P–12 learners. An extensive Review of Research on Teacher Education provides the conceptual and empirical rationale for edTPA’s three-task design and the rubrics’ representation of initial competencies needed to be ready to teach. The assessment examines an authentic cycle of teaching aimed at subject-specific student learning goals, using evidence derived from candidates’ practice in their student teaching or internship placement. Across all three tasks, candidates are assessed on their ability to develop, collect, and submit authentic and job-related evidence – including lesson plans, instructional materials, student assignments and assessments, feedback on student work, and unedited video recordings of instruction. Also assessed through the three tasks are candidates’ abilities to develop their students’ academic language and to justify and analyze their own teaching practices.

A cycle of teaching, captured by the three tasks that compose an edTPA portfolio, include 1) Planning: Developing 3-5 content-specific lessons for a class, including 3 focus students, 2) Instruction: Teaching and video recording their lessons and interaction with students, and 3) Assessment of student learning: Monitoring student learning and providing evidence of feedback.

In each task, candidates have an opportunity to reflect and analyze their teaching and propose revisions, considering their knowledge of students and data collected (e.g., monitoring student learning). In Task 3, assessment, feedback, and support for applying feedback includes consideration of the strengths and needs of three focus students. Consistent with the 2016 TPEs, California candidates are to include focus students meeting the following categories: English learner, a student with an identified disability, and a student from an underserved education group or a group that needs to be served differently.

To assist candidates and program with determining which handbook to use, edTPA created an “edTPA for California” webpage.

The edTPA Elementary Education Handbooks assess candidates’ performance of both Elementary Literacy and Elementary Mathematics. The Elementary Education Handbooks follows the edTPA common architecture with Tasks 1–3 assessing candidate performance on Education (e.g., the Elementary Literacy Handbook), and includes an additional Task 4 assessing candidate performance on Elementary Mathematics or Elementary Literacy. To assess Task 4, the handbook includes an additional three rubrics that focus on the candidates’ ability to assess students’ mathematical or literacy learning and to plan and support the re-engagement of students in a focused learning experience. Both versions of the Elementary Education Handbook are currently approved for use in California and used in programs across the state.

The fee for submitting an edTPA portfolio is $300.00.

edTPA Analytic Rubric Scoring Process

Candidate responses on and edTPA portfolio are scored according to a total of 15 rubrics (5 per task) except for the Elementary Education Handbook, which is scored according to a total of 18 rubrics (three additional rubrics addressing Task 4). The World Language/Classical Language Handbooks, have 13 rubrics each. All rubrics are on a five-point scale with “1” being the lowest and “5” the highest and level 3 representing the knowledge and skills of a candidate who is qualified to teach. Across the five levels, each rubric represents a learning progression or trajectory of practice that evaluates a range of performance. The progression addresses candidates’ expanding repertoire of field specific skills and strategies and their deepening rationale and reflection on practice. The rubrics address common outcomes across all fields and are uniquely adapted to address learning and pedagogy specific to each subject-specific handbook. Each rubric includes descriptors and address a wide range of performance. These descriptions are in Appendix E.

Condition Codes

Submissions that have not followed the instructions provided in the edTPA Submission Guidelines are assigned a condition code for the appropriate rubric(s). Each condition code describes the aspect of the submission requirements that were not followed according to the outlined procedures. Condition codes are described publicly with the edTPA Submission Guidelines. If a candidate receives a condition code, the candidate must address and correct the error, and resubmit in order to obtain scores. Programs are provided data on condition codes for their candidates with each reporting date, so that support and/or alterations in program delivery can be offered, with the goal of eliminating condition codes and increasing candidate performance.

Assessor Recruitment, Training, and Support

Because of the subject-specific nature of edTPA, trainers and scorers of edTPA submissions by California candidates must have 1) pedagogical and subject matter knowledge, including relevant experience in roles that support teaching and learning in the content area and grade level span in which they will score; 2) Recent experiences that make them aware of appropriate expectations for teacher candidates who are in the early stages of learning to teach. (University faculty and supervisors must have recent experience teaching methods or clinical supervision responsibilities in the subject matter area they will score. P–12 educators must have recent experience teaching P–12 learners the subject matter area they will score, as well as guiding or mentoring beginning teachers); and 3) a state-agency-issued license or credential to teach English Language Learners or must have experience with English Language Learners in the last ten years.

Scorers of the edTPA are drawn from a national “academy” of trained, calibrated assessors including university faculty and administrators, field supervisors, cooperating teachers, induction mentors/coaches, school site principals, National Board Certified Teachers, subject matter organization members, and retired teachers and principals who are current in their content field.

Scorer training and calibration is designed and developed by Stanford University; the implementation of candidate registration, portfolio submission, and scoring processes are provided by Evaluation Systems.

Each scorer participates in rigorous training to become qualified as an edTPA scorer. The scorers are carefully monitored during scoring activities to maintain high quality and consistency in scoring across portfolios. Scorers who are new to edTPA will complete the training curriculum composed of over 20 hours of independent, online training modules, as well as independent scoring and the opportunity to discuss any questions about scoring with a trainer before completing the qualification scoring exercises. Scorers who qualified and have scored within 120 days may continue scoring. Scorers who have qualified and been inactive (not scored within 120 days) will need to re-qualify, as detailed below. Scorers who began, but have not completed, training and qualification will need to resume the New Scorer process. A detailed description of scorer expectations and requirements for edTPA scorers is available here.

Scoring for edTPA occurs year-around. Given this ongoing scoring model, scorer quality monitoring is ongoing. Quality monitoring of edTPA is an integral part of the scoring process. The scoring quality control processes include activities such as calibration, on-going monitoring, and reliability checks. Additionally, all scorers must train and qualify on the prevention of bias (see Design Standard 1(h), Bias Prevention Scoring Module). For more information on edTPA’s Scoring Model and the technical aspects of scoring quality control, please see the 2016 edTPA Administrative Report.

Included as a part of annual program activities to review and improve scorer interaction and training, SCALE and Evaluation Systems plans and implements an annual review of the scorer training, which includes quantitative and qualitative feedback from scorers, scoring trainers, and scoring supervisors. As a result of this feedback, scorer training modules have been improved and delivered to scorers, and supplemental support resources have been developed to assist scorers with new program year startup activities such as training refreshers, handbook revision summaries, and other materials to support scorers in their activities.

Appendix F provides additional information regarding edTPA assessors.

edTPA Program and Candidate Supports

edTPA is intended to be embedded in a teacher preparation program and to be “educative” for candidates, faculty, and programs. Candidates deepen their understanding of teaching while preparing for edTPA and reflecting on their strengths and challenges as they move forward into the profession. For faculty and programs, the various edTPA resources and candidate, program, and campus results can be used to determine areas of program strength and curricular renewal. In addition, the professional growth plan resource uses edTPA results and other evidence of teaching to inform candidates’ goal setting for induction and the early years of teaching. The website (edtpa.org) also includes publicly available materials for various stakeholders. In addition to the website, edTPA offers a National Academy of experienced consultants available to provide professional development to new users and to network in a learning community across the country. Programs using edTPA are provided with a variety of tools and reporting formats to access, analyze, and make decisions about their own candidate performance data, as well as state and national summary reports. Lastly, edTPA offers a collection of videos and webinars to provide overviews of the edTPA handbooks and resources. See edtpa.org for more information and access to resources.

Fresno Assessment of Student Teachers (FAST)

FAST Overview

The Fresno Assessment of Student Teachers (FAST) is a Commission-approved TPA model designed by and used exclusively by Fresno State. FAST was born out of the Renaissance Teacher Work Sample developed through a Title II U.S. Department of Education grant in which Fresno State participated. The Teacher Work Sample was modified to measure California’s TPEs and became the Teaching Sample Project, one of two tasks required of candidates by FAST (Torgerson, Macy, Beare, & Tanner, 2009).

FAST was originally reviewed and approved by the Commission in 2007 and has been in use at Fresno State since that time. In 2018, the FAST was revised to align with changes in the Teaching Performance Expectations. FAST 2.0 was submitted for review and approved by the Commission.

Since its development, FAST has been an integral part of the teacher preparation programs at Fresno State for both faculty and candidates. Faculty are involved in the assessment of the FAST, which provides them with insider knowledge of the specific tasks candidates are asked to complete as well as areas where candidates excel and areas where they experience challenges. Faculty are then able to use this information to inform their instruction.

Candidates within Fresno State’s Multiple Subject and Single Subject programs are only given the option of taking the FAST to satisfy the teacher performance assessment requirement.

FAST 2.0 Overview

FAST 2.0 consists of two parts: the Site Visitation Project (SVP), completed during candidates’ initial student teaching, and the Teaching Sample Project (TSP), completed during candidates’ final student teaching. Each task in FAST 2.0 requires a complex performance that demonstrates a candidate’s level of competence related to at least two domains of the TPEs. Collectively the tasks measure key aspects of the six major TPE domains. The four-level task-specific rubrics articulate different dimensions of teaching and focus on the key qualities embedded in the TPE elements assessed. Rubrics are organized by sections aligned with specific TPE elements.

Site Visitation Project (SVP)

The SVP assesses teacher candidates’ ability to plan, implement, and evaluate instruction. The three parts of the project include (1) Planning: planning documentation for a single lesson incorporating state-adopted content standards and English language development, (2) Implementation: an in-person observation and videotaping of the teaching of the lesson, (3) Reflection: a review of the entire video, selection of a 3- to 5-minute video segment, and a written evaluation of the lesson.

For this project candidates are expected to plan, teach, and evaluate a lesson that is observed by their University Supervisor. The entire lesson should be completed within a 20- to 45-minute time frame. For Single Subject candidates, the focus is on their content area. For Multiple Subject candidates, the content area is mathematics. The lesson plan and instruction should address goals and skills in the content area and align with adopted California State Standards and curriculum frameworks. The lesson is observed by a University Supervisor and/or a subject-matter expert and videotaped. After watching the video of the lesson, the candidate evaluates their implementation across the entire lesson and selects a segment of the video to demonstrate their use of subject-specific pedagogy.

The three parts of the (SVP) are scored on three different rubrics: Lesson Planning, Observation of Lesson, Self-Evaluation of Lesson. Candidates are not required to pay any fees to take the SVP, as the administration and scoring of FAST II are embedded within the operating expenses of the Fresno State Credential Program.

Teaching Sample Project (TSP)

Within the TSP, teacher candidates document how they are addressing the needs of all their students in the planning, teaching, and assessing of the content. The TSP assesses candidates’ ability to (a) identify the context of the classroom, (b) plan and teach a series of at least five cohesive lessons (a unit of study) with a focus on content knowledge and literacy, (c) assess students’ learning before, during, and after the unit, (d) document their teaching and their students’ learning, and (e) reflect on the effectiveness of their teaching.

To prepare for this unit, candidates must be familiar with the California subject matter content standards for their discipline area; the California Common Core State Standards for English Language and Literacy in History/Social Studies, Science, and Technical Subjects (CA CCSS ELA/Literacy); the California English Language Development Standards (CA ELD Standards), and the English Language Arts/English Language Development Framework for California Public Schools: Kindergarten Through Grade Twelve (ELA/ELD Framework).The TSP is divided into seven components, each of which is scored on a task-specific four-point rubric. Candidates are not required to pay any fees to take the TSP, as the administration and scoring of FAST II are embedded within the operating expenses of the Fresno State Credential Program.

FAST Analytic Rubric Scoring Process

As stated above, both the SVP and the TSP have task-specific rubrics. Each rubric, specifically designed to align with the specific project component, consists of four score points. Candidates are required to score at least a 2 on each rubric in order to pass the project. If a candidate scores a 1 on any rubric, they do have the opportunity to revise and resubmit that task. For example, if a candidate scores a 1 on the Analysis of Student Learning component of the TSP, they are provided the opportunity to revise and resubmit just that component. Candidates cannot pass the TSP or the SVP until they score a 2 on each rubric.

Assessor Training

Both the SVP and the TSP are scored by at least two faculty members, including the university coach assigned to mentor the teacher candidate. Mandatory calibration sessions are held annually, and all scorers must participate in the norming process each year. The scorer training for each task makes a distinction between experienced and new scorers. All training sessions include assessor guidelines and bias training and have the opportunity for scorers to establish a common understanding of the particular task. Differentiation is made during the calibration and re-calibration phase of each session; experienced scorers have the opportunity to work independently to re-calibrate with marker papers or performances, while new scorers experience more in-depth, guided practice to establish initial calibration with scoring norms. Such practices help assure reliability in scoring while acknowledging the formidable experience of the majority of FAST scorers. The inter-rater reliability is higher than the norm for such assessments.

Reliability of Scores

Every two years, a psychometric analysis of the Site Visitation Project (SVP) is performed. Our most recent analysis found that, of the 15% of the SVPs that were double scored, 70% gave the same score and 100% were within +/-1. 94.7% agreed on the determination of whether the SVP should pass or not.

In an effort to examine scoring equity, a series of non-parametric statistical tests are also calculated to determine whether significant differences in scoring corresponded to students’ demographic (gender and ethnicity) characteristics. In the most recent analysis conducted in August 2021, there were no statistically significant differences found in gender and ethnicity in the final scores. Across the seven tasks of the TSP, there were no statistically significant differences found with regard to gender. However, there were differences related to ethnicity for 6 out of 7 of the subtests. Further examination of these differences, however, revealed that the amount of variance explained by this particular demographic variable was consistently very low, typically around 3% of total score variance.

Part Two: Teacher Candidate Score Results and Analysis

This section provides data on candidate performance for each of the Commission approved performance assessments (CalTPA, edTPA, FAST) over the last three program years.

CalTPA Results and Analysis

Table 2 illustrates the percentages of all Multiple Subject and Single Subject CalTPA candidates who have passed both Cycle 1 and Cycle 2 of the CalTPA. The N reflects the number of candidates who took Cycles 1 and 2 in the program year (all attempts, including retakers), with scorable responses on both cycles (i.e., no condition codes). The % reflects the percent of those candidates who passed both cycles (on any attempt) in the program year.

* Lower Passing Standard applied

Year 1 (2018-2019) pass rates were higher than subsequent years as the provisional passing score requirement was lower for that first year in anticipation of a passing standard setting study that occurred in 2019. Once operational passing standards were adopted by the Commission, pass rates slightly decreased. The passing rate for Year 2 was 91% while the passing rate for Year 3 stands at 89%. The lower passing rate for 2019-20 and 2020-21 is likely due to several factors. Candidates and programs were navigating COVID-19 impacts as instruction moved from in person to online and then to hybrid models with some students in the classroom and some online; faculty adjusting to instruction of teacher preparation from in-person to online; challenges of securing student teacher placements; student teachers moving into substitute teaching assignments; and the toll that COVID-19 has had on overall emotional well-being of faculty, supervising teachers, administrators, students, families/guardians, and teacher candidates.

The total number of candidates across the three program years has also declined each year. This decline in the number of candidates completing the CalTPA is likely due to the COVID-19 pandemic. Under the Governor’s Executive Order, candidates impacted by school closures and the interruption of synchronous instruction allowed programs to recommend candidates for the preliminary credential without passing a TPA. Credential data indicates that several thousand candidates (12,000+) did not complete the TPA in 2019-20 and 2020-21 academic years. The TPA COVID-19 Flexibility Executive Order remains in place for candidates until August of 2022. The downward trend in data is starting to reverse and turn upward with candidate submission numbers thus far for Year 4 (August-December 2021) indicating a return to the candidate submission rates of 2018-19.

Table 3: CalTPA: All Attempt Pass Rates by Cycle: All Submissions Without Condition Codes

* Lower Passing Standard applied

Table 3 illustrates passing rates by cycle across the three program years for all Multiple Subject and Single Subject submissions. Year 1 (2018-19) has a higher passing rate of 98% over the passing rates of subsequent years due to the provisional passing score requirement being lower for that year (Cycle 1 15 points, Cycle 2 17 points). Passing rates for Year 2 (2019-20) were 77% for Cycle 1 and 89% for Cycle 2. While these rates represent what is closer to what can be expected with the passing standard being set at 19 for Cycle 1 and 21 for Cycle 2, the figures for Year 2 were also influenced by the onset of the COVID pandemic, which likely limited some candidates’ abilities to submit, revise and resubmit cycles. Candidates who were successful with Cycle 1 may not have been able to successfully complete Cycle 2 and may have postponed completion due to school closures. Executive Order N-66-20 (May 2020) followed by SB 820 (September 2020) allowed programs to recommend candidates for preliminary credentials without passing a TPA. Year 3 (2020-21) data also reveals a significant reduction in the total number of attempts for both Cycle 1 and Cycle 2, which was again likely influenced by COVID school closures, complexities of teaching in online settings, and student teaching placements. Pass rates for Cycle 1 from Year 2 to Year 3 remained consistent (77% to 74%), as well as for Cycle 2 (89% to 87%), indicating that candidates who were able to submit, revise, and/or resubmit were able to pass at similar rates across years, even though the pandemic presented challenges to programs and teacher candidates.

Table 4: CalTPA First Attempt Score Means by Rubric*: Cycle 1 (MS & SS)

*Rubric descriptions are provided in Appendix C

The data in Table 4 indicates that most rubric score means for Cycle 1 have been relatively consistent cross all program years. The rubric in which candidates demonstrate the strongest performance is rubric 1.1, Plan, which measures candidates’ abilities to plan one lesson in which the learning goals and activities align with students’ prior knowledge and assets, and how they plan to support, engage, and challenge all students and three focus students to meet the content and ELD learning goals.

Rubric 1.6, focused on monitoring students’ deep learning of content and connecting to next steps for learning has shown consistently lower scores across all three program years. The analysis from assessors indicates that candidates struggle with the concept of next steps for content learning. Instead of addressing what content instruction comes next, candidates will often speak with students about the next activity that is coming up such as recess or the next assignment such as homework. Programs have received advice about supporting candidates with understanding the concept of next steps for content learning, and revisions to the CalTPA guidebook have added language to the prompts to more clearly guide candidates toward that understanding.

While particular attention has been paid to effective performance related to rubric 1.6, Commission staff has also focused efforts on supporting programs and candidates to improve scores across other Cycle 1 rubrics. These efforts include increasing understanding of asset-based instruction and the concept of integrating student funds of knowledge and cultural and linguistic resources into Step 1 - Plan rubrics (1.1, 1.2, 1.3, 1.4), connecting lesson content to prior learning for rubric 1.5, and developing candidates’ capacity for reflective writing while citing evidence to support claims (rubric 1.7). Cycle 1 revisions have also included reworking the prompts and rubric for Step 3 - Reflect to more adequately support candidates’ evidence-based reflection abilities.

Table 5: CalTPA First Attempt Score Means by Rubric*: Cycle 2 (MS & SS)

*Rubric descriptions are provided in Appendix C

In general, as illustrated in Table 5, candidate scores for Cycle 2 have been stronger across the rubrics than in Cycle 1, and scores across rubrics and program years have remained relatively consistent. Possible explanations for the relative strength and consistency of mean scores by rubric likely have to do with candidates’ levels of experience. By the time candidates begin Cycle 2, they have completed Cycle 1 and received feedback. In addition, candidates generally complete Cycle 2 later in their preliminary programs and deeper into their clinical practice (candidates complete 600 hours of clinical practice) and so have more teaching and planning experience and therefore more teaching experience which is reflected across the Cycle 2 rubrics.

Rubrics 2.1 and 2.2 - Plan have consistently shown mean scores near “3.” These rubrics measure the degree to which candidates are effective in developing a “learning segment” of three to five lessons, designing an array of informal, formal, and student self-assessments that are aligned to the defined standards, content, and ELD learning goals of the lessons.

Rubrics 2.3, 2.4 and 2.5 - Teach and Assess also demonstrate consistently strong candidate performance. These rubrics measure the candidate’s ability to support student development and demonstration of academic language, incorporating educational technology to provide opportunities for students to achieve or demonstrate content-specific learning goals, and using informal assessment to monitor student learning and adjust instruction as needed to meet the needs of their students.

Rubric 2.6 - Teach and Assess has consistently demonstrated lower candidate performance. This rubric assesses the candidate’s ability to engage students in self-assessment of their learning. While this is a critically important pedagogy to support students in deep reflective learning, it continues to be a somewhat unfamiliar concept in the field. Among the issues candidates have had in engaging students in self-assessment is their capacity to develop rubrics for students to use as they review their own product, process, or performance. Commission staff have provided programs and candidates with information about the possible components of student rubrics in quarterly coordinator meetings, webinars, and during candidate and program office hours. In addition, changes were made to the video clip requirement for this rubric. In prior versions, candidates were required to be seen providing students with feedback about their self-assessment. In Version 4.1, candidates can now document the feedback they provided in a written narrative or in the video clip, which provides them with more time in the self-assessment clip to model the assessment and have student(s) use the assessment.

While particular attention has been paid to rubric 2.6, Commission staff have also focused efforts on supporting programs and candidates to improve scores across other Cycle 2 rubrics. These efforts include increasing understanding of the use of educational technology (rubric 2.4) in ways that provide students with opportunities to enhance and improve their learning and/or demonstrate their knowledge, skills, and abilities beyond using technology for rote forms of learning. Support was also provided to guide the use of educational technology with young learners in Transitional Kindergarten to second grade settings. Efforts have also included supporting candidates to effectively provide students with “actionable” feedback based on assessment results (rubric 2.5) and increasing teacher candidates’ expertise in evidence-based reflection (rubric 2.8).

Table 6: CalTPA Number (N) of Submissions Receiving Condition Codes (CC) (MS & SS)

The analysis of condition codes that candidates have received over the program years has also been complicated by issues of the COVID-19 pandemic. During the first program year, the rate at which candidates received condition codes was at 2% across both cycles. During the 2019-20 program year, assignment of condition codes for Cycle 1 rose by 1%, while the rate for Cycle 2 increased to 9%. A possible explanation for this increase is that many candidates may have completed Cycle 1 early in their preliminary program in the fall of 2019 before schools began experiencing closures and moving to online learning and were engaged in completing Cycle 2 during the initial months of school closures, leading to candidate’s inability to complete Cycle 2.

For 2020-21, the rate remained at 8% for Cycle 1 and 13% for Cycle 2. Most of these condition codes were assigned due to the lack of visibility of students and candidates in the video clips.

Teacher candidates and students have struggled through the pandemic year engaging in learning in remote contexts. One aspect of this issue is the ability to see and hear one another across digital platforms. This problem represented a challenge for candidates to capture their teaching and student participation in video clips. Young learners were distracted, while middle and high-school students did not want to turn their cameras on during instruction. Throughout the period of online learning, Commission and Evaluation Systems staff provided information and resources to programs and candidates to help mitigate this issue including modification and clarification of expectations and providing examples of acceptable and unacceptable video clips during webinars, coordinator meetings, and office hours. In order to further support candidates in using video to demonstrate their teaching capacity, Commission staff and preparation program faculty have developed a webinar called “Digging Deeper: Evidence-based Best Practices in Performance Assessments” focused on using video techniques for professional learning and documenting teaching. The Commission’s “Digging Deeper” webinar series will offer performance assessment topics that programs have requested during office hours and emails.

The results for the 2021-22 program year so far have indicated a substantial decrease in the number of video clip related condition codes.

Figure 1 and Figure 2 in Appendix B provide the ethnic distribution of teacher candidates passing the CalTPA for years 2018-19, 2019-20 and 2020-21.

edTPA Results and Analysis

Table 7: edTPA First Attempt Pass Rates*

*Includes first attempts with a total score for handbooks with 15 and 18 rubrics.

Table 7 illustrates the percentages of edTPA candidates who have passed on first attempts. Year 1 (2018-19) and Year 2 (2019-20) pass rates are higher than Year 3 (2020-21), with Year 2 having the highest pass rate of the three years above. The lower passing rate for 2020-21 is likely due to several factors. As stated above, candidates and programs were navigating COVID-19 impacts as instruction moved from in person to online, and then to hybrid models with some students in the classroom and some online; faculty adjusting to instruction of teacher preparation from in-person to online; challenges of securing student teacher placements; student teachers moving into substitute teaching assignments, and the toll that COVID-19 has had on overall emotional well-being of faculty, supervising teachers, administrators, students, families/guardians and teacher candidates.

The total number of candidates across the three program years has also declined each year. This decline in the number of candidates completing edTPA is likely due to the COVID-19 pandemic. As stated above, under the Governor’s Executive Order, candidates impacted by school closures and the interruption of synchronous instruction allowed programs to recommend candidates for the preliminary credential without passing a TPA. Credential data indicates that several thousand candidates (12,000+) did not complete the TPA in 2019-20 and 2020-21 years. The TPA COVID-19 Flexibility Executive Order remains in place for candidates until August of 2022.

Table 8: edTPA All Attempt Pass Rates*

*Includes All attempts with a total score for handbooks with 15 and 18 rubrics.

Table 8 illustrates the percentages of edTPA candidates who have passed after all attempts. Year 1 (2018-19), Year 2 (2019-20) and Year 3 (2020-21), show similar passing rates. Year 3 (2020-21) shows a lower N. The lower N in Year 3 is likely due to several factors. Candidates and programs were navigating COVID-19 impacts as instruction moved from in person to online, and then to hybrid models with some students in the classroom and some online; faculty adjusting to instruction of teacher preparation from in-person to online; challenges of securing student teacher placements; student teachers moving into substitute teaching assignments, and the toll that COVID-19 has had on overall emotional well-being of faculty, supervising teachers, administrators, students, families/guardians and teacher candidates.

Table 9: edTPA First Attempt Score Means by Individual Rubric*

*Includes first attempts with a total score for handbooks with 15 and 18 rubrics.

Table 9.1: (Table 9 cont.)

Table 9 shows means for each rubric have remained relatively consistent cross all program years. The rubric in which candidates demonstrate the strongest performance is Rubric 12 – Providing Feedback to Guide Further Learning – which measures candidates’ ability to provide specific and developmentally appropriate feedback related to the learning goals of one assessment. The candidate is required to submit evidence of feedback for all the three focus students. Although rubric means trend lower in Year 3 (2020-21), the variation in means is slight, with several of the means remaining consistent. Rubric 10 – Analyzing Teaching Effectiveness – trends the lowest and shows a mean of 2.8 across all years.

Table 10: edTPA First Attempts Submissions that Received Condition Codes* (CC)

*Includes first attempts without a total score due to condition codes for handbooks with 15 and 18 rubrics.

Condition codes are assigned when a scorer is unable to assign a score (e.g., due to missing evidence or technical issues such as a video with no audio). Candidates with two or more condition codes do not receive a total score. After receiving their score reports, candidates are able to address any task/rubric that received a condition code and resubmit their edTPA.

During Year 1 (2018-19) and Year 2 (2019-20) the rate at which candidates received condition codes was consistent, 4% and 4%, respectively. During Year 3 (2020-21) the assignment of condition codes for increased to 8%. The analysis of condition codes that candidates have received over the program years has been complicated by issues of the COVID-19 pandemic. Across Years 1 (2018-19) and Year 3 (2020-21) the highest instance of a condition assigned was D1: No response to commentary prompt(s) or response is unrelated to edTPA tasks, assigned 31 and 37 times respectively. In Year 2 (2019-20) the highest instance of a condition assigned was F1: Video in Task 2 has been edited and is NOT "continuous and unedited, with no interruption in the events". Across Years 1 (2018-19) and Year 2 the next highest instance of a condition assigned was G10: The work sample evidence for one or more focus students is missing or does not demonstrate what students know and/or are able to d0 (e.g., evidence is limited to a student survey, self-assessment, peer assessment, or work completed by a group), assigned 32 and 22 times, respectively.

Teacher candidates, students, and those who support candidates and teachers have struggled through the pandemic year engaging in learning in remote contexts. One aspect of this issue is the ability to see and hear one another across digital platforms. This problem represented a challenge for candidates to capture their teaching and student participation in video clips. Young learners were distracted, while middle and high-school students did not want to turn their cameras on during instruction. In response, edTPA and Evaluation Systems staff have provided ongoing information and resources to support programs and candidates--including modification and clarification of expectations and providing examples of acceptable and unacceptable video clips during webinars, coordinator meetings (https://www.edtpa.com/PageView.aspx?f=GEN_PlacementInAVirtualSetting.html). In order to further support candidates in using video to demonstrate their teaching capacity, edTPA provided multiple webinars called focused on using video techniques for professional learning and documenting teaching.

Figure 3 in Appendix D provides the ethnic distribution of teacher candidates passing the edTPA for years 2018-19, 2019-20 and 2020-21.

FAST Results and Analysis

Tables 11 and 12, below, provide pass rates for the FAST 2.0 Site Visitation Project and Teaching Sample Project in Academic Years 2018-19, 2019-20, and 2020-21. Although the COVID-19 pandemic forced candidates to shift to remote instruction, overall, we did not see a significant impact on their overall pass rates or their scores on the individual rubrics, as evidenced in the data provided. Candidates do not receive conditional codes on the FAST 2.0.

Table 11: FAST All Attempt Pass Rates: All Submissions

Table 11 indicates the total number of candidates who attempted either the Site Visitation Project or the Teaching Sample Project each academic year and provides the number (and percent) who passed on their first attempt and the number (and percent) who passed on their second attempt. The final column provides the number (and percent) of candidates who passed either the Site Visitation Project or the Teaching Sample Project within two attempts. As demonstrated in the data, overall, candidates are highly successful in passing both the Site Visitation Project and the Teaching Sample Project. In fact, across the three years of data, only two individuals did not pass. Both had extenuating circumstances that were exacerbated by the shift to remote instruction in spring 2020 due to the COVID-19 pandemic.

Table 12: FAST First Attempts of Site Visit Performance Score Means by Rubric

Table 13: FAST First Attempts of Teaching Sample Project Rubric Score Means

*TSP 1: Context Rubric
TSP 2: Outcomes Rubric
TSP 3: Assessment Rubric
TSP 4: Instruction Rubric
TSP 5: Decisions Rubric
TSP 6: Analysis Rubric
TSP 7: Reflection Rubric

Table 13 provides the mean rubric scores for each program area in each academic year. As the data indicate, means scores have remained fairly consistent by program across the three years of data. In fact, on all rubrics with the exception of Assessment on the TSP in 2018-19 for both programs, and Reflection on the TSP in 2018-19 and 2020-21 for Multiple Subject, the mean scores fall between 2.5 and 2.9. Worth noting is that the Assessment mean scores did increase in 2019-20, though they did fall again slightly in 2020-21. Similarly, the Reflection mean scores for Multiple Subject increased from 2.4to 2.5 from 2018-19 to 2019-20, but they then increased slightly to 2.6 in 2020-21. Traditionally, candidates in the Multiple Subject program have struggled more with the Reflection component, perhaps because it is the final part of the project. Still, scores of candidates in the Single Subject program do not indicate that they have the same challenges.

Looking closely at the data also reveals that there were slight dips in the scores of Multiple Subject candidates on two tasks of the Site Visitation Project and five of the seven tasks on the Teaching Sample Project. The dips were more significant for the Site Visitation Project Tasks than they were for the Teaching Sample Project tasks. For example, the mean score for Multiple Subject candidates for the SVP Reflection task was 2.9 in 2019-20, and it dropped to 2.7 in 2020-21, a difference of 0.2 points. In contrast, the TSP Decisions task means score for Multiple Subject candidates was 2.61 in 2019-20, and it dropped to 2.59 in 2020-21, a difference of only 0.0236. These minor shifts were likely due to the challenges candidates experienced completing the majority of their field experience through remote instruction during the 2020-21 academic year.

Consistency in Scores

The overall consistency in scores across the three years are due, at least in part, to the way in which the FAST is integrated into the program. Because faculty are familiar with the expectations of the assessments, they are able to guide candidates in seeing connections between their coursework and the tasks associated with the SVP and TSP.

Using the Data to Inform Instruction

Programmatically, the goal is for the mean scores of candidates from both the Multiple Subject and Single Subject programs on each of the rubrics to be closer to 3. Currently, program faculty are engaging in an analysis of the data to consider changes they can make in their coursework in an effort to better prepare candidates for the tasks where scores indicate they do have more challenges, including Learning Outcomes, Assessment Plan, Instructional Decision-Making, Analysis of Student Learning, and Reflection and Self-Evaluation in the Multiple Subject Program and Learning Outcomes and Instructional Decision-Making in the Single Subject Program.

Figures 4 and 5 in Appendix G provides the ethnic distribution of teacher candidates passing FAST for years 2018-19, 2019-20 and 2020-21.

Part Three: Background and Overview of the Administrator Performance Assessment (CalAPA)

Background

The Commission began discussing the development of a performance assessment for administrative services credential candidates in September 2012, approving the idea in September 2013. In 2015 the project was funded through the state budget act and design work began. After two years of development with the CalAPA Design Team, Preliminary Administrative Services Credential (PASC) programs experienced a gradual implementation process. Following a pilot and a field-test year, candidates participated in a non-consequential scoring year. During the non-consequential year, funded by the Commission, candidates were required to complete the CalAPA but with no passing score requirement. During the 2019-20 academic year, the CalAPA became “consequential,” and candidates were required to meet a Commission-adopted passing standard for each leadership cycle.

The Commission’s model CalAPA is the sole performance assessment for potential school administrators currently approved for use in California. CalAPA is built upon the California Professional Standards for Education Leaders (CPSEL) and measures the California Administrator Performance Expectations (CAPE). The CAPE identify the knowledge, skills, and abilities that a school administrator should demonstrate at the beginning of an administrative services career.

Commission-approved Program Standards 7 and 8 for Preliminary Administrative Services Credentials outline required candidate supports that all programs must build into their program design and implement on an ongoing basis. Accreditation activities assess the degree to which an institution is aligned to the standards, assigning additional timelines should the alignment be less than required. Examples of program support requirements include making assessment materials accessible to candidates, providing practice opportunities, supporting candidates to review and resubmit evidence, and using data to improve program design and implementation.

To further support candidates and programs, Commission and ES staff have developed a series of outreach opportunities that occur at regularly scheduled intervals. These opportunities include candidate and program office hours, quarterly program coordinator meetings, a variety of webinars focused on best practices in supporting candidates through completion of the CalAPA, and an annual performance assessment implementation conference designed for programs to share evidence-based practices. For more detail, please see the table outlining administrator candidate supports offered by Commission staff and ES in Appendix A.

Since its inception in 2018, a total of 4504 candidates have taken all three leadership cycles of the CalAPA, with a total of 3563 consequential candidates (represented in the 2019-20 and 2020-21 program years) required to pass the assessment in order to be recommended for the Preliminary Administrative Services Credential. Of this number, 98% have passed the assessment (all three leadership cycles).

CalAPA Overview

The CalAPA features three leadership cycles that PASC candidates complete during their preliminary preparation program. The cycles require candidates to engage in a recursive four-step process of Investigate, Plan, Act, and Reflect (IPAR), with Cycles 2 and 3 including a video component. Once submitted via Pearson’s electronic platform, trained and calibrated assessors score the cycle, returning a score report within three weeks of the submission window’s closing.

Program Structure: Three Equitable Leadership Cycles

The topics of the cycles focus on administrative services performance expectations (CAPE) that California’s education administrators engage in regularly as school leaders.

Leadership Cycle 1: Analyzing Data to Inform School Improvement and Promote Equity

Leadership Cycle 1 focuses on analyzing multiple sources of school data for the purpose of identifying equity gaps and inform an initial draft plan for equitable improvement in line with the school’s vision, mission, and goals. Candidates collect and analyze multiple sources of longitudinal quantitative and qualitative data. They then conduct an equity gap analysis to identify potential factors, institutional and/or structural, all culminating in a problem statement that defines a specific area of educational need related to equity. Candidates seek input from one or more education partners at the school site and adjust their plan as needed to more effectively address the equity issue. To close, the candidate reflects on equitable leadership and specific growth they have experienced while completing this cycle. This cycle has 8 rubrics and requires a score of 14 points to pass. The fee for cycle one is $125.00.

Leadership Cycle 2: Facilitating Communities of Practice

Leadership Cycle 2 focuses on facilitating collaborative professional learning within a community of practice for the purpose of improving teaching and student learning or well-being. Candidates begin by identifying and working with a small group of educators to identify a problem of practice. That group selects an evidence-based instructional strategy to address the problem of practice that will strengthen and increase equitable learning and/or well-being for all students. Then, during initial implementation of the selected strategy, they facilitate meetings with the group and collaboratively lead the professional learning of the community of practice. Finally, candidates reflect on how their facilitation supports the group in addressing the problem of practice, understanding early implementation findings, and how they responded to the group’s feedback on their facilitation. This cycle has 7 rubrics and requires a score of 12 points to pass. The fee for cycle two, with its video component, is $150.00.

Leadership Cycle 3: Supporting Teacher Growth

Leadership Cycle 3 focuses on coaching an individual teacher to strengthen teaching practices and improve student learning and/or well-being. Candidates familiarize themselves with coaching and observation practices at the school; identify a volunteer teacher with whom they work; and conduct a full coaching cycle, including a pre-observation meeting, a focused classroom observation to collect evidence of practice related to the California Standards for the Teaching Profession (CSTP), and conduct a post-observation meeting. Throughout this leadership cycle, candidates reflect on their strengths and areas for professional growth as an instructional coach and an equity-minded leader. This cycle has 7 rubrics and requires a score of 12 points to pass. The fee for cycle three, with its video component, is $150.00.

Analytic Rubric Scoring Process

Each of the CalAPA analytic rubrics consist of a five-point scale. With 8 rubrics, Cycle 1 has a possible score total of 40 points, while Cycles 2 and 3, with 7 rubrics each, have possible score totals of 35 points. Each rubric has an essential question, outlining the specific performance that will be assessed in that rubric. Each of the five levels of the rubric provide qualitative descriptions of performance based on the California Administrator Performance Expectations (CAPE) measured in the cycle. Scoring practices follow the CalTPA process described above in Part 1: should a candidate provide exactly the evidence required in the essential question, they receive a score of “3.” Providing additional information, as outlined in levels 4 and 5, allows the candidate to improve their score while providing off-target or incomplete evidence results in a lower score of “1” or “2.” Often, the level 1 score is applied when the candidate is missing key evidence outlined in the rubric level description and essential question. A candidate is required to reach a cumulative score for each cycle to pass. Rubric descriptions are provided in Appendix J.

Condition Codes

Submissions that have not followed the instructions provided in the Assessment Guides are assigned a condition code for the appropriate rubric(s). Each condition code describes the aspect of the submission requirements that was not followed according to the outlined procedures. Condition codes are described publicly on the CalAPA Assessments Policy web page. If a candidate receives a condition code, the candidate must address and correct the error and resubmit the cycle in order to obtain scores. Programs are provided data on condition codes for their candidates with each reporting date, so that support and/or alterations in program delivery can be offered with the goal of eliminating condition codes and increasing candidate performance.

Assessor Recruitment, Training, and Support

Starting in fall 2017, Evaluation Systems began recruiting specific assessors and identified lead assessors for the field test scoring process. Assessors must meet three eligibility requirements. 1) Be a current California education professional or have retired within the past three years in one or more of the following capacities: a) University/program educator providing instruction to administrative services candidates within a Commission-accredited preliminary or clear administrative services preparation program or b) TK–12 administrator; 2) Hold a current California Clear or Life Administrative Services Credential; and 3) adhere to the following confidentiality requirements: a) Maintain the confidentiality of the assessment materials and knowledge gained as a result of participating in scoring the assessment, and will not share information with anyone without direct permission from the Commission and Pearson and b) agree not to participate in any professional activity, beyond employment in a TK-12 school/district/county office or institution that requires candidates to use the CalAPA to fulfill the APA requirement, that results in payment for services related to supporting candidates in completing any California-approved performance assessment (e.g., CalAPA independent tutoring or consulting positions).

Assessors are divided into three groups: one group is trained to score Cycle 1, a second group to score Cycle 2, and a third to score Cycle 3. Trainings include a mix of online and in-person experiences. Three lead assessors for each cycle are identified from the pool of assessors and serve for a three-year term. Lead assessors participate in selecting marker papers, facilitated by Commission and ES staff, and are trained to support the facilitation of the ongoing assessor trainings. To support assessor participation, trainings were held in southern and northern California prior to COVID restrictions. During the pandemic, assessor training temporarily moved to an online format.

Table 14: Number (N) of CalAPA Assessors per Year 2018-2021

Prior to the synchronous training, assessors are required to attend an online orientation facilitated by Commission and ES staff that provides an overview of the leadership cycles, evidence to be collected, associated analytic rubrics, and the Commission’s Using the Tools of Cultural Proficiency to Address Implicit Bias training. Assessors then attend a two-day synchronous training where lead assessors facilitate a review of five submissions representing a range of performance. Assessors are required to independently score a calibration submission and meet the calibration criteria. Once notified they have met the calibration criteria, assessors begin scoring candidate submissions using Pearson’s secure online scoring system. Assessors who do not meet the calibration requirement are provided coaching by lead assessors and given the opportunity to score an additional calibration submission.

CalAPA submissions are scored, as needed, by up to three assessors. Scoring quality is monitored on an ongoing basis, including use of the following metrics to monitor ongoing assessor calibration:

• Inter-rater reliability — agreement rates between assessors on double-scored submissions.
• Validity submissions — pre-scored submissions are sent to each assessor. Assessors are not aware that they are scoring a validity submission (i.e., blind scoring) and their performance is monitored on an ongoing basis. Assessors who do not meet the established agreement rates are flagged for additional review.
• Backreading — supervisors and lead assessors monitor and read-behind assessors as submissions are scored. Supervisors and lead assessors intervene and remediate assessors on any areas identified.

An administrative services candidate is required to pass all three Cycles: 1 (14 points of 40), 2 (12 points of 35) and 3 (12 points of 35). Passing the performance assessment and completing required coursework and clinical practice qualifies the candidate for the Preliminary Administrative Services Credential if hired into and administrative services position or a certificate of eligibility (CE) that remains in place until the candidate finds employment and applies for the preliminary credential. Administrative Services Credential candidates who earn

their preliminary credential and secure employment have five years to complete their clear credential requirements. Requirements for the clear credential and credential renewal do not become relevant until the preliminary credential is issued.

CalAPA Program and Candidate Supports

Commission-approved Program Standards 7 and 8 for Preliminary Administrative Services Credentials outline required candidate supports that all programs are required to build into their program design and implement on an ongoing basis. Accreditation activities assess the degree to which an institution is aligned to the standards, assigning additional timelines should the alignment be less than required. Examples of program support requirements include making assessment materials accessible to candidates, providing practice opportunities, supporting candidates to review and resubmit evidence, and using data to improve program design and implementation.

To further support candidates and programs, Commission and ES staff have developed a series of outreach opportunities that occur at regularly scheduled intervals. These opportunities include candidate and program office hours, quarterly program coordinator meetings, a variety of webinars focused on best practices in supporting candidates through completion of the CalAPA, and an annual performance assessment implementation conference designed for programs to share evidence-based practices. For more detail, please see the table outlining administrator candidate supports offered by Commission staff and Evaluation Systems in Appendix A.

Part Four: Analysis of Data Results for CalAPA 2018-19 to 2020-21

Table 15: CalAPA First Attempt Pass Rates All Submissions without Condition Codes

Table 15 provides the passing rate for candidates attempting the CalAPA for the first time. There has been a fluctuation in submission numbers during the years of CalAPA implementation. From 2018-19 to 2019-20 there was an increase of nearly 2,000 submissions, followed by a decrease in 2020-21 due to COVID-19 conditions--schools were closed and candidates were unable to complete the CalAPA. The passing scores (14 for Cycle 1, which has 40 total points possible, and 12 for Cycles 2 and 3, both of which have 35 total points possible), were deliberately set at a point that recognized both faculty and candidates were still learning the performance assessment’s structure, requirements, and supports needed.

Table 16 illustrates the percentages of CalAPA candidates who have passed all Cycles (1, 2, and 3) of the CalAPA. The N reflects the number of candidates who took Cycles 1, 2, and 3 in the program year (all attempts, including retakers), with scorable responses on all cycles (i.e., no condition codes). The percent (%) reflects the percent of those candidates who passed all cycles (on any attempt) in the program year.

Table 16: Percentages of CalAPA Candidates by Year Who Passed All Cycles

Table 17: CalAPA Performance by Cycle – All Attempt Pass Rates (Submissions without Condition Codes)

Table 18: CalAPA First Attempt Score Means by Individual Rubric

* Rubric descriptions are in Appendix J

A rubric score of 3.0 reflects the candidates’ ability to adequately respond to the requirements of the rubric. With 22 rubrics total, six rubrics had a mean of 3.0 or greater in 2018-19, and five in 2019-20 and 2020-21. In 2020-21 there are eight rubric means with 2.9 (up from five in 2018-19 and one in 2019-20) indicating that candidate submissions are coming closer each year to meeting the rubric requirements. Rubrics 7 and 8 ask candidates to reflect upon their learning during the Performance Assessment’s cycle. Lower mean scores highlight the need for programs to address candidate skills in this area.

Table 19: Candidate Submissions Receiving Condition Codes

Table 19 presents the number of submissions that received condition codes, rendering them unable to be scored because the candidate had not followed the requirements of cycle submissions. Cycle 1 has consistently received the least number of condition codes, while Cycles 2 and 3 have video components which represent the most frequent conditions codes assigned, highlighting the need that candidates are still learning the skills of video recording, annotating, and editing.

Figure 6 in Appendix I provides the ethnic distribution of candidates passing the CalAPA assessment.

Part Five: Performance Assessment Implementation Information

Section 5 outlines the processes the Performance Assessment team and Evaluation Systems follow in the implementation of the CalTPA and CalAPA to assure validity and reliability of the assessments, to provide information for both candidates and programs regarding results, and how the data informs program accreditation. Also included in this section is information about COVID flexibilities and next steps and upcoming initiatives for the Performance Assessment team.

Testing/Assessment Bias

In the field of large-scale assessments, differential passing rates by subgroups are not considered bias in and of themselves. Commission assessments are designed, in part, to uncover differences in scores according to various subgroups, particularly groups based on gender, race and ethnicity, to help understand gaps in education among the population of candidates coming into teacher preparation. Processes to avoid bias are built into the Commission’s examination development and administration processes, including a Bias Review Committee, which reviews all assessment content and materials for potential bias. The Commission employs these procedures specifically to reduce measurement error that might be caused by bias so that results by gender, race, and ethnicity can be accurately reported.

Institutional Data Reports for CalTPA, edTPA, and CalAPA Candidates

During registration, candidates select the institution to which they would like their scores reported. Candidate results are reported to the selected programs, and the programs have secure access to online results reports through Pearson’s edReports portal. edReports contains files of candidate results, which programs may import into their own data management system for program purposes. edReports provides access to the ResultsAnalyzer reporting tool, which provides the ability to review candidate-level and assessment level data. The ResultsAnalyzer system contains results reported to the program with filters and reports which, when selected, provide quick access to frequently generated data such as assessment or cycle/task pass rates, cycle/task scores, rubric scores, condition codes, and retake information. Evaluation Systems provides training on how to use these data analysis platforms multiple times each year.

Use of Performance Assessment Data in Accreditation Activities

Performance Assessment information is used throughout the accreditation cycle to assess an institution’s adherence to Commission standards. During Initial Institutional Approval, the program describes their program design and outlines support structures provided to their candidates. During program review, and site visits, Board of Institutional Review (BIR) members review program design, implementation, and candidate success, using this information to evaluate the degree to which an institution is in alignment to the standards, possibly requiring corrective measures to increase that alignment.

Over the past two years, Commission staff has worked to incorporate data about the performance assessments in the Accreditation Data Dashboards (ADD) that are available to all institutions and to accreditation teams in preparation for accreditation site visits. On an annual basis, institutional personnel may review their data along with other outcomes data such as program completer survey data for the purpose of program improvement. Accreditation teams have been trained and are instructed to review the data to inform their review of programs and in particular to use the data to help identify potential areas of strength and needed program growth, and to develop probing questions that will illuminate and inform their findings related to program standards and quality.

COVID-19 Flexibilities Regarding Performance Assessments

Executive Order N 66-20 and subsequent legislation AB 130 and AB 820 deferred the requirement to pass a performance assessment due to school closures for preliminary teacher and administrative services credential candidates and moved the requirement to the clear program. This deferral affected candidates enrolled in programs from April 2019 through August 2021; it was then extended for candidates to August of 2022. This flexibility enabled candidates who could not complete their performance assessment due to COVID-19 school closures and related disruption to earn a preliminary credential and begin teaching. As of January 22, 2022, 12,941 teaching and 823 administrative services credential candidates need to pass a performance assessment to clear their preliminary credential. Candidates completing performance assessments post-preliminary program are expected to be supported by their Induction and Clear programs and mentors while they complete their assessment. Commission staff are providing regular and ongoing technical assistance and support to Induction and Clear programs including weekly office hours and online candidate Academies. The Commission is aware of the challenge these numbers present to Induction and Clear programs and to the candidates and is working with both preliminary programs and Induction/Clear programs to provide additional supports in the coming years.

Next Steps

The Commission’s performance assessment work continues to grow with new assessments currently being developed for Preliminary Education Specialists and Early Childhood Education.

Education Specialist Performance Assessment

The Commission’s Mild to Moderate Support Needs (MMSN) and Extensive Support Needs (ESN) performance assessments (EdSp CalTPA) are currently being field tested, with a standard setting study planned for May 2022. Once approved by the Commission at the June 2022 meeting, operational implementation of the MMSN and ESN assessments will begin in fall 2022. Early Childhood Special Education (ECSE), Deaf and Hard-of-Hearing (DHH), and Visually Impaired (VI) performance assessments will continue to be developed with a pilot study in spring of 2022 and field test in 2022-23. A standard setting will be conducted spring of 2023. If approved by the Commission at the June 2023 meeting, operational implementation of the ECSE, DHH, and VI assessments will begin in fall of 2023.

SCALE (Stanford Center for Assessment, Learning, and Equity), developers of the edTPA, have submitted an Education Specialist edTPA, for review and approval by the Commission at the June 2022 meeting.

CSU Fresno, developers of FAST, are planning to submit an Education Specialist version of its assessment for review and approval by the Commission at the June 2022 meeting.

Early Childhood Performance Assessment (ECE CalTPA)

Commission staff and Evaluation Systems are implementing a pilot of an ECE Performance Assessment during spring of 2022, followed by a field test fall of 2022. At the June 2023 Commission meeting, staff expect to seek adoption of the assessment with local implementation planned for fall of 2023.

Reading/Literacy Performance Assessment

In Summer of 2022, Commission staff will begin work with an appointed Design Team and technical vendor for two years to develop a Reading/Literacy Performance Assessment to replace the current RICA examination. The Reading/Literacy performance assessment is planned to be operational fall of 2025. Agenda item 3C in this Commission meeting’s agenda provides further details.

Administrative Services Performance Assessment (CalAPA)

A standard setting panel for the CalAPA is being reconvened in May 2022. Based upon additional data from the 2021-22 administrator assessment, the panel will recommend passing scores to the Commission at the June 2022 meeting for the 2022-23 academic year.

Commission staff, in accordance with the Assessment Design Standards, plans to present an annual report on all of California’s performance assessment systems.

Appendix A

CalTPA and CalAPA Supports for Programs and Candidates

Appendix B

CalTPA Data

Table 1: CalTPA Number of Assessors by Multiple Subject and Single Subject

Table 2: CalTPA Pass Rates for Candidates Who Took Both CalTPA Cycles

* Lower Passing Standard applied

Table 3: CalTPA All Attempt Pass Rates by Cycle: All Submissions Without Condition Codes

* Lower Passing Standard applied

Table 4: CalTPA First Attempt Score Means by Rubric: Cycle 1

Table 5: CalTPA First Attempt Score Means by Rubric: Cycle 2

Table 6: CalTPA Number of Submissions Receiving Condition Codes

Figure 1: CalTPA Passing Rates by Ethnicity First Attempts: Cycle 1

* Lower Passing Standard applied

Figure 2: CalTPA Passing Rates by Ethnicity First Attempts: Cycle 2

* Lower Passing Standard applied

Appendix C

CalTPA Rubric Descriptions

CalTPA Cycle 1 Rubric Essential Questions

Rubric 1.1 How does the candidate’s proposed learning goal(s) connect with students’ prior knowledge? How do proposed learning activities, instructional strategies, and grouping strategies support, engage, and challenge all students to meet the learning goal(s)?

Rubric 1.2 How does the candidate plan instruction using knowledge of FS1’s (student with a different language learning need) assets and learning needs to support meaningful engagement with the content-specific learning goal(s)?

Rubric 1.3 How does the candidate plan instruction using knowledge of FS2’s assets, learning needs, and IEP/504/GATE goals/plans to support meaningful engagement with the content-specific learning goal(s)?

Rubric 1.4 How does the candidate plan instruction using knowledge of FS3’s assets and learning needs to support meaningful engagement with the content-specific learning goal(s) and address the student’s well-being by creating a safe and positive learning environment during or outside of the lesson?

Rubric 1.5 How does the candidate maintain a positive and safe learning environment that supports all students to access and meet the content-specific learning goal(s)? How does the candidate support students in making connections between prior learning and the current lesson and establish clear learning expectations?

Rubric 1.6 How does the candidate actively engage students in deep learning of content, monitor/informally assess their understanding, and explain to students next steps for learning?

Rubric 1.7 How does the candidate reflect on (citing evidence from Steps 1 and/or 2) the impact of their asset- and needs-based lesson planning, teaching, and informal assessment of student learning and analyze how effective the lesson was in supporting the whole class and the 3 focus students in meeting the content-specific learning goal(s)?

Rubric 1.8 How will the candidate apply what they have learned in Cycle 1 (citing evidence from Steps 1, 2, and/or 3) about students’ learning to strengthen and extend students’ understanding of content and academic language development and determine next steps for instruction?

CalTPA Cycle 2 Rubric Essential Questions

Rubric 2.1 How does the candidate’s learning segment plan provide appropriate content-specific learning goal(s) and, if appropriate, ELD goal(s), assessments, and rubrics that offer multiple ways for all students to demonstrate knowledge and affirm and validate students’ assets, including strengths, experiences, and backgrounds?

Rubric 2.2 How does the candidate plan a learning segment in which assessments and rubrics, instructional strategies, and lessons align and build on one another to provide a progression of learning that develops the students’ concepts and skills to achieve the standards-based learning goal(s)?

Rubric 2.3 How does the candidate support student development and demonstration of academic language in relation to the content-specific learning goal(s)?

Rubric 2.4 How does the candidate incorporate educational technology (digital/virtual tools and resources) to provide opportunities for students to use these tools and resources to enhance, improve, and/or demonstrate knowledge, skills, and/or abilities related to the learning goal(s)?

Rubric 2.5 How does the candidate use informal assessment to monitor and support the students’ deep learning of content (age and/or developmentally appropriate higher-order thinking skills) and adjust instruction to meet the needs of learners?

Rubric 2.6 How does the candidate model and engage the students in self-assessment to build their awareness of what they have learned, provide feedback, and support their progress toward meeting content-specific learning goal(s) and ELD goal(s), if appropriate?

Rubric 2.7 How does the candidate use results of informal assessments and/or student self-assessment to provide actionable feedback to students about how to improve or revise their work to continue progress toward and/or beyond the learning goal(s)?

Rubric 2.8 How does the candidate analyze the formal assessment results based on the rubric and identify and describe learning patterns and/or trends for the students and determine what was most and least effective in relation to the learning goal(s)?

Rubric 2.9 How does the candidate use the analysis of results from informal assessment, student self-assessment, and formal assessment to plan and teach a follow-up learning activity and provide a rationale for the activity choice, citing evidence?

Appendix D

EdTPA Data Tables

Table 7: edTPA First Attempt Pass Rates*

*Includes first attempts with a total score for handbooks with 15 and 18 rubrics.

Table 8: edTPA All attempt Pass Rates**

**Includes ALL attempts with a total score for handbooks with 15 and 18

Table 9: edTPA First Attempt Score Means by individual rubric

Table 9.1: (Table 9 cont.)

Table 10: edTPA First Attempts Submissions that Received Condition Codes (CC)

*Includes first attempts without a total score due to condition codes for handbooks with 15 and 18 rubrics.

Figure 3: edTPA Ethnicity Data for 1st attempts*

*Includes first attempts with a total score for handbooks with 15 and 18 rubrics.

Appendix E

edTPA Rubrics, Evidence, and Tasks

The following table shows the common design of edTPA across handbooks, including the Tasks 1-3, the required authentic records of practice, and the scoring rubrics used to evaluate candidate performance:

Appendix F

Additional Information Regarding edTPA Assessors

Assessor Training: The assessor training curriculum, developed by SCALE and delivered and monitored by Pearson, includes the following:

• Overview of edTPA, including purpose, architecture, and outcomes measured
• Prevention of bias
• Academic language, including edTPA constructs, definitions, and examples
• edTPA Task 1 rubric constructs, identifying evidence, and score distinction exercises
• edTPA Task 2 rubric constructs, identifying evidence, and score distinction exercises
• edTPA Task 3 rubric constructs, identifying evidence, and score distinction exercises
• Introduction to scoring processes, the tools provided to assist in scoring, and the individuals who support scorers as they train and score
• Information on the scoring system used to score the responses
• Training on how to communicate with supervisors who monitor scoring processes
• Information on how to identify tasks that cannot be scored and further information on the qualifying standards a scorer is expected to meet

Calibration in Scoring: The cornerstone of the edTPA performance scoring is quality achieved through consistent, reliable scoring. Quality is the ultimate goal that unifies all edTPA processes and procedures related to scoring. Through auditing, edTPA processes are kept standardized, repeatable, documented, and followed. edTPA scoring process measures include, but are not limited to, qualification (training certification), inter-rater reliability (IRR), validity, and scoring timeliness. The following is an overview of such steps:

• Qualification (Training Certification) – Applicants must successfully complete training and qualifying before scoring.
• Second Scoring – Allows edTPA scoring staff to closely monitor scorer performance and to provide inter-rater reliability statistics.
• Backreading – edTPA supervisory staff reviews portfolio scores to confirm that the scores were correctly assigned and to give feedback and remediation.
• Validity – Verifies that scorers are applying the same standards throughout the project and, therefore, guards against scorer drift and ultimately group drift.
• Validity as Review – Select validity portfolios that are annotated and flagged for review. These portfolios are sent automatically to individual scorers for review if the scorer does not assign the correct score, and aid in preventing scorer drift before it occurs.
• Inter-Rater Reliability – Allows scoring supervisors and scoring directors to monitor individual and group performance. Based on reviews of these reports, scoring experts can target individuals for increased backreading and feedback and, if necessary, retraining.
• Reporting and Data Analysis – Our scoring system automatically captures and tracks all score data. By reviewing up-to-date scorer performance statistics, we quickly identify particular scorers whose performance falls outside of group norms while also keeping close track of the group as a whole.

Appendix G

Fresno Assessment of Student Teachers Data Tables

Table 11: FAST All Attempt Pass Rates: All Submissions

Table 12: FAST First Attempts of Site Visit Performance Score Means by Rubric

Table 13: FAST First Attempt Score Means for Teaching Sample Project by Individual Rubric

* TSP 1: Context Rubric
TSP 2: Outcomes Rubric
TSP 3: Assessment Rubric
TSP 4: Instruction Rubric
TSP 5: Decisions Rubric
TSP 6: Analysis Rubric
TSP 7: Reflection Rubric

Appendix H

FAST Rubric Descriptions

Site Visitation Performance Rubric Descriptions of Level 2 “Meets Expectations”

Planning Pedagogy: The lesson plan includes content and related activities or consistent with current subject-specific pedagogy and standards, that support the acquisition or use of academic language.

Applying Knowledge: Information about students, gathered by the candidate, provides useful information for planning...candidate plans appropriate activities or strategies to promote access to the content.

Student Engagement: Candidate’s plan for engaging students is appropriate to the grade level.

Implementation Pedagogy: Candidate effectivelyimplements instruction consistent with subject-specific pedagogy to teach theidentified academic content standard(s). Instruction supports the acquisition or use of academic languageappropriate for students at this grade level.

Applying Knowledge: Candidate uses knowledge of the learning needs, backgrounds or interests of studentstokeep them on task.

Student Engagement: Candidate uses primarily management techniques to promote and monitorparticipation by students in the learning activities. Candidate expressesand reinforces expectations for social or academic behavior. Candidate modelsgenerally positiveinteractions.

Reflection Pedagogy: Candidate realistically describes strengths andweaknesses of lesson. Provides general justification for how theactivity or strategy in the selected video clip represents subject- specific pedagogy. Demonstrates a realisticunderstanding of the relationship between content knowledge and planning or teaching.

Applying Knowledge: Candidate realistically describes how the lesson promotes accessto the content for the focus student, using evidence of participation. Provides appropriate suggestions to improveaccess to content for students, in general.

Student Engagement: Candidate provides general examples of interactions from thelesson. Realistically describeshow these interactions promote productive student learning, multiple perspectives, or equitable participation.

Teaching Sample Project

Students in Context

Implications for Instruction: Descriptions of instructional approaches are generally appropriate for at least two of the following student groups: different levels of English proficiency, identified special needs, and different instructional needs.

Creating and Maintaining Effective Environments: Expectations for, and responses to, behavior include general examples related to at least two of the following: individual responsibility, intolerance, an inclusive climate. Routines focus on management, with a general description of how they were communicated to students.

Learning Outcomes

Learning Outcomes and Standards: Outcomes primarily address either content or literacy standards. Most outcomes represent the content and level of learning (e.g., DOK level) reflected in the content standards, though they primarily focus on lower levels of learning.

Appropriateness For Students: Description of unit and rationale provide general justification for development of either content knowledge or literacyskills, and past experiences, pre-requisite knowledge, or future learning, relevance for students at that grade level.

Assessment Plan

Congruence with Learning Outcomes and Content: Most assessment methods are congruent with learning outcomesin either content or level of learning. Attention to assessment of content knowledge or literacy skills.

Variety in Methods of Assessment: The assessment plan assesses student knowledge or performance before, during, and after instruction, with some variety in the assessment methods.

Clarity of Assessment Methods: Prompts, directions, scoring procedures, and criteria for meeting learning outcomes are given for most assessment methods. Candidate describes how the format of the assessments match the learning outcomes and the purpose of assessing

Design for Instruction

Use of Contextual Information and Data to Inform Instruction: Lessons show some attention to pre-assessment results and contextual factors. Lessons provide access to content or develop literacy skills for at least two of the following: identified special needs, different levels of English language proficiency, and different instructional needs.

Alignment with Learning Outcomes & Standards: Lessons are aligned with unit learning outcomes and are consistent with current subject-specific pedagogy in the content area of instruction or literacy skills.

Variety in Instruction: A variety of instructional methods and engagement strategies. Appropriate use of technology to engage students or promote access to content, though primarily used by teacher.

Instructional Decision-Making

Monitoring Student Learning: Evidence of monitoring students during instruction is implied or general. Focus is primarily on behavior or lesson structure rather than student learning

Adjustments Based on Knowledge of Student Learning and Providing Access to Curriculum: Some adjustments of the instructional plan are made to address general student needs, with some connections to knowledge of student learning or providing access to curriculum.

Alignment Between Adjustments and Learning Outcomes: Adjustments to instruction are generally aligned with learning outcomes. Reasons for adjustments address efforts to improve student progress.

Analysis of Student Learning

Analysis and Interpretation of Data: Analysis and interpretation of data provides some evidence of the number of students meeting at least one of the learning outcomes, is generally accurate, with some supporting evidence, and describes how the data/scores reflect learning related to at least one of the learning outcomes.

Progress Report: Progress report for student who struggled that uses some data or examples to describe strengths or areas for growth related to one of the unit outcomes and provides general suggestions for improving student learning.

Reflection and Self-Evaluation

Insights on Effective Instruction and Assessment: Describes effective instructional activities for at least two of these categories: a range of English proficiency, students with identified special learning needs, or students with different learning needs. Identifies the alignment between assessments and learning outcomes. Describes subject matter knowledge related to this unit.

Implications for Future Teaching: Provides appropriate suggestions for redesigning learning outcomes, instruction, or assessment.

Implications for Professional Development: Presents a reasonable professional learning goal connected to teaching in general. Appropriate steps described in general terms.

Appendix I

California Administrator Performance Assessment (CalAPA) Data Tables

Table 15: CalAPA First Attempt Pass Rates All Submissions without Condition Codes

Table 18: CalAPA First Attempt Score Means by Individual Rubric*

* Rubric descriptions are in Appendix J

Table 19: Candidate Submissions Receiving Condition Codes

Figure 6 CalAPA Passing Rate by Ethnicity

N/A=less than 10 submissions received

CalAPA Cycle 2

N/A=less than 10 submissions received

CalAPA Cycle 3

N/A=less than 10 submissions received

Appendix J

CalAPA Rubric Descriptions

CalAPA Leadership Cycle 1 Rubric Essential Questions

Rubric 1.1 Based on the chosen California state indicator, how does the candidate select and analyze quantitative data sources across the three most recent years, identify patterns and/or trends related to equity, choose one student group, and relate their analysis to the school’s vision, mission, and/or goals?

Rubric 1.2 How does the candidate collect and analyze relevant qualitative data and explain their relation to quantitative data findings and the student group equity issues?

Rubric 1.3 How does the candidate conduct an equity gap analysis based on the chosen California state indicator to inform their understanding of the equity issues for a student group?

Rubric 1.4 How does the candidate determine contributing factors, including institutional and/or structural factors, that created or added to the identified equity gap affecting a student group and cite the research supporting their determination?

Rubric 1.5 How does the candidate use the equity gap analysis and identification of potential contributing factors to develop a feasible problem statement related to student achievement and/or well-being?

Rubric 1.6 Are the strategies proposed for equitable school improvement for the student group well informed by the findings of the equity gap analysis, including contributing factors, and responsive to the problem statement? Are proposed strategies aligned to the school’s vision, mission, and/or goals?

Rubric 1.7 How does the candidate apply the feedback received from a key stakeholder(s) familiar with the school culture and context and describe next steps for creating stakeholder buy- in and potential implications for the adjusted set of strategies?

Rubric 1.8 How does the candidate reflect on and analyze what they have learned about equity-driven leadership in Cycle 1 (citing from Steps 1, 2, and/or 3) and how, based on the school contexts, they might address a single equity gap for a group of students at the school? How does the candidate use this learning to identify strengths and areas for leadership growth?

CalAPA Leadership Cycle 2 Rubric Essential Questions

Rubric 2.1 How does the candidate describe and analyze the role of current practices of professional collaboration at the school as the current practices relate to student learning and/or well-being?

Rubric 2.2 How does the candidate select an area of educational focus based on student data and choose a group of educators to participate in a community of practice about student learning and/or well-being that corresponds to the school’s vision, mission, and/or goals?

Rubric 2.3 Based on the agreed-upon area of educational focus, how does the candidate collaboratively work with the group to select a problem of practice (how practitioners may improve instructional practice or the system) related to student learning and/or well-being and build group ownership?

Rubric 2.4 How does the candidate explain the collaborative process used to select the relevant evidence-based strategy and work with the group to learn about and monitor implementation of that strategy to address the selected problem of practice? How does the candidate describe the potential impact on student learning and/or well-being?

Rubric 2.5 How does the candidate co-facilitate group learning—including establishing, reviewing, and using norms; documenting decisions; facilitating a collaborative process (group consensus, feedback, and progress); supporting diverse viewpoints; maintaining group focus and energy; and jointly determining next steps?

Rubric 2.6 How does the candidate demonstrate leadership as they co-facilitate group meetings and support members, individually and as a group, in learning to implement the evidence-based strategy and use initial results and feedback from members to help inform the learning process?

Rubric 2.7 How does the candidate use initial implementation results and feedback from the group—citing evidence from any of the four steps—to analyze their leadership skills and practices in order to identify areas for growth and identify next steps for equitably co-facilitating a community of practice?

CalAPA Leadership Cycle 3 Rubric Essential Questions

Rubric 3.1 How does the candidate describe and analyze the current role of teacher coaching, observation, and/or instructional feedback practices at the school, and explain the implications for their approach to conducting a coaching cycle?

Rubric 3.2 How does the candidate listen to and talk with the volunteer teacher to understand the learning goals, classroom context, and student assets and learning needs; jointly select with the volunteer teacher one or two CSTP elements, including evidence to be collected; and plan for the observation?

Rubric 3.3 How does the candidate recognize and document qualities of teaching practice related to the selected CSTP element(s) and learning goals of the lesson?

Rubric 3.4 How does the candidate foster a learning conversation in a post-observation meeting using CSTP-focused observation evidence, lesson observation video, and student work with the volunteer teacher regarding strengths and area(s) for growth?

Rubric 3.5 In partnership with the volunteer teacher, how does the candidate co-determine next steps for professional development, including resources and additional coaching support based on the CSTP-related evidence during the post-observation meeting?

Rubric 3.6 How does the candidate analyze their capacity to conduct a CSTP-focused coaching and observation process, based on their experience and feedback from the volunteer teacher, and cite evidence to demonstrate their ability to facilitate and maintain a coaching partnership?

Rubric 3.7 How does the candidate, informed by a continuous improvement mindset and focus on equitable leadership, understand the potential impact of coaching and reflect on benefits to both teachers and students?

Download the PDF

This PDF is provided as a print-only reference document and may not be fully accessible to users of assistive technology. For more information, please visit Title II Rule on Digital Accessibility page.