All of our MS coaches have participated in CREATe Rubric training. 60% of the MS coaches (N15) are fully reliable, and 7 more are partially reliable in that they are half way through the process and on their way to being fully reliable. 3 Faculty in Residence and 3 Teachers in Residence along with additional leadership team members from our district partners and Fresno State’s Continuous Improvement Lead participated in CREATe training. Each residency hosted the trainings which also included 2 live 20-minute observations with placing the teacher on the continuum followed by immediate discussion and reflection using the rubric after visiting the classroom. This model of training became a monthly community of practice to build connections and support between the three residencies (Fresno Unified, Sanger Unified, Central Unified). We also ensure interrater reliability using a rigorous observer training protocol in which observers must pass a paired observation based on reliability criteria before collecting live data in the field. Once we were able to establish interrater reliability, we collaborated with a partner school district’s New Teacher Induction Program to examine the validity of the CREATe Rubric. 

For concurrent validity, we used The New Teacher Project (TNTP) Core Rubric, which has established validity and is widely used. Pairs of calibrated CREATe and TNTP Core observers were assigned randomly to simultaneously observe in pre-selected classrooms. A strong partnership with the partner district New Teacher Induction program was the foundation to initiating and completing this validation study; the induction program staff helped us to identify a sample of 28 first-year teachers. The 28 participating teachers included nine graduates of the focal STaR Residency EPP and 15 non-residency graduates. The 15 non-residency group were a mix of candidates from other EPPs as well as candidates from the focal EPP who were not in the STaR Residency program (completed other pathways). 24 teachers were successfully observed during the observation window, with 4 teachers excluded due to scheduling/observation conflicts. Observer data was compiled and merged to enable data analysis and compare performance across measures to determine validity, and to provide information regarding necessary changes and revisions needed for continuous improvement in the STaR Residency program. For the current validation study, we used a paired observational design to observe 24 first-year teachers over a six-week period in the Spring 2019 semester using two independent observation tools: CREATe (Yun & Bennett, 2018) and the Core Teaching Rubric (TNTP, 2017). The TNTP Core Rubric was chosen because its four dimensions are conceptually well-aligned to the four dimensions of CREATe. Data collected by the observers included observation notes and rating scores. Notes and scores were collected either on paper or electronically on a laptop. Notes for the TNTP Core Rubric were sent to TNTP for compilation and a score report was generated for each of the observed teachers. Notes and ratings for CREATe were collected using a CREATe Score Sheet; ratings were manually entered into a spreadsheet. Data sources also include communications between the research team and district personnel as well as project documents such as timelines, calendars, and meeting notes. Analyses include confirmatory factor analysis, Pearson correlation, document analysis, and qualitative coding. Document analysis also suggests that CREATe has face validity when compared with Core. The four dimensions of the CREATe were well aligned with the four TNTP dimensions: Culture of Learning - Positive Environment; Academic Ownership - Instructional Design and Implementation; Essential Content; Rigorous and Appropriate Content; Demonstration of Learning - Reflection-in-Action (Core; CREATe) (see Figure 1). 

Regarding concurrent validity, CREATe and Core are measuring performance similarly; average performance on the Core rubric was significantly correlated with average performance on CREATe (r=0.35, p<0.10). This correlation suggests that mean performance were consistent across the two instruments. When the teachers were ranked according to average Core and CREATe scores, the rankings aligned 100%.

Confirmatory factor analysis (CFA) of CREATe suggests that the four proposed dimensions of the rubric hold up relatively well, with the exception of the Items 12 (Content Accessibility) and 13 (Interdisciplinary Integration). CREATe scores on 14 indicators, distributed across four dimensions. With the exception of the third dimension (which includes Items 12 and 13), a CFA and analysis of internal consistency both suggest that the other dimensions of the rubric hold together as outlined in the original design of the rubric (CFA, p<0.01;  factor loadings excluding #12/#13; a>0.8). Overall findings suggest that CREATe differentiates performance trends across three of its four predetermined dimensions (internal consistency/reliability), and that overall performance trends on CREATe are consistent across a different observation instrument, suggesting that CREATe measures teacher performance relatively well. Qualitative coding indicated that qualitative feedback for CREATe provides more evidence about the lesson itself, while qualitative feedback for Core provides more evidence about the scoring decision. This finding suggests fidelity in use of CREATe as the protocol calls for time-stamped scripted notes of classroom activities. 

Survey and Interview data with various stakeholders engaged with the CREATe Rubric have been conducted to better understand their experience with the measure. The CREATe Rubric has been well-received by district leadership and candidates. Coaches and faculty have been split on their reception of the CREATe rubric. Some of their concerns center on the length and cognitive load of the tool as well as disagreements around who is responsible for introducing the rubric to the credential students. When faculty who were the lead developers of the CREATe moved on from the program and coaches continued to question the utility of the CREATe Rubric, an assessment of all of the available rubrics were conducted. Additionally presentations and surveys to gather coach, faculty, and district feedback was administered. Based on this input, a decision was made in December 2020 to transition to the New Teacher Project (TNTP) Core rubric, which Chico State adapted to align with the CTC Standards. A Rubric Advisory Board had formed consisting of representatives from all three basic credential programs and an implementation timeline plus additional adaptations were made. However, by March 2020 California moved into its first COVID-19 lockdown putting the new formative rubric implementation timeline on hold.  

Of all the stakeholders, the candidates themselves have seemed to demonstrate the most positive responses. Candidates appreciate the explicit teacher behaviors in CREATe and their alignment to the TPEs. This helps candidates make the connections between the TPEs/CSTPs and their enacted practice. It also helps candidates to operationalize the concepts of developmentally appropriate practices, universal design for learning, culturally and linguistically sustaining practices, and inquiry. CREATe helps candidates construct what teaching looks like and focus on specific moves they can practice and improve. The quotes below demonstrate candidate perceptions of CREATe.

“The Create Rubric worked well for me in my clinical practice. It was a great guideline in what I need to be striving for and if I was meeting my goals as a teacher. In my opinion, without a rubric, we wouldn't have anything to base our teaching on, or have anything to strive for. It also sets a great goal and framework as teachers in different categories.” -Teacher Candidate Spring 2018

“The use of the CREATe Rubric as a planning tool for preservice teachers is a great resource because it helps us guide our lesson planning and implementation of the different aspects of what as teachers we should be doing every single day.” -Teacher Candidate Fall 2018

One thing is evident, ALL candidates responded well to a formative rubric that provided common language to make the skills of teaching more visible so they could receive actionable feedback from their coaches. This makes moving toward initial implementation of the new rubric ever more pressing.  

The existence of this CSU-wide service allows each campus to track the  effects of program changes designed to improve performance. Because  the instrument was designed and is implemented systemwide with  graduates throughout the state, we believe it is a fair and trustworthy  measure. 

Fresno State has initiated a college-wide data summit to consider the  findings of this statewide survey and triangulate them with campus data,  including the percentage of First Generation students, access to  resources like scholarships, and culture and context of the cohorts in  which prospective teachers are placed. Through this triangulation  process, we are able to determine the alignment of the finding from the  survey with our other measures, further assuring us of the survey’s  trustworthiness as an instrument. In the process, we are also able to  inform the impact on program changes on our own students with respect  to the unique diversity of culture and needs in the Central Valley. 

Beginning in 2019, the CTC sends an annual survey to employers of recent completers of all educators preparation programs in the state. The goal of the survey is to compile evidence statewide about the extent to which K-12 educators are prepared for their most important responsibilities.  

The survey includes 11 items. The first five of these ask basic demographic details about where the new educator was prepared and about the responding employer. The remaining six items are aligned with the California Teaching Performance Expectations.

The results of the survey are disaggregated for CSUs, UCs, Private, and Local Education Agencies. 79% of  respondents are principals. Of the 766 employers who responded to the survey in 2018-2019  (the last year for which data are available), 53% employed are recent graduates from a CSU teacher preparation program. 

Used statewide, the survey serves as a valid measure of employers' perceptions of how well educator preparation programs prepared alumni for their first year. Additionally, items included in the survey are directly aligned with the California Teacher Performance Expectations.  

All employers respond to items asking about the preparation of new teachers in general pedagogical skills, such as their perception of how well the program prepared new teachers to differentiate instruction in the classroom. In this way, the survey is a valid measure of employers’ perceptions of the program.

Uncertainty about evaluation findings comes from two principal sources: the number of evaluation participants and the extent of their concurrence with each other. The evaluation findings become increasingly certain to the extent that the questions are answered by increasing numbers of program completers and their employment  supervisors.  

Each year the data set yields the percent of respondents who gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents and  the concurrence or homogeneity of responses. 

Surveys are sent to all administrators within the state, giving all the opportunity to share their perspective on the educator preparation program their new teachers attended. In this way, the survey does not discriminate in who is invited to respond and whose voice is heard. Additionally, employers have a three-month window (October 1-December 31 annually) in which to complete the survey, providing ample opportunity to respond. 

The existence of this CCTC-wide service allows each preparation program to track the effects of program changes designed to improve performance. Because the instrument was designed and is implemented to employers throughout the state, we believe it is a fair and trustworthy measure. 

The California State University’s Education Quality Center (EdQ) oversees the  administration of a survey of all individuals who completed a CSU teacher-preparation programs after their first year on the job. The survey is administered annually April through July. In April, the EdQ Center emails an initial survey invitation to all completers of MS-SSES Credential Programs serving as first-year teachers in public schools, charter schools, or private schools in all locations. Follow-up reminders are sent every two weeks throughout the duration  of the survey window. 

In addition to asking questions about the completer’s demographics and educational  background, the survey also contains items to capture data about the school where the completer is employed. Additionally, the survey includes items asking about candidates’  perceptions of various aspects of the preparation program and the field placement experience. Campuses have access to annual results from the survey by utilizing the EdQ Dashboard. Results can be disaggregated by various measures including campus, year of completion, respondent race/ethnicity, and type of credential. Note: the CTC also distributes a Credential  Program Completer Survey which gives an overall view of CA Educator Preparation Programs. 

Used systemwide, the survey serves as a valid measure of graduates' perceptions of how well the teacher preparation program prepared them  for their first-year of teaching because it asks questions directly aligned  with the California Teacher Performance Expectations and California  Standards for the Teaching Profession.  

Additionally, the survey’s content is tailored to the type of program  each respondent completed, making the content valid for each  individual. For example, the survey for a Single Subject English  teachers contains an item about how well the program prepared them to develop students' understanding and use of academic language and  vocabulary whereas the survey for a Single Subject Social Science  teacher contains an item about how well the program prepared them to  develop students' Historical Interpretation skills. Similarly, surveys sent  to teachers with Multiple Subjects credentials or Educational Specialist  credentials respond to items directly aligned to standards associated  with their credentials. 

All graduates respond to items asking about their preparation of general pedagogical skills, such as their perception of how well the program  prepared them to differentiate instruction in the classroom. In this way,  the survey is a valid measure of completers’ perceptions of the  program.

Uncertainty about evaluation findings comes from two principal  sources: the number of evaluation participants and the extent of their  concurrence with each other. The evaluation findings become  increasingly certain to the extent that the questions are answered by  increasing numbers of program completers and their employment  supervisors.  

Each year the data set yields the percent of respondents who gave  specified answers to each item and includes reliability estimates in the  form of confidence intervals based on the number of respondents and  the concurrence or homogeneity of responses. The CSU Deans of  Education grouped together questions into "composites" (e.g.,  Preparing for Equity and Diversity in Education) for a more reliable  interpretation. The reliability for the composite scores for the system  and the individual campuses generally range from 0 to 2 percentage  points at the 90% confidence level. 

Data were not constructed with bias, and data show positive predictive  value (statistical parity) among groups and support equalized odds. 

The existence of this CSU-wide service allows each campus to track  the effects of program changes designed to improve performance.  Because the instrument was designed and is implemented systemwide  with completers throughout the state, we believe it is a fair and  trustworthy measure. 

Fresno State has initiated a college-wide data summit to consider the  findings of this statewide survey and triangulate them with campus  data, including the percentage of First Generation students, access to  resources like scholarships, and culture and context of the cohorts in  which prospective teachers are placed. Through this triangulation  process, we are able to determine the alignment of the finding from the  survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to  inform the impact on program changes on our own students with  respect to the unique diversity of culture and needs in the Central  Valley. 

New teachers are introduced to and provided support in using new tools to help them set goals to focus their continuous improvement efforts. This new teacher development practice is part of the teacher evaluation process which is recognized as a desirable method to achieve the improvement of instruction, to identify skills and abilities that contribute to the success of the educational program, and to redirect skills and abilities that do not result in optimum student growth. The goals of the evaluation are as follows: 

• To recognize the performance of outstanding employees 
• To enhance and improve performance through communications that is direct, clear, honest, immediate, frequent, and evidence based 
• To align professional growth to employees’ strengths and areas of improvement
• To provide avenues for informal and formal communications that build relationships

Fresno Unified is the third-largest school district in California, educating just over 73,000 K-12 students in the 2019-2020 academic year (CDE, n.d.). Of those students, 68% identified as Latinx, 10% identified as Asian, 9% identified as White, 8% identified as Black, and 2% identified with two or more races, and 60% of those children are dual-language learners (CDE Dataquest, n.d.).  Additionally, 85.9% of the students in the local school district receive free or reduced meals (CDE, n.d.). Fresno Unified also consistently places the largest number of Multiple Subject Credential Candidates with experienced mentor teachers in their district for their clinical experience (N 73; 26%). Moreover, Fresno Unified hires the majority of our MS program completers; 60-70 new hires each year. For these reasons, we launched a plan in March 2020 where Fresno Unified agreed to strengthen their data collection systems so that we could receive their teacher induction program data. 

The Formative & Summative New Teacher Evaluation is recognized as a desirable method to achieve the improvement of instruction, to identify skills and abilities that contribute to the success of the educational program, and to redirect skills and abilities that do not result in optimum student growth. The goals of the evaluation are as follows: 

• To recognize the performance of outstanding employees 
• To enhance and improve performance through communications that is direct, clear, honest, immediate, frequent, and evidence based 
• To align professional growth to employees’ strengths and areas of improvement
• To provide avenues for informal and formal communications that build relationships.  

District Instructional Coaches are trained and calibrated to use the district’s formative and summative evaluation rubric that is aligned with the California Standards for the Teaching Profession (CSTP), which is also aligned with the Teaching Performance Expectations that serve as a guide to our program’s curriculum and clinical experiences. New teachers are formally observed twice per year. The first formal observation takes place by the end of November and it is formative in nature. A full lesson is observed and followed up by a debrief within five days from the observation. The second formal observation takes place by the end of May and it is summative in nature.