Appendix E: Evidence of Data Quality

Multiple Subject Credential Program

Midterm & Final Fieldwork Evaluation

Quantitative Data Measure: Midterm & Final Fieldwork Evaluation (CREATe Rubric used formatively)
Description of Measure CREATe is a locally developed observation tool that provides a common language for preservice teachers, coaches, faculty, mentors, and district administrators to orient their feedback in an actionable manner. Through synthesis of existing district observation tools, and by aligning this synthesis to the new Teacher Performance Expectations (2016 CCTC adopted TPEs), the CREATe Rubric consists of 14 items organized within the following four domains:  1) Positive Environment, 2) Instructional Design and Implementation, 3) Rigorous and Appropriate Content, and 4) Reflection-In-Action. Each of the 14 items is rated along a seven-point developmental continuum with the following rating categories: Unobserved, Attempting, Exploring, Emerging, Developing, Skillful, and Masterful. Each rating category has an anchor descriptor that operationalizes each of the 14 items with action-oriented, observable “look-fors.” 
Evidence (or plans) regarding validity Rubric development began with close examination of the new state standards to ensure that the rubric would measure the skills required of program completion. A team with university and district representatives analyzed the six standards and 45 substandards to identify those that are critical for novice teachers and could be directly observed in a classroom setting. The selected observable standards were compared and synthesized, with nearly 100% consensus across all constituents regarding which observable standards should be represented on the rubric as a form of face validity. From this analysis, 14 rubric items were developed, aligned to 17 preservice substandards. Next, extant district inservice rubrics were synthesized and incorporated into CREATe so that it could be aligned and used across districts as a continuum of development from novice to expert teacher. Different districts’ inservice rubrics varied in regard to complexity and explicitness of how inservice standards were operationalized. These three district inservice rubrics were coded and synthesized to develop evidence-based language for common “look-fors” as descriptive anchors in the more advanced performance categories of CREATe Rubric. Based on the analysis and synthesis of the three inservice district rubrics, seven performance categories were developed for CREATe: five (unobserved, attempting, exploring, emerging, developing) spanning the expected developmental trajectory of preservice teachers and two (skillful, masterful) extending teacher development into inservice practice. Integration of the district inservice rubrics into CREATe explicitly bridges the instrument from preservice to inservice.
CREATe 2019 Validation Study 1-page overview
Evidence (or plans) regarding reliability

All of our MS coaches have participated in CREATe Rubric training. 60% of the MS coaches (N15) are fully reliable, and 7 more are partially reliable in that they are half way through the process and on their way to being fully reliable. 3 Faculty in Residence and 3 Teachers in Residence along with additional leadership team members from our district partners and Fresno State’s Continuous Improvement Lead participated in CREATe training. Each residency hosted the trainings which also included 2 live 20-minute observations with placing the teacher on the continuum followed by immediate discussion and reflection using the rubric after visiting the classroom. This model of training became a monthly community of practice to build connections and support between the three residencies (Fresno Unified, Sanger Unified, Central Unified). We also ensure interrater reliability using a rigorous observer training protocol in which observers must pass a paired observation based on reliability criteria before collecting live data in the field. Once we were able to establish interrater reliability, we collaborated with a partner school district’s New Teacher Induction Program to examine the validity of the CREATe Rubric. 

For concurrent validity, we used The New Teacher Project (TNTP) Core Rubric, which has established validity and is widely used. Pairs of calibrated CREATe and TNTP Core observers were assigned randomly to simultaneously observe in pre-selected classrooms. A strong partnership with the partner district New Teacher Induction program was the foundation to initiating and completing this validation study; the induction program staff helped us to identify a sample of 28 first-year teachers. The 28 participating teachers included nine graduates of the focal STaR Residency EPP and 15 non-residency graduates. The 15 non-residency group were a mix of candidates from other EPPs as well as candidates from the focal EPP who were not in the STaR Residency program (completed other pathways). 24 teachers were successfully observed during the observation window, with 4 teachers excluded due to scheduling/observation conflicts. Observer data was compiled and merged to enable data analysis and compare performance across measures to determine validity, and to provide information regarding necessary changes and revisions needed for continuous improvement in the STaR Residency program. For the current validation study, we used a paired observational design to observe 24 first-year teachers over a six-week period in the Spring 2019 semester using two independent observation tools: CREATe (Yun & Bennett, 2018) and the Core Teaching Rubric (TNTP, 2017). The TNTP Core Rubric was chosen because its four dimensions are conceptually well-aligned to the four dimensions of CREATe. Data collected by the observers included observation notes and rating scores. Notes and scores were collected either on paper or electronically on a laptop. Notes for the TNTP Core Rubric were sent to TNTP for compilation and a score report was generated for each of the observed teachers. Notes and ratings for CREATe were collected using a CREATe Score Sheet; ratings were manually entered into a spreadsheet. Data sources also include communications between the research team and district personnel as well as project documents such as timelines, calendars, and meeting notes. Analyses include confirmatory factor analysis, Pearson correlation, document analysis, and qualitative coding. Document analysis also suggests that CREATe has face validity when compared with Core. The four dimensions of the CREATe were well aligned with the four TNTP dimensions: Culture of Learning - Positive Environment; Academic Ownership - Instructional Design and Implementation; Essential Content; Rigorous and Appropriate Content; Demonstration of Learning - Reflection-in-Action (Core; CREATe) (see Figure 1). 

Regarding concurrent validity, CREATe and Core are measuring performance similarly; average performance on the Core rubric was significantly correlated with average performance on CREATe (r=0.35, p<0.10). This correlation suggests that mean performance were consistent across the two instruments. When the teachers were ranked according to average Core and CREATe scores, the rankings aligned 100%.

Confirmatory factor analysis (CFA) of CREATe suggests that the four proposed dimensions of the rubric hold up relatively well, with the exception of the Items 12 (Content Accessibility) and 13 (Interdisciplinary Integration). CREATe scores on 14 indicators, distributed across four dimensions. With the exception of the third dimension (which includes Items 12 and 13), a CFA and analysis of internal consistency both suggest that the other dimensions of the rubric hold together as outlined in the original design of the rubric (CFA, p<0.01;  factor loadings excluding #12/#13; a>0.8). Overall findings suggest that CREATe differentiates performance trends across three of its four predetermined dimensions (internal consistency/reliability), and that overall performance trends on CREATe are consistent across a different observation instrument, suggesting that CREATe measures teacher performance relatively well. Qualitative coding indicated that qualitative feedback for CREATe provides more evidence about the lesson itself, while qualitative feedback for Core provides more evidence about the scoring decision. This finding suggests fidelity in use of CREATe as the protocol calls for time-stamped scripted notes of classroom activities. 

Evidence (or plans) regarding fairness

Survey and Interview data with various stakeholders engaged with the CREATe Rubric have been conducted to better understand their experience with the measure. The CREATe Rubric has been well-received by district leadership and candidates. Coaches and faculty have been split on their reception of the CREATe rubric. Some of their concerns center on the length and cognitive load of the tool as well as disagreements around who is responsible for introducing the rubric to the credential students. When faculty who were the lead developers of the CREATe moved on from the program and coaches continued to question the utility of the CREATe Rubric, an assessment of all of the available rubrics were conducted. Additionally presentations and surveys to gather coach, faculty, and district feedback was administered. Based on this input, a decision was made in December 2020 to transition to the New Teacher Project (TNTP) Core rubric, which Chico State adapted to align with the CTC Standards. A Rubric Advisory Board had formed consisting of representatives from all three basic credential programs and an implementation timeline plus additional adaptations were made. However, by March 2020 California moved into its first COVID-19 lockdown putting the new formative rubric implementation timeline on hold.  

Of all the stakeholders, the candidates themselves have seemed to demonstrate the most positive responses. Candidates appreciate the explicit teacher behaviors in CREATe and their alignment to the TPEs. This helps candidates make the connections between the TPEs/CSTPs and their enacted practice. It also helps candidates to operationalize the concepts of developmentally appropriate practices, universal design for learning, culturally and linguistically sustaining practices, and inquiry. CREATe helps candidates construct what teaching looks like and focus on specific moves they can practice and improve. The quotes below demonstrate candidate perceptions of CREATe.

“The Create Rubric worked well for me in my clinical practice. It was a great guideline in what I need to be striving for and if I was meeting my goals as a teacher. In my opinion, without a rubric, we wouldn't have anything to base our teaching on, or have anything to strive for. It also sets a great goal and framework as teachers in different categories.” -Teacher Candidate Spring 2018

“The use of the CREATe Rubric as a planning tool for preservice teachers is a great resource because it helps us guide our lesson planning and implementation of the different aspects of what as teachers we should be doing every single day.” -Teacher Candidate Fall 2018

One thing is evident, ALL candidates responded well to a formative rubric that provided common language to make the skills of teaching more visible so they could receive actionable feedback from their coaches. This makes moving toward initial implementation of the new rubric ever more pressing.  

Fresno Assessment of Student Teachers II (FAST II)

Quantitative Data Measure: Fresno Assessment of Student Teachers II (FAST II)
Description of Measure FAST II consists of two projects: the Site Visitation Project (SVP) is completed during initial student teaching (EHD 178) and the Teaching Sample Project (TSP) is completed during final student teaching (EHD 170). The SVP assesses teacher candidates’ ability to plan, implement, and evaluate instruction. The three parts of the project include (1) Planning: planning documentation for a single lesson incorporating state-adopted content standards and English language development, (2) Implementation: an in-person observation and videotaping of the teaching of the lesson, (3) Reflection: a review of the entire video, selection of a 3- to 5-minute video segment, and a written evaluation of the lesson. (TPE 1.1, 1.3, 1.5, 1.8, 2.2, 2.6, 3.1, 3.2, 3.3, 3.5, 4.1, 4.2, 4.7, 6.1). The Teaching Sample Project assesses teacher candidates’ ability to (a) identify the context of the classroom, (b) plan and teach a series of at least five cohesive lessons with a focus on content knowledge and literacy, (c) assess students’ learning related to the unit, (d) document their teaching and their students’ learning, and (e) reflect on the effectiveness of their teaching. Teacher candidates document how they are addressing the needs of all their students in the planning, teaching, and assessing of the content. (TPE 1.5, 1.6, 1.8, 2.1, 2.3, 2.6, 3.1, 3.2, 3.3, 4.1, 4.3, 4.4, 4.7, 5.1, 5.2, 5.5, 5.8, 6.1, 6.3, 6.5). 
Evidence (or plans) regarding validity The SVP assesses the candidate’s ability to plan, implement and reflect upon instruction.  Each of these abilities is assessed with performance tasks: the lesson plan (planning), teaching the lesson (implementation) and self-evaluation of the lesson (reflect upon instruction). In order to assess the teaching performance expectations (TPE) the tasks each have a rubric which share the same categories: subject specific pedagogy, applying knowledge of students and student engagement. The categories are rated on a 4-point scale (1-does not meet expectations, 2-meets expectations, 3-meets expectations at a high level, 4-exceeds expectations). The wording in the rubrics is adapted to each of the three specific tasks. Data from the FAST indicate that students are developing the competencies that are essential to effective classroom teaching practice.  
Evidence (or plans) regarding reliability Every 2 years, a psychometric analysis of the Site Visitation Project (SVP) is performed. Our most recent analysis found that of the 15% of the SVPs that were double scored, 70% gave the same score and 100% were within +/-1. 94.7% agreed on the determination of whether the SVP should pass or not. 
Evidence (or plans) regarding fairness To monitor equity, the three subtests and the final score were examined as part of our psychometric analysis in regards to differences based on students’ ethnicity, gender, whether the student first language was English, the students’ self-rated degree of English language fluency on a 5-point Likert scale, and self-reported disability.  In an effort to examine scoring equity, a series of non-parametric statistical tests were calculated to determine whether significant differences in scoring corresponded to students’ demographic characteristics. When examining the three subtests only one comparison showed statistically significant differences, the self-rated degree of English language fluency in the observation task. The statistical analyses for disability were not conducted, because of a very small sample size of 2 students self-reporting a disability. The scores were tabulated and inspected, all scores were passing.
Evidence regarding Trustworthiness Developed over a number of years with the support of the Renaissance Group and a Title II grant, the FAST addresses each of California’s TPEs. Each assessment is scored by at least two faculty members, including the university coach assigned to mentor the teacher candidate. Mandatory calibration sessions are held annually, and all scorers must participate in the norming process each year. The inter-rater reliability is higher than the norm for such assessments. Moreover, students who fail the assessment have the opportunity to revise and resubmit.

Journey Mapping

Quantitative Data Measure: Journey Mapping
Description of Measure Journey maps help understand candidate’s experiences but they provide unique insights as they document individual candidates’ experiences over time by anchoring them to memorable emotional highs and lows during their time in the program (Rains, 2017). The Journey Map is collected as a reflective, in-class activity facilitated by an instructor on record or a member from the program research team. As a result, journey maps offer opportunities for multi-levels of reflective purpose: at the individual candidate level, at the individual instructor level who can reflect on their class of candidates as a whole, and at the program level where trends can be identified across time.
Evidence (or plans) regarding validity The journey map measures what is intended to measure in that it captures students' experiences of program milestones and their own memorable emotional highs and lows in the program in a way that privileges a qualitative approach.  
Evidence (or plans) regarding reliability n/a
Evidence (or plans) regarding fairness The journey maps are collected during the last two weeks of the final phase of the program offering all students the opportunity to participate in providing feedback on their lived experience in the program. It can be administered either face to face or virtually. 
Evidence regarding Trustworthiness Journey maps are coded to identify emergent themes pointing to potential areas for program improvement. After reading through a sample of the journey maps to become familiar with the data, a focused coding scheme was developed to help us look for similar information across all of the candidates maps while also noting new themes as they emerged. The focused coding scheme also allowed us to engage in interrater reliability practices. Two coders would first meet to discuss what they see in the same journey map, then they would analyze the map using the scheme and then discuss where the analysis was confirming and disconfirming to then determine how to interpret the scheme moving forward. The coders then analyze a set of five journey maps from the same cohort and meet to again compare and discuss their analysis. Once the coders saw more alignment in their coding practices then they would code a full cohort. Then, a third coder who has also been trained to use the coding scheme would code 10% of the other coders’ maps to compare the analysis as one way to strengthen the trustworthiness of the journey map data. The focused coding scheme allowed us to calculate counts of candidates' experiences while we also paid close attention and still documented their experiences from a nuanced, qualitative perspective. Events were coded as either negative or positive. Program highlights were also identified.

CSU Educator Quality Center Completer Survey

Quantitative Data Measure: CSU Educator Quality Center Completer Survey
Description of Measure The California State University’s Education Quality Center (EdQ) oversees the  administration of a completer-survey to exiting candidates of all CSU teacher-preparation  programs. The survey is available year-round and campuses are encouraged to make completion of the survey a component of graduates’ final paperwork. The survey contains items asking  about candidates’ perceptions of various aspects of the preparation program and the field  placement experience. Campuses have access to annual results from the survey by utilizing the EdQ Dashboard. Results can be disaggregated by various measures including campus, year of  completion, respondent race/ethnicity, and type of credential. Note: the CTC also distributes a Credential Program Completer Survey which gives an overall view of CA Educator Preparation  Programs. 
Evidence (or plans) regarding validity Used systemwide, the survey serves as a valid measure of program completers’ perceptions of the teacher preparation program because it asks questions directly aligned with the California Teacher Performance Expectations and California Standards for the Teaching Profession. Additionally, the survey’s content is tailored to the type of program each respondent completed, making the content valid for each individual. For example, the survey for a Single Subject English program completer contains an item about how well the program prepared them to develop students' understanding and use of academic language and vocabulary whereas the survey for a Single Subject Social Science program completer contains an item about how well the program prepared them  to develop students' Historical Interpretation skills. All program completers respond to items asking about their preparation of general  pedagogical skills, such as their perception of how well the program prepared them to differentiate instruction in the classroom. In this way, the survey is a valid measure of completers’ perceptions of the program.
Evidence (or plans) regarding reliability Uncertainty about evaluation findings comes from two principal sources: the number of evaluation participants and the extent of their  concurrence with each other. The evaluation findings become  increasingly certain to the extent that the questions are answered by  increasing numbers of program completers and their employment  supervisors. Each year the data set yields the percent of respondents who  gave specified answers to each item and includes reliability estimates in  the form of confidence intervals based on the number of respondents  and the concurrence or homogeneity of responses. The CSU Deans of  Education grouped together questions into "composites" (e.g., Preparing  for Equity and Diversity in Education) for a more reliable interpretation.  The reliability for the composite scores for the system and the individual campuses generally range from 0 to 2 percentage points at the 90%  confidence level. 
Evidence (or plans) regarding fairness

The existence of this CSU-wide service allows each campus to track the  effects of program changes designed to improve performance. Because  the instrument was designed and is implemented systemwide with  graduates throughout the state, we believe it is a fair and trustworthy  measure. 

Fresno State has initiated a college-wide data summit to consider the  findings of this statewide survey and triangulate them with campus data,  including the percentage of First Generation students, access to  resources like scholarships, and culture and context of the cohorts in  which prospective teachers are placed. Through this triangulation  process, we are able to determine the alignment of the finding from the  survey with our other measures, further assuring us of the survey’s  trustworthiness as an instrument. In the process, we are also able to  inform the impact on program changes on our own students with respect  to the unique diversity of culture and needs in the Central Valley. 

CCTC Employer Survey

Quantitative Data Measure: CCTC Employer Survey 
Description of Measure

Beginning in 2019, the CTC sends an annual survey to employers of recent completers of all educators preparation programs in the state. The goal of the survey is to compile evidence statewide about the extent to which K-12 educators are prepared for their most important responsibilities.  

The survey includes 11 items. The first five of these ask basic demographic details about where the new educator was prepared and about the responding employer. The remaining six items are aligned with the California Teaching Performance Expectations.

The results of the survey are disaggregated for CSUs, UCs, Private, and Local Education Agencies. 79% of  respondents are principals. Of the 766 employers who responded to the survey in 2018-2019  (the last year for which data are available), 53% employed are recent graduates from a CSU teacher preparation program. 

Evidence (or plans) regarding validity

Used statewide, the survey serves as a valid measure of employers' perceptions of how well educator preparation programs prepared alumni for their first year. Additionally, items included in the survey are directly aligned with the California Teacher Performance Expectations.  

All employers respond to items asking about the preparation of new teachers in general pedagogical skills, such as their perception of how well the program prepared new teachers to differentiate instruction in the classroom. In this way, the survey is a valid measure of employers’ perceptions of the program.

Evidence (or plans) regarding reliability

Uncertainty about evaluation findings comes from two principal sources: the number of evaluation participants and the extent of their concurrence with each other. The evaluation findings become increasingly certain to the extent that the questions are answered by increasing numbers of program completers and their employment  supervisors.  

Each year the data set yields the percent of respondents who gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents and  the concurrence or homogeneity of responses. 

Evidence (or plans) regarding fairness

Surveys are sent to all administrators within the state, giving all the opportunity to share their perspective on the educator preparation program their new teachers attended. In this way, the survey does not discriminate in who is invited to respond and whose voice is heard. Additionally, employers have a three-month window (October 1-December 31 annually) in which to complete the survey, providing ample opportunity to respond. 

The existence of this CCTC-wide service allows each preparation program to track the effects of program changes designed to improve performance. Because the instrument was designed and is implemented to employers throughout the state, we believe it is a fair and trustworthy measure. 

CSU Year One Completer Survey

Quantitative Data Measure: CSU Year One Completer Survey
Description of Measure

The California State University’s Education Quality Center (EdQ) oversees the  administration of a survey of all individuals who completed a CSU teacher-preparation programs after their first year on the job. The survey is administered annually April through July. In April, the EdQ Center emails an initial survey invitation to all completers of MS-SSES Credential Programs serving as first-year teachers in public schools, charter schools, or private schools in all locations. Follow-up reminders are sent every two weeks throughout the duration  of the survey window. 

In addition to asking questions about the completer’s demographics and educational  background, the survey also contains items to capture data about the school where the completer is employed. Additionally, the survey includes items asking about candidates’  perceptions of various aspects of the preparation program and the field placement experience. Campuses have access to annual results from the survey by utilizing the EdQ Dashboard. Results can be disaggregated by various measures including campus, year of completion, respondent race/ethnicity, and type of credential. Note: the CTC also distributes a Credential  Program Completer Survey which gives an overall view of CA Educator Preparation Programs. 

Evidence (or plans) regarding validity

Used systemwide, the survey serves as a valid measure of graduates' perceptions of how well the teacher preparation program prepared them  for their first-year of teaching because it asks questions directly aligned  with the California Teacher Performance Expectations and California  Standards for the Teaching Profession.  

Additionally, the survey’s content is tailored to the type of program  each respondent completed, making the content valid for each  individual. For example, the survey for a Single Subject English  teachers contains an item about how well the program prepared them to develop students' understanding and use of academic language and  vocabulary whereas the survey for a Single Subject Social Science  teacher contains an item about how well the program prepared them to  develop students' Historical Interpretation skills. Similarly, surveys sent  to teachers with Multiple Subjects credentials or Educational Specialist  credentials respond to items directly aligned to standards associated  with their credentials. 

All graduates respond to items asking about their preparation of general pedagogical skills, such as their perception of how well the program  prepared them to differentiate instruction in the classroom. In this way,  the survey is a valid measure of completers’ perceptions of the  program.

Evidence (or plans) regarding reliability

Uncertainty about evaluation findings comes from two principal  sources: the number of evaluation participants and the extent of their  concurrence with each other. The evaluation findings become  increasingly certain to the extent that the questions are answered by  increasing numbers of program completers and their employment  supervisors.  

Each year the data set yields the percent of respondents who gave  specified answers to each item and includes reliability estimates in the  form of confidence intervals based on the number of respondents and  the concurrence or homogeneity of responses. The CSU Deans of  Education grouped together questions into "composites" (e.g.,  Preparing for Equity and Diversity in Education) for a more reliable  interpretation. The reliability for the composite scores for the system  and the individual campuses generally range from 0 to 2 percentage  points at the 90% confidence level. 

Evidence (or plans) regarding fairness

Data were not constructed with bias, and data show positive predictive  value (statistical parity) among groups and support equalized odds. 

The existence of this CSU-wide service allows each campus to track  the effects of program changes designed to improve performance.  Because the instrument was designed and is implemented systemwide  with completers throughout the state, we believe it is a fair and  trustworthy measure. 

Fresno State has initiated a college-wide data summit to consider the  findings of this statewide survey and triangulate them with campus  data, including the percentage of First Generation students, access to  resources like scholarships, and culture and context of the cohorts in  which prospective teachers are placed. Through this triangulation  process, we are able to determine the alignment of the finding from the  survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to  inform the impact on program changes on our own students with  respect to the unique diversity of culture and needs in the Central  Valley. 

Pre and Post Dispositions Survey 

Quantitative Data Measure: Pre and Post Dispositions Survey 
Description of Measure Complementing the 23 CSU Completer Exit Survey of graduates, the Pre-Post Dispositions Survey is administered at the beginning and end of each fieldwork course (EHD 178 and EHD 170). The California Commission on Teacher Credentialing (CCTC) requires all candidates to  demonstrate personality and character traits that satisfy the standards of the teaching profession through a 9-item measure of these traits. Thus, Fresno State developed this Teacher Commitment Statement consisting of six professional dispositions teacher candidates complete as part of their entrance requirements and when they complete their credential program. 
Evidence (or plans) regarding validity Items included within the commitment statement align with the dispositions CTC requires of candidates credentialed to teach within the state. It should be noted that commitment statements rely upon self reported dispositions, which can be inaccurately represented.
Evidence (or plans) regarding reliability Reliability of the candidate’s commitment is dependent upon self reported dispositions that can be inaccurately represented.
Evidence (or plans) regarding fairness The commitment statement is intended to reinforce the values of fairness  among the candidates as well as an expectation of non-biased  dispositions of the candidates toward students of all backgrounds,  languages, cultures, and experiences. In the instance when incoming  candidates or exiting graduates perform aberrantly on this commitment  statement, they are identified, counseled, and advised about their pursuit  of the profession. In the rare instance of candidates not agreeing with the  necessary commitments, they may resubmit. 

Course grades/Candidate Performance in Courses

Quantitative Data Measure: Course grades/Candidate Performance in Courses
Description of Measure Credential candidates must maintain a 3.00 GPA in all credential courses with no  individual grade lower than a “C”. Any grade listed as “I”, “IC”, “WU”, “NC”, “D”, or “F”  does not meet Fresno State’s credential program requirements. 
Evidence (or plans) regarding validity Course grades of ‘C’ or ‘Credit’ are required for program completion. Therefore, with few exceptions, all candidates must complete and receive a ‘C’ grade or better for the articulated courses. Faculty hired to teach within the programs are considered experts in their specific field and all have relevant experience for the content of the courses which they are teaching. 
CTC requires a 3.0 GPA for the preliminary credential.
Evidence (or plans) regarding reliability Requirements in courses stay relatively consistent over time since they are aligned with the CCTC program standards. In addition, the courses are staffed by the same faculty, on the whole. 
Evidence (or plans) regarding fairness Course grades are required at the end of the course. With few exceptions, all program completers must complete and receive a grade for courses taken. For each course, specific details are provided within course syllabi about requirements for course assignments and to earn a passing grade. 

Sample Narrative Community Context Assignment

Quantitative Data Measure: Sample Narrative Community Context Assignment
Description of Measure For this assignment, teacher candidates conducted a thorough examination of the community, the school, and the specific classroom in which they are completing their fieldwork. Teacher candidates are required to research the community in which they serve to better understand and support the students in the classroom in which they complete their fieldwork experience. Candidates are also required to develop an understanding of the school in which they work, the intricacies of the school, the demographics of the school, as well as the aspects that make the school special. Finally, teacher candidates get to know the students in their classroom on a deeper level, and determine how to best meet the needs of all of their students. Students are instructed to use asset-based perspectives when referring to each aspect of the context. 
Evidence (or plans) regarding validity The data source measures the content it intends to measure through a focused, predetermined coding scheme focused on asset-based or strengths driven perspectives (Green & Haines, 2011) and knowledge of specific cultural capital that students, families, and communities contribute to the classroom and community (Yosso, 2005). The codersare instructors who teach the inquiry and puzzle of practice series with a deep knowledge of the literature used to develop the focused coding scheme. The coders review their codes together to norm the coding process to help increase the overall validity of the process. 
Evidence (or plans) regarding reliability This is a new measure and reliability will be assessed to determine whether the focused coding scheme is applied consistently by the trained raters over time. 
Evidence (or plans) regarding fairness Student work from phase 2 and 3 of the inquiry and puzzle of practice series is randomly selected for the sample.

New Teacher Goals for Continuous Improvement

Quantitative Data Measure: New Teacher Goals for Continuous Improvement
Description of Measure

New teachers are introduced to and provided support in using new tools to help them set goals to focus their continuous improvement efforts. This new teacher development practice is part of the teacher evaluation process which is recognized as a desirable method to achieve the improvement of instruction, to identify skills and abilities that contribute to the success of the educational program, and to redirect skills and abilities that do not result in optimum student growth. The goals of the evaluation are as follows: 

  • To recognize the performance of outstanding employees 
  • To enhance and improve performance through communications that is direct, clear, honest, immediate, frequent, and evidence based 
  • To align professional growth to employees’ strengths and areas of improvement
  • To provide avenues for informal and formal communications that build relationships
Evidence (or plans) regarding validity District Instructional Coaches are trained to support new teachers in developing an evaluation plan where they set goals for continuous improvement during the first six weeks of the academic year. Overtime, the district instructional coaches introduce the new teachers to tools aimed at helping them improve their instructional practice. They log how often a new teacher uses various tools.   
Evidence (or plans) regarding reliability District induction coaches are trained to use the observation rubric. Norming and calibration is consistently checked for during induction coach meetings. 
Evidence (or plans) regarding fairness The process is governed by the new teacher’s collective bargaining agreement which offers the teachers protections and clearly defined processes.

Formative & Summative New Teacher Evaluation

Quantitative Data Measure: Formative & Summative New Teacher Evaluation
Description of Measure

Fresno Unified is the third-largest school district in California, educating just over 73,000 K-12 students in the 2019-2020 academic year (CDE, n.d.). Of those students, 68% identified as Latinx, 10% identified as Asian, 9% identified as White, 8% identified as Black, and 2% identified with two or more races, and 60% of those children are dual-language learners (CDE Dataquest, n.d.).  Additionally, 85.9% of the students in the local school district receive free or reduced meals (CDE, n.d.). Fresno Unified also consistently places the largest number of Multiple Subject Credential Candidates with experienced mentor teachers in their district for their clinical experience (N 73; 26%). Moreover, Fresno Unified hires the majority of our MS program completers; 60-70 new hires each year. For these reasons, we launched a plan in March 2020 where Fresno Unified agreed to strengthen their data collection systems so that we could receive their teacher induction program data. 

The Formative & Summative New Teacher Evaluation is recognized as a desirable method to achieve the improvement of instruction, to identify skills and abilities that contribute to the success of the educational program, and to redirect skills and abilities that do not result in optimum student growth. The goals of the evaluation are as follows: 

  • To recognize the performance of outstanding employees 
  • To enhance and improve performance through communications that is direct, clear, honest, immediate, frequent, and evidence based 
  • To align professional growth to employees’ strengths and areas of improvement
  • To provide avenues for informal and formal communications that build relationships.  

District Instructional Coaches are trained and calibrated to use the district’s formative and summative evaluation rubric that is aligned with the California Standards for the Teaching Profession (CSTP), which is also aligned with the Teaching Performance Expectations that serve as a guide to our program’s curriculum and clinical experiences. New teachers are formally observed twice per year. The first formal observation takes place by the end of November and it is formative in nature. A full lesson is observed and followed up by a debrief within five days from the observation. The second formal observation takes place by the end of May and it is summative in nature.  

Evidence (or plans) regarding validity Tool developed by the New Teachers Center that measures employees’ strengths and areas of improvement over time.  
Evidence (or plans) regarding reliability District induction coaches are trained to use the observation rubric. Norming and calibration is consistently checked for during induction coach meetings. 
Evidence (or plans) regarding fairness The process is governed by the new teacher’s collective bargaining agreement which offers the teachers protections and clearly defined processes. 

New Teacher Professional Learning Participation

Quantitative Data Measure: New Teacher Professional Learning Participation
Description of Measure Fresno Unified is the third largest school district in California, educating just over 73,000 K-12 students in the 2019-2020 academic year (CDE, n.d.). Of those students, 68% identified as Latinx, 10% identified as Asian, 9% identified as White, 8% identified as Black, and 2% identified with two or more races, and 60% of those children are dual-language learners (CDE Dataquest, n.d.).  Additionally, 85.9% of the students in the local school district receive free or reduced meals (CDE, n.d.). Fresno Unified also consistently places the largest number of Multiple Subject Credential Candidates with experienced mentor teachers in their district for their clinical experience (N 73; 26%). Moreover, Fresno Unified hires the majority of our MS program completers; 60-70 new hires each year. For these reasons, we launched a plan in March 2020 where Fresno Unified agreed to strengthen their data collection systems we could have opportunities to receive their teacher induction program data. New teachers are invited to participate in four-hour professional learning sessions twice a month on Saturdays. During these sessions new teachers are working collaboratively with their colleagues to examine their practice as well as discuss how to apply new practices in their classrooms. New teacher attendance to the professional learning sessions is an indicator of their ability to collaborate with various colleagues, learn from each other, and share knowledge and resources with each other. 
Evidence (or plans) regarding validity Attendance data measures the frequency in which new teachers participate in Saturday PL which is what we intend to measure by this data. 
Evidence (or plans) regarding reliability New teachers are reminded to sign-in through an electronic form which then links to the Teacher Development Database. Teachers are accustomed to the routine of signing in to prove their participation in PL. 
Evidence (or plans) regarding fairness All new teachers are invited to participate at no-cost. 

Appendix E: