Appendix E: Evidence of Data Quality

Single Subject Credential Program

Fresno Assessment of Student Teachers II (FAST II)

Quantitative Data Measure: Fresno Assessment of Student Teachers II (FAST II)
Description of Measure FAST II consists of two projects: the Site Visitation Project (SVP) is completed during initial student teaching (EHD 178) and the Teaching Sample Project (TSP) is completed during final student teaching (EHD 170). The SVP assesses teacher candidates’ ability to plan, implement, and evaluate instruction. The three parts of the project include (1) Planning: planning documentation for a single lesson incorporating state-adopted content standards and English language development, (2) Implementation: an in-person observation and videotaping of the teaching of the lesson, (3) Reflection: a review of the entire video, selection of a 3- to 5-minute video segment, and a written evaluation of the lesson. (TPE 1.1, 1.3, 1.5, 1.8, 2.2, 2.6, 3.1, 3.2, 3.3, 3.5, 4.1, 4.2, 4.7, 6.1). The Teaching Sample Project assesses teacher candidates’ ability to (a) identify the context of the classroom, (b) plan and teach a series of at least five cohesive lessons with a focus on content knowledge and literacy, (c) assess students’ learning related to the unit, (d) document their teaching and their students’ learning, and (e) reflect on the effectiveness of their teaching. Teacher candidates document how they are addressing the needs of all their students in the planning, teaching, and assessing of the content. (TPE 1.5, 1.6, 1.8, 2.1, 2.3, 2.6, 3.1, 3.2, 3.3, 4.1, 4.3, 4.4, 4.7, 5.1, 5.2, 5.5, 5.8, 6.1, 6.3, 6.5). 
Evidence (or plans) regarding validity The SVP assesses the candidate’s ability to plan, implement and reflect upon instruction.  Each of these abilities is assessed with performance tasks: the lesson plan (planning), teaching the lesson (implementation) and self-evaluation of the lesson (reflect upon instruction). In order to assess the teaching performance expectations (TPE) the tasks each have a rubric which share the same categories: subject specific pedagogy, applying knowledge of students and student engagement. The categories are rated on a 4-point scale (1-does not meet expectations, 2-meets expectations, 3-meets expectations at a high level, 4-exceeds expectations). The wording in the rubrics is adapted to each of the three specific tasks. Data from the FAST indicate that students are developing the competencies that are essential to effective classroom teaching practice.
Evidence (or plans) regarding reliability Every 2 years, a psychometric analysis of the Site Visitation Project (SVP) is performed. Our most recent analysis found that, of the 15% of the SVPs that were double scored, 70% gave the same score and 100% were within +/-1. 94.7% agreed on the determination of whether the SVP should pass or not. 
Evidence (or plans) regarding fairness To monitor equity, the three subtests and the final score were examined as part of our psychometric analysis in regards to differences based on students’ ethnicity, gender, whether the student first language was English, the students’ self-rated degree of English language fluency on a 5-point Likert scale, and self-reported disability.  In an effort to examine scoring equity, a series of non-parametric statistical tests were calculated to determine whether significant differences in scoring corresponded to students’ demographic characteristics. When examining the three subtests only one comparison showed statistically significant differences, the self-rated degree of English language fluency in the observation task. The statistical analyses for disability were not conducted, because of a very small sample size of 2 students self-reporting a disability. The scores were tabulated and inspected, all scores were passing.
Evidence regarding Trustworthiness Developed over a number of years with the support of the Renaissance Group and a Title II grant, the FAST addresses each of California’s TPEs. Each assessment is scored by at least two faculty members, including the university coach assigned to mentor the teacher candidate. Mandatory calibration sessions are held annually, and all scorers must participate in the norming process each year. The inter-rater reliability is higher than the norm for such assessments. Moreover, students who fail the assessment have the opportunity to revise and resubmit.

CSU Educator Quality Center Completer Survey

Quantitative Data Measure: CSU Educator Quality Center Completer Survey
Description of Measure The California State University’s Education Quality Center (EdQ) oversees the  administration of a completer-survey to exiting candidates of all CSU teacher-preparation  programs. The survey is available year-round and campuses are encouraged to make completion  of the survey a component of graduates’ final paperwork. The survey contains items asking  about candidates’ perceptions of various aspects of the preparation program and the field  placement experience. Campuses have access to annual results from the survey by utilizing the  EdQ Dashboard. Results can be disaggregated by various measures including campus, year of  completion, respondent race/ethnicity, and type of credential. 
Evidence (or plans) regarding validity Used systemwide, the survey serves as a valid measure of program  completers’ perceptions of the teacher preparation program because it  asks questions directly aligned with the California Teacher Performance  Expectations and California Standards for the Teaching Profession.  
Evidence (or plans) regarding reliability Uncertainty about evaluation findings comes from two principal sources: the number of evaluation participants and the extent of their  concurrence with each other. The evaluation findings become  increasingly certain to the extent that the questions are answered by  increasing numbers of program completers and their employment  supervisors. Each year the data set yields the percent of respondents who  gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents  and the concurrence or homogeneity of responses. 
The CSU Deans of Education grouped together questions into "composites" (e.g., Preparing  for Equity and Diversity in Education) for a more reliable interpretation.  The reliability for the composite scores for the system and the individual campuses generally range from 0 to 2 percentage points at the 90%  confidence level. 
Evidence (or plans) regarding fairness/trustworthiness The existence of this CSU-wide service allows each campus to track the  effects of program changes designed to improve performance. Because  the instrument was designed and is implemented systemwide with  graduates throughout the state, we believe it is a fair and trustworthy  measure.
Fresno State has initiated a college-wide data summit to consider the  findings of this statewide survey and triangulate them with campus data,  including the percentage of First Generation students, access to  resources like scholarships, and culture and context of the cohorts in  which prospective teachers are placed. Through this triangulation  process, we are able to determine the alignment of the finding from the  survey with our other measures, further assuring us of the survey’s  trustworthiness as an instrument. In the process, we are also able to  inform the impact on program changes on our own students with respect  to the unique diversity of culture and needs in the Central Valley. 

CSU Educator Quality Center Employer Survey 

Quantitative Data Measure: CSU Educator Quality Center Employer Survey 
Description of Measure Until 2018, the CSU distributed an employer survey to employers of recent  graduates of CSU teacher preparation programs. Like the CSU completer survey, the employer  survey items were tailored to the type of preparation program from the new teacher completed  (Multiple Subject, Single Subject-Math, Single Subject-English, Education Specialist, etc.). 
Survey items target the following areas: Engaging and Supporting all Students in Learning,  Creating and Maintaining Effective Environments for Student Learning, Understanding and  Organizing Subject Matter for Student Learning, Planning Instruction and Designing Learning Experiences for All Students, Assessing Students for Learning, Developing as a Professional  Educator, and an overall assessment of how well prepared graduates of the institution are to be  teachers.  
The results are disaggregated by campus and allow campuses to evaluate their effectiveness from the perspective of employers of completers.
Evidence (or plans) regarding validity Used systemwide, the survey serves as a measure of employers' perceptions of how well programs prepared their completers for their first-year of teaching. Items on the survey are aligned with the California Teacher Performance Expectations and California Standards for the Teaching Profession.  
All employers respond to items asking about their preparation of the new teachers’ general  pedagogical skills, such as their perception of how well the program  prepared them to differentiate instruction in the classroom. In this way,  the survey is a valid measure of employers’ perceptions of the  program.
Evidence (or plans) regarding reliability Uncertainty about evaluation findings comes from two principal  sources: the number of evaluation participants and the extent of their  concurrence with each other. The evaluation findings become  increasingly certain to the extent that the questions are answered by  increasing numbers of program completers and their employment  supervisors.  
Each year the data set yields the percent of respondents who gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents and  the concurrence or homogeneity of responses. The CSU Deans of  Education grouped together questions into "composites" (e.g., Preparing for Equity and Diversity in Education) for a more reliable  interpretation. The reliability for the composite scores for the system  and the individual campuses generally range from 0 to 2 percentage  points at the 90% confidence level. 
Evidence (or plans) regarding fairness/trustworthiness Data were not constructed with bias, and data show positive predictive  value (statistical parity) among groups and support equalized odds. 
The existence of this CSU-wide service allows each campus to track  the effects of program changes designed to improve performance. Because the instrument was designed and is implemented systemwide  with completers throughout the state, we believe it is a fair and trustworthy measure. 
Fresno State has initiated a college-wide data summit to consider the findings of this statewide survey and triangulate them with campus  data, including the percentage of First Generation students, access to resources like scholarships, and culture and context of the cohorts in which prospective teachers are placed. Through this triangulation process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to  inform the impact on program changes on our own students with  respect to the unique diversity of culture and needs in the Central  Valley. 

CSU Educator Quality Center Year One Completer Survey

Quantitative Data Measure: CSU Educator Quality Center Year One Completer Survey
Description of Measure The California State University’s Education Quality Center (EdQ) oversees the  administration of a survey of all individuals who completed a CSU teacher-preparation programs after their first year on the job. The survey is administered annually April through  July. In April, the EdQ Center emails an initial survey invitation to all completers of MS-SSES  Credential Programs serving as first-year teachers in public schools, charter schools, or private  schools in all locations. Follow-up reminders are sent every two weeks throughout the duration  of the survey window. 
In addition to asking questions about the completer’s demographics and educational  background, the survey also contains items to capture data about the school where the completer is employed. Additionally, the survey includes items asking about candidates’  perceptions of various aspects of the preparation program and the field placement experience. Campuses have access to annual results from the survey by utilizing the EdQ Dashboard.  Results can be disaggregated by various measures including campus, year of completion, respondent race/ethnicity, and type of credential. Note: the CTC also distributes a Credential Program Completer Survey which gives an overall view of CA Educator Preparation Programs. 
Evidence (or plans) regarding validity Used systemwide, the survey serves as a valid measure of graduates'  perceptions of how well the teacher preparation program prepared them  for their first-year of teaching because it asks questions directly aligned  with the California Teacher Performance Expectations and California  Standards for the Teaching Profession.  
Additionally, the survey’s content is tailored to the type of program  each respondent completed, making the content valid for each  individual. For example, the survey for a Single Subject English  teachers contains an item about how well the program prepared them to develop students' understanding and use of academic language and  vocabulary whereas the survey for a Single Subject Social Science  teacher contains an item about how well the program prepared them to develop students' Historical Interpretation skills. Similarly, surveys sent to teachers with Multiple Subjects credentials or Educational Specialist  credentials respond to items directly aligned to standards associated  with their credentials. 
All graduates respond to items asking about their preparation of general  pedagogical skills, such as their perception of how well the program  prepared them to differentiate instruction in the classroom. In this way,  the survey is a valid measure of completers’ perceptions of the  program.
Evidence (or plans) regarding reliability Uncertainty about evaluation findings comes from two principal sources: the number of evaluation participants and the extent of their  concurrence with each other. The evaluation findings become  increasingly certain to the extent that the questions are answered by  increasing numbers of program completers and their employment  supervisors.  
Each year the data set yields the percent of respondents who gave  specified answers to each item and includes reliability estimates in the  form of confidence intervals based on the number of respondents and  the concurrence or homogeneity of responses. The CSU Deans of  Education grouped together questions into "composites" (e.g.,  Preparing for Equity and Diversity in Education) for a more reliable  interpretation. The reliability for the composite scores for the system  and the individual campuses generally range from 0 to 2 percentage points at the 90% confidence level. 
Evidence (or plans) regarding fairness/trustworthiness Data were not constructed with bias, and data show positive predictive  value (statistical parity) among groups and support equalized odds.
The existence of this CSU-wide service allows each campus to track  the effects of program changes designed to improve performance.  Because the instrument was designed and is implemented systemwide  with completers throughout the state, we believe it is a fair and  trustworthy measure. 
Fresno State has initiated a college-wide data summit to consider the  findings of this statewide survey and triangulate them with campus  data, including the percentage of First Generation students, access to  resources like scholarships, and culture and context of the cohorts in  which prospective teachers are placed. Through this triangulation  process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to  inform the impact on program changes on our own students with  respect to the unique diversity of culture and needs in the Central  Valley. 

CCTC Program Completer Survey

Quantitative Data Measure:  CCTC Program Completer Survey
Description of Measure Beginning in 2018, the CTC has administered a survey that is completed by completers of all credential programs. The survey, administered between September 1 and December 31, examines the effectiveness of individual educator preparation programs approved to operate in California. In 2019-2020, 97% of  the 4717 individuals who completed a Single Subject Credential program in California responded to the survey. Of those who responded, 37.7% completed their credential at a California State University.
Evidence (or plans) regarding validity Used throughout the state, the survey serves as a valid measure of completers'  perceptions of how well the teacher preparation program prepared them. Items included on the survey are directly aligned  with the California Teacher Performance Expectations and California  Standards for the Teaching Profession.  
Additionally, the survey’s content is tailored to the type of program  each respondent completed, making the content valid for each  individual. For example, the survey for a Single Subject English  teachers contains an item about how well the program prepared them to develop students' understanding and use of academic language and  vocabulary whereas the survey for a Single Subject Social Science teacher contains an item about how well the program prepared them to develop students' Historical Interpretation skills. Similarly, surveys sent  to teachers with Multiple Subjects credentials or Educational Specialist credentials respond to items directly aligned to standards associated with their credentials. 
All completers respond to items asking about their preparation of general pedagogical skills, such as their perception of how well the program prepared them to differentiate instruction in the classroom. In this way, the survey is a valid measure of completers’ perceptions of the program.
Evidence (or plans) regarding reliability Uncertainty about evaluation findings comes from two principal  sources: the number of evaluation participants and the extent of their  concurrence with each other. The evaluation findings become increasingly certain to the extent that the questions are answered by  increasing numbers of program completers and their employment  supervisors.  
Each year the data set yields the percent of respondents who gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents and  the concurrence or homogeneity of responses. The CSU Deans of  Education grouped together questions into "composites" (e.g., Preparing for Equity and Diversity in Education) for a more reliable  interpretation. The reliability for the composite scores for the system and the individual campuses generally range from 0 to 2 percentage points at the 90% confidence level. 
Evidence (or plans) regarding fairness Data were not constructed with bias, and data show positive predictive value (statistical parity) among groups and support equalized odds.
The existence of this statewide service allows each program to track  the effects of program changes designed to improve performance.  Because the instrument was designed and is implemented with completers throughout the state, we believe it is a fair and  trustworthy measure. 
Fresno State has initiated a college-wide data summit to consider the findings of this statewide survey and triangulate them with campus data, including the percentage of First Generation students, access to resources like scholarships, and culture and context of the cohorts in  which prospective teachers are placed. Through this triangulation process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to  inform the impact on program changes on our own students with respect to the unique diversity of culture and needs in the Central  Valley. 

Pre and Post Dispositions Survey 

Quantitative Data Measure: Pre and Post Dispositions Survey 
Description of Measure Complementing the 23 CSU Completer Exit Survey of graduates, the Teacher  Candidate Commitment is an instrument administered at the beginning and end of each fieldwork course (EHD 155A and EHD 155B). The California Commission on Teacher Credentialing (CCTC) requires all candidates to  demonstrate personality and character traits that satisfy the standards of the teaching profession through a 9-item measure of these traits. Thus, Fresno State developed this commitment  statement that students complete as part of their entrance requirements and when they complete  their credential program. 
Evidence (or plans) regarding validity Items included within the commitment statement align with the  dispositions CTC requires of candidates credentialed to teach within the  state. It should be noted that commitment statements rely upon self reported dispositions, which can be inaccurately represented.
Evidence (or plans) regarding reliability Reliability of the candidate’s commitment is dependent upon self reported dispositions that can be inaccurately represented.
Evidence (or plans) regarding fairness The commitment statement is intended to reinforce the values of fairness  among the candidates as well as an expectation of non-biased  dispositions of the candidates toward students of all backgrounds,  languages, cultures, and experiences. In the instance when incoming  candidates or exiting graduates perform aberrantly on this commitment  statement, they are identified, counseled, and advised about their pursuit  of the profession. In the rare instance of candidates not agreeing with the  necessary commitments, they may resubmit. 

Midterm and Final Fieldwork Evaluations

Quantitative Data Measure: Midterm and Final Fieldwork Evaluations
Description of Measure Description: The Mid-term and Final Evaluation rubrics are locally developed observation tools that provide a common language for preservice teachers, coaches, and mentors to orient their feedback in an actionable manner. Each rubric is aligned to the CCTC adopted Teacher Performance Expectations (TPEs). TPEs are divided amongst four different rubrics: the EHD 155A Midterm Evaluation, the EHD 155A Final Evaluation, the EHD 155B Midterm Evaluation, and the EHD 155B Final Evaluation. Coaches and Mentor Teachers work together to evaluate the teacher candidate’s performance. Teacher candidates are rated on a 4 point likert scale: 1=Does Not Meet Expectations 2= Meets Expectations 3=Meets Expectations at a High Level 4=Exceeds Expectations. 
In addition to assessments related to the TPEs, in all the evaluations, the teacher candidate is assessed in 6 professional competencies related to professional behaviors at the school site. The coach and the mentor teacher have the opportunity to give qualitative feedback in written form, and the teacher candidate has the opportunity to respond. A last step is for the University Coach to assess whether the teacher candidate should continue in the program (at the midterm)/complete the program or not (final). 
Evidence (or plans) regarding validity Rubric development began with close examination of the TPEs to ensure that the rubric would measure the skills required of program completion. By aligning the evaluation rubrics directly with the TPEs, our assessments reflect the standards identified by the Commission on Teacher Credentialing as essential for new teachers to possess. Content validity of the measures used to evaluate the success of the Single Subject Credential Program candidates is established through connections to the California Teaching Performance Expectations and supported by faculty-developed coursework rubrics that address similar if not the same content. Content validity is also established through the FAST, a state adopted teacher certification exam developed by Fresno State, and approved by the State of California.  Both the in-house measures and the stated adopted measure ensure that our candidates have had sufficient content knowledge to be effective in the classroom once they earn their preliminary teaching credential.
Evidence (or plans) regarding reliability Throughout California, the TPEs are the standard measurement for teacher candidates. Within Fresno State, all university coaches who supervise student teaching participate in an orientation session and regular meetings with the Coordinator of the Single Subject Program. Reliability is established through the solicitation of program judgments from its program graduates, P-12 schools (cooperating teachers, school administrators, mentor teachers, and induction programs). 
Evidence (or plans) regarding fairness This observation rubric focuses on the TPEs to help observers and teacher candidates adopt principles of good teaching. We believe that this simple but comprehensive tool will better serve the needs of our teacher candidates, creating opportunities for specific feedback that will be more easily digested and internalized. Fairness is ensured through the inclusion and equitable treatment of all individuals through the equal allocation of time, resources, and materials to all those involved. 
Evidence regarding Trustworthiness The TPEs are all areas essential to high-quality instruction. The language used within the rubric is clear and direct and provides effective feedback for teacher candidates. Trustworthiness is established through coursework audit trails. These audit trails highlight every step of data analysis that is made in order to provide a rationale for the programmatic decisions made by the coursework faculty. There are also occasions when the coursework faculty ask another course-like faculty member to perform an inquiry audit in order to ensure that the qualitative findings are consistent and could be repeated. 

Course Grades/Candidate Performance

Quantitative Data Measure: Course grades/Candidate Performance in Courses (CI 152 Learning Theories Application; CI 152 Learning Theories Description; SpEd 158 Universal Design for Learning Assignment; LEE 156 Discussions)
Description of Measure Credential candidates must maintain a 3.00 GPA in all credential courses with no  individual grade lower than a “C”. Any grade listed as “I”, “IC”, “WU”, “NC”, “D”, or “F”  does not meet Fresno State’s credential program requirements. 
Evidence (or plans) regarding validity The content of all key assignments aligns with course foci and with CCTC program standards.
Faculty hired to  teach within the programs are considered experts in their specific field and all have relevant experience for the content of the courses which  they are teaching. 
Course grades of ‘C’ or ‘Credit’ are required for program completion. Therefore, with few exceptions, all candidates must complete and receive a ‘C’ grade or better for the articulated courses.
Evidence (or plans) regarding reliability Requirements in courses stay relatively consistent over time since they are aligned with the CCTC program standards. In addition, the courses are staffed by the same faculty, on the whole. 
Evidence (or plans) regarding fairness Course grades are required at the end of the course. With few exceptions, all program completers must complete and receive a grade for courses taken. For each course, specific details are provided within course syllabi about requirements for course assignments and to earn a passing grade. 
To ensure fairness, program faculty will analyze assignment instructions to make sure expectations are clear. Where discrepancies exist between what we intend candidates to do and what they understand, we will revise instructions to ensure they are clear to all. We will also be sure that clear details are provided about how the assignment will be assessed, including specific rubrics and, whenever possible, samples of previous students' successful work.

Appendix E: