AAQEP Accreditation 2022
Appendix E: Evidence of Data Quality
Single Subject Credential Program
| Quantitative Data Measure: Fresno Assessment of Student Teachers II (FAST II) | |
| Description of Measure | FAST II consists of two projects: the Site Visitation Project (SVP) is completed during initial student teaching (EHD 178) and the Teaching Sample Project (TSP) is completed during final student teaching (EHD 170). The SVP assesses teacher candidates’ ability to plan, implement, and evaluate instruction. The three parts of the project include (1) Planning: planning documentation for a single lesson incorporating state-adopted content standards and English language development, (2) Implementation: an in-person observation and videotaping of the teaching of the lesson, (3) Reflection: a review of the entire video, selection of a 3- to 5-minute video segment, and a written evaluation of the lesson. (TPE 1.1, 1.3, 1.5, 1.8, 2.2, 2.6, 3.1, 3.2, 3.3, 3.5, 4.1, 4.2, 4.7, 6.1). The Teaching Sample Project assesses teacher candidates’ ability to (a) identify the context of the classroom, (b) plan and teach a series of at least five cohesive lessons with a focus on content knowledge and literacy, (c) assess students’ learning related to the unit, (d) document their teaching and their students’ learning, and (e) reflect on the effectiveness of their teaching. Teacher candidates document how they are addressing the needs of all their students in the planning, teaching, and assessing of the content. (TPE 1.5, 1.6, 1.8, 2.1, 2.3, 2.6, 3.1, 3.2, 3.3, 4.1, 4.3, 4.4, 4.7, 5.1, 5.2, 5.5, 5.8, 6.1, 6.3, 6.5). |
| Evidence (or plans) regarding validity | The SVP assesses the candidate’s ability to plan, implement and reflect upon instruction. Each of these abilities is assessed with performance tasks: the lesson plan (planning), teaching the lesson (implementation) and self-evaluation of the lesson (reflect upon instruction). In order to assess the teaching performance expectations (TPE) the tasks each have a rubric which share the same categories: subject specific pedagogy, applying knowledge of students and student engagement. The categories are rated on a 4-point scale (1-does not meet expectations, 2-meets expectations, 3-meets expectations at a high level, 4-exceeds expectations). The wording in the rubrics is adapted to each of the three specific tasks. Data from the FAST indicate that students are developing the competencies that are essential to effective classroom teaching practice. |
| Evidence (or plans) regarding reliability | Every 2 years, a psychometric analysis of the Site Visitation Project (SVP) is performed. Our most recent analysis found that, of the 15% of the SVPs that were double scored, 70% gave the same score and 100% were within +/-1. 94.7% agreed on the determination of whether the SVP should pass or not. |
| Evidence (or plans) regarding fairness | To monitor equity, the three subtests and the final score were examined as part of our psychometric analysis in regards to differences based on students’ ethnicity, gender, whether the student first language was English, the students’ self-rated degree of English language fluency on a 5-point Likert scale, and self-reported disability. In an effort to examine scoring equity, a series of non-parametric statistical tests were calculated to determine whether significant differences in scoring corresponded to students’ demographic characteristics. When examining the three subtests only one comparison showed statistically significant differences, the self-rated degree of English language fluency in the observation task. The statistical analyses for disability were not conducted, because of a very small sample size of 2 students self-reporting a disability. The scores were tabulated and inspected, all scores were passing. |
| Evidence regarding Trustworthiness | Developed over a number of years with the support of the Renaissance Group and a Title II grant, the FAST addresses each of California’s TPEs. Each assessment is scored by at least two faculty members, including the university coach assigned to mentor the teacher candidate. Mandatory calibration sessions are held annually, and all scorers must participate in the norming process each year. The inter-rater reliability is higher than the norm for such assessments. Moreover, students who fail the assessment have the opportunity to revise and resubmit. |
| Quantitative Data Measure: CSU Educator Quality Center Completer Survey | |
| Description of Measure | The California State University’s Education Quality Center (EdQ) oversees the administration of a completer-survey to exiting candidates of all CSU teacher-preparation programs. The survey is available year-round and campuses are encouraged to make completion of the survey a component of graduates’ final paperwork. The survey contains items asking about candidates’ perceptions of various aspects of the preparation program and the field placement experience. Campuses have access to annual results from the survey by utilizing the EdQ Dashboard. Results can be disaggregated by various measures including campus, year of completion, respondent race/ethnicity, and type of credential. |
| Evidence (or plans) regarding validity | Used systemwide, the survey serves as a valid measure of program completers’ perceptions of the teacher preparation program because it asks questions directly aligned with the California Teacher Performance Expectations and California Standards for the Teaching Profession. |
| Evidence (or plans) regarding reliability | Uncertainty about evaluation findings comes from two principal sources: the number
of evaluation participants and the extent of their concurrence with each other. The
evaluation findings become increasingly certain to the extent that the questions
are answered by increasing numbers of program completers and their employment supervisors.
Each year the data set yields the percent of respondents who gave specified answers
to each item and includes reliability estimates in the form of confidence intervals
based on the number of respondents and the concurrence or homogeneity of responses. The CSU Deans of Education grouped together questions into "composites" (e.g., Preparing for Equity and Diversity in Education) for a more reliable interpretation. The reliability for the composite scores for the system and the individual campuses generally range from 0 to 2 percentage points at the 90% confidence level. |
| Evidence (or plans) regarding fairness/trustworthiness | The existence of this CSU-wide service allows each campus to track the effects of
program changes designed to improve performance. Because the instrument was designed
and is implemented systemwide with graduates throughout the state, we believe it
is a fair and trustworthy measure. Fresno State has initiated a college-wide data summit to consider the findings of this statewide survey and triangulate them with campus data, including the percentage of First Generation students, access to resources like scholarships, and culture and context of the cohorts in which prospective teachers are placed. Through this triangulation process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to inform the impact on program changes on our own students with respect to the unique diversity of culture and needs in the Central Valley. |
| Quantitative Data Measure: CSU Educator Quality Center Employer Survey | |
| Description of Measure | Until 2018, the CSU distributed an employer survey to employers of recent graduates
of CSU teacher preparation programs. Like the CSU completer survey, the employer survey
items were tailored to the type of preparation program from the new teacher completed
(Multiple Subject, Single Subject-Math, Single Subject-English, Education Specialist,
etc.). Survey items target the following areas: Engaging and Supporting all Students in Learning, Creating and Maintaining Effective Environments for Student Learning, Understanding and Organizing Subject Matter for Student Learning, Planning Instruction and Designing Learning Experiences for All Students, Assessing Students for Learning, Developing as a Professional Educator, and an overall assessment of how well prepared graduates of the institution are to be teachers. The results are disaggregated by campus and allow campuses to evaluate their effectiveness from the perspective of employers of completers. |
| Evidence (or plans) regarding validity | Used systemwide, the survey serves as a measure of employers' perceptions of how well
programs prepared their completers for their first-year of teaching. Items on the
survey are aligned with the California Teacher Performance Expectations and California
Standards for the Teaching Profession. All employers respond to items asking about their preparation of the new teachers’ general pedagogical skills, such as their perception of how well the program prepared them to differentiate instruction in the classroom. In this way, the survey is a valid measure of employers’ perceptions of the program. |
| Evidence (or plans) regarding reliability | Uncertainty about evaluation findings comes from two principal sources: the number
of evaluation participants and the extent of their concurrence with each other. The
evaluation findings become increasingly certain to the extent that the questions
are answered by increasing numbers of program completers and their employment supervisors.
Each year the data set yields the percent of respondents who gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents and the concurrence or homogeneity of responses. The CSU Deans of Education grouped together questions into "composites" (e.g., Preparing for Equity and Diversity in Education) for a more reliable interpretation. The reliability for the composite scores for the system and the individual campuses generally range from 0 to 2 percentage points at the 90% confidence level. |
| Evidence (or plans) regarding fairness/trustworthiness | Data were not constructed with bias, and data show positive predictive value (statistical
parity) among groups and support equalized odds. The existence of this CSU-wide service allows each campus to track the effects of program changes designed to improve performance. Because the instrument was designed and is implemented systemwide with completers throughout the state, we believe it is a fair and trustworthy measure. Fresno State has initiated a college-wide data summit to consider the findings of this statewide survey and triangulate them with campus data, including the percentage of First Generation students, access to resources like scholarships, and culture and context of the cohorts in which prospective teachers are placed. Through this triangulation process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to inform the impact on program changes on our own students with respect to the unique diversity of culture and needs in the Central Valley. |
| Quantitative Data Measure: CSU Educator Quality Center Year One Completer Survey | |
| Description of Measure | The California State University’s Education Quality Center (EdQ) oversees the administration
of a survey of all individuals who completed a CSU teacher-preparation programs after
their first year on the job. The survey is administered annually April through July.
In April, the EdQ Center emails an initial survey invitation to all completers of
MS-SSES Credential Programs serving as first-year teachers in public schools, charter
schools, or private schools in all locations. Follow-up reminders are sent every
two weeks throughout the duration of the survey window. In addition to asking questions about the completer’s demographics and educational background, the survey also contains items to capture data about the school where the completer is employed. Additionally, the survey includes items asking about candidates’ perceptions of various aspects of the preparation program and the field placement experience. Campuses have access to annual results from the survey by utilizing the EdQ Dashboard. Results can be disaggregated by various measures including campus, year of completion, respondent race/ethnicity, and type of credential. Note: the CTC also distributes a Credential Program Completer Survey which gives an overall view of CA Educator Preparation Programs. |
| Evidence (or plans) regarding validity | Used systemwide, the survey serves as a valid measure of graduates' perceptions of
how well the teacher preparation program prepared them for their first-year of teaching
because it asks questions directly aligned with the California Teacher Performance
Expectations and California Standards for the Teaching Profession. Additionally, the survey’s content is tailored to the type of program each respondent completed, making the content valid for each individual. For example, the survey for a Single Subject English teachers contains an item about how well the program prepared them to develop students' understanding and use of academic language and vocabulary whereas the survey for a Single Subject Social Science teacher contains an item about how well the program prepared them to develop students' Historical Interpretation skills. Similarly, surveys sent to teachers with Multiple Subjects credentials or Educational Specialist credentials respond to items directly aligned to standards associated with their credentials. All graduates respond to items asking about their preparation of general pedagogical skills, such as their perception of how well the program prepared them to differentiate instruction in the classroom. In this way, the survey is a valid measure of completers’ perceptions of the program. |
| Evidence (or plans) regarding reliability | Uncertainty about evaluation findings comes from two principal sources: the number
of evaluation participants and the extent of their concurrence with each other. The
evaluation findings become increasingly certain to the extent that the questions
are answered by increasing numbers of program completers and their employment supervisors.
Each year the data set yields the percent of respondents who gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents and the concurrence or homogeneity of responses. The CSU Deans of Education grouped together questions into "composites" (e.g., Preparing for Equity and Diversity in Education) for a more reliable interpretation. The reliability for the composite scores for the system and the individual campuses generally range from 0 to 2 percentage points at the 90% confidence level. |
| Evidence (or plans) regarding fairness/trustworthiness | Data were not constructed with bias, and data show positive predictive value (statistical
parity) among groups and support equalized odds. The existence of this CSU-wide service allows each campus to track the effects of program changes designed to improve performance. Because the instrument was designed and is implemented systemwide with completers throughout the state, we believe it is a fair and trustworthy measure. Fresno State has initiated a college-wide data summit to consider the findings of this statewide survey and triangulate them with campus data, including the percentage of First Generation students, access to resources like scholarships, and culture and context of the cohorts in which prospective teachers are placed. Through this triangulation process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to inform the impact on program changes on our own students with respect to the unique diversity of culture and needs in the Central Valley. |
| Quantitative Data Measure: CCTC Program Completer Survey | |
| Description of Measure | Beginning in 2018, the CTC has administered a survey that is completed by completers of all credential programs. The survey, administered between September 1 and December 31, examines the effectiveness of individual educator preparation programs approved to operate in California. In 2019-2020, 97% of the 4717 individuals who completed a Single Subject Credential program in California responded to the survey. Of those who responded, 37.7% completed their credential at a California State University. |
| Evidence (or plans) regarding validity | Used throughout the state, the survey serves as a valid measure of completers' perceptions
of how well the teacher preparation program prepared them. Items included on the survey
are directly aligned with the California Teacher Performance Expectations and California
Standards for the Teaching Profession. Additionally, the survey’s content is tailored to the type of program each respondent completed, making the content valid for each individual. For example, the survey for a Single Subject English teachers contains an item about how well the program prepared them to develop students' understanding and use of academic language and vocabulary whereas the survey for a Single Subject Social Science teacher contains an item about how well the program prepared them to develop students' Historical Interpretation skills. Similarly, surveys sent to teachers with Multiple Subjects credentials or Educational Specialist credentials respond to items directly aligned to standards associated with their credentials. All completers respond to items asking about their preparation of general pedagogical skills, such as their perception of how well the program prepared them to differentiate instruction in the classroom. In this way, the survey is a valid measure of completers’ perceptions of the program. |
| Evidence (or plans) regarding reliability | Uncertainty about evaluation findings comes from two principal sources: the number
of evaluation participants and the extent of their concurrence with each other. The
evaluation findings become increasingly certain to the extent that the questions are
answered by increasing numbers of program completers and their employment supervisors.
Each year the data set yields the percent of respondents who gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents and the concurrence or homogeneity of responses. The CSU Deans of Education grouped together questions into "composites" (e.g., Preparing for Equity and Diversity in Education) for a more reliable interpretation. The reliability for the composite scores for the system and the individual campuses generally range from 0 to 2 percentage points at the 90% confidence level. |
| Evidence (or plans) regarding fairness | Data were not constructed with bias, and data show positive predictive value (statistical
parity) among groups and support equalized odds. The existence of this statewide service allows each program to track the effects of program changes designed to improve performance. Because the instrument was designed and is implemented with completers throughout the state, we believe it is a fair and trustworthy measure. Fresno State has initiated a college-wide data summit to consider the findings of this statewide survey and triangulate them with campus data, including the percentage of First Generation students, access to resources like scholarships, and culture and context of the cohorts in which prospective teachers are placed. Through this triangulation process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to inform the impact on program changes on our own students with respect to the unique diversity of culture and needs in the Central Valley. |
| Quantitative Data Measure: Pre and Post Dispositions Survey | |
| Description of Measure | Complementing the 23 CSU Completer Exit Survey of graduates, the Teacher Candidate Commitment is an instrument administered at the beginning and end of each fieldwork course (EHD 155A and EHD 155B). The California Commission on Teacher Credentialing (CCTC) requires all candidates to demonstrate personality and character traits that satisfy the standards of the teaching profession through a 9-item measure of these traits. Thus, Fresno State developed this commitment statement that students complete as part of their entrance requirements and when they complete their credential program. |
| Evidence (or plans) regarding validity | Items included within the commitment statement align with the dispositions CTC requires of candidates credentialed to teach within the state. It should be noted that commitment statements rely upon self reported dispositions, which can be inaccurately represented. |
| Evidence (or plans) regarding reliability | Reliability of the candidate’s commitment is dependent upon self reported dispositions that can be inaccurately represented. |
| Evidence (or plans) regarding fairness | The commitment statement is intended to reinforce the values of fairness among the candidates as well as an expectation of non-biased dispositions of the candidates toward students of all backgrounds, languages, cultures, and experiences. In the instance when incoming candidates or exiting graduates perform aberrantly on this commitment statement, they are identified, counseled, and advised about their pursuit of the profession. In the rare instance of candidates not agreeing with the necessary commitments, they may resubmit. |
| Quantitative Data Measure: Midterm and Final Fieldwork Evaluations | |
| Description of Measure | Description: The Mid-term and Final Evaluation rubrics are locally developed observation
tools that provide a common language for preservice teachers, coaches, and mentors
to orient their feedback in an actionable manner. Each rubric is aligned to the CCTC
adopted Teacher Performance Expectations (TPEs). TPEs are divided amongst four different
rubrics: the EHD 155A Midterm Evaluation, the EHD 155A Final Evaluation, the EHD 155B Midterm Evaluation, and the EHD 155B Final Evaluation. Coaches and Mentor Teachers work together to evaluate the teacher candidate’s performance.
Teacher candidates are rated on a 4 point likert scale: 1=Does Not Meet Expectations
2= Meets Expectations 3=Meets Expectations at a High Level 4=Exceeds Expectations. In addition to assessments related to the TPEs, in all the evaluations, the teacher candidate is assessed in 6 professional competencies related to professional behaviors at the school site. The coach and the mentor teacher have the opportunity to give qualitative feedback in written form, and the teacher candidate has the opportunity to respond. A last step is for the University Coach to assess whether the teacher candidate should continue in the program (at the midterm)/complete the program or not (final). |
| Evidence (or plans) regarding validity | Rubric development began with close examination of the TPEs to ensure that the rubric would measure the skills required of program completion. By aligning the evaluation rubrics directly with the TPEs, our assessments reflect the standards identified by the Commission on Teacher Credentialing as essential for new teachers to possess. Content validity of the measures used to evaluate the success of the Single Subject Credential Program candidates is established through connections to the California Teaching Performance Expectations and supported by faculty-developed coursework rubrics that address similar if not the same content. Content validity is also established through the FAST, a state adopted teacher certification exam developed by Fresno State, and approved by the State of California. Both the in-house measures and the stated adopted measure ensure that our candidates have had sufficient content knowledge to be effective in the classroom once they earn their preliminary teaching credential. |
| Evidence (or plans) regarding reliability | Throughout California, the TPEs are the standard measurement for teacher candidates. Within Fresno State, all university coaches who supervise student teaching participate in an orientation session and regular meetings with the Coordinator of the Single Subject Program. Reliability is established through the solicitation of program judgments from its program graduates, P-12 schools (cooperating teachers, school administrators, mentor teachers, and induction programs). |
| Evidence (or plans) regarding fairness | This observation rubric focuses on the TPEs to help observers and teacher candidates adopt principles of good teaching. We believe that this simple but comprehensive tool will better serve the needs of our teacher candidates, creating opportunities for specific feedback that will be more easily digested and internalized. Fairness is ensured through the inclusion and equitable treatment of all individuals through the equal allocation of time, resources, and materials to all those involved. |
| Evidence regarding Trustworthiness | The TPEs are all areas essential to high-quality instruction. The language used within the rubric is clear and direct and provides effective feedback for teacher candidates. Trustworthiness is established through coursework audit trails. These audit trails highlight every step of data analysis that is made in order to provide a rationale for the programmatic decisions made by the coursework faculty. There are also occasions when the coursework faculty ask another course-like faculty member to perform an inquiry audit in order to ensure that the qualitative findings are consistent and could be repeated. |
| Quantitative Data Measure: Course grades/Candidate Performance in Courses (CI 152 Learning Theories Application; CI 152 Learning Theories Description; SpEd 158 Universal Design for Learning Assignment; LEE 156 Discussions) | |
| Description of Measure | Credential candidates must maintain a 3.00 GPA in all credential courses with no individual grade lower than a “C”. Any grade listed as “I”, “IC”, “WU”, “NC”, “D”, or “F” does not meet Fresno State’s credential program requirements. |
| Evidence (or plans) regarding validity | The content of all key assignments aligns with course foci and with CCTC program standards. Faculty hired to teach within the programs are considered experts in their specific field and all have relevant experience for the content of the courses which they are teaching. Course grades of ‘C’ or ‘Credit’ are required for program completion. Therefore, with few exceptions, all candidates must complete and receive a ‘C’ grade or better for the articulated courses. |
| Evidence (or plans) regarding reliability | Requirements in courses stay relatively consistent over time since they are aligned with the CCTC program standards. In addition, the courses are staffed by the same faculty, on the whole. |
| Evidence (or plans) regarding fairness | Course grades are required at the end of the course. With few exceptions, all program
completers must complete and receive a grade for courses taken. For each course, specific
details are provided within course syllabi about requirements for course assignments
and to earn a passing grade. To ensure fairness, program faculty will analyze assignment instructions to make sure expectations are clear. Where discrepancies exist between what we intend candidates to do and what they understand, we will revise instructions to ensure they are clear to all. We will also be sure that clear details are provided about how the assignment will be assessed, including specific rubrics and, whenever possible, samples of previous students' successful work. |