AAQEP Accreditation 2022
Appendix E: Evidence of Data Quality
Agricultural Specialist Credential Program
| Quantitative Data Measure: Midterm and Final Fieldwork Evaluations | |
| Description of Measure | The Mid-term and Final Evaluation rubrics are locally developed observation tools
that provide a common language for preservice teachers, coaches, and mentors to orient
their feedback in an actionable manner. Each rubric is aligned to the CCTC adopted
Teacher Performance Expectations (TPEs). TPEs are divided amongst four different rubrics:
the EHD 155A Midterm Evaluation, the EHD 155A Final Evaluation, the EHD 155B Midterm Evaluation, and the EHD 155B Final Evaluation. Coaches and Mentor Teachers work together to evaluate the teacher candidate’s performance.
Teacher candidates are rated on a 4 point likert scale: 1=Does Not Meet Expectations
2= Meets Expectations 3=Meets Expectations at a High Level 4=Exceeds Expectations. In addition to assessments related to the TPEs, in all the evaluations, the teacher candidate is assessed in 6 professional competencies related to professional behaviors at the school site. The coach and the mentor teacher have the opportunity to give qualitative feedback in written form, and the teacher candidate has the opportunity to respond. A last step is for the University Coach to assess whether the teacher candidate should continue in the program (at the midterm)/complete the program or not (final). |
| Evidence (or plans) regarding validity | Rubric development began with close examination of the TPEs to ensure that the rubric would measure the skills required of program completion. By aligning the evaluation rubrics directly with the TPEs, our assessments reflect the standards identified by the Commission on Teacher Credentialing as essential for new teachers to possess. Content validity of the measures used to evaluate the success of the Single Subject Credential Program candidates is established through connections to the California Teaching Performance Expectations and supported by faculty-developed coursework rubrics that address similar if not the same content. Content validity is also established through the FAST, a state adopted teacher certification exam developed by Fresno State, and approved by the State of California. Both the in-house measures and the stated adopted measure ensure that our candidates have had sufficient content knowledge to be effective in the classroom once they earn their preliminary teaching credential. |
| Evidence (or plans) regarding reliability | Throughout California, the TPEs are the standard measurement for teacher candidates. Within Fresno State, all university coaches who supervise student teaching participate in an orientation session and regular meetings with the Coordinator of the Single Subject Program. Reliability is established through the solicitation of program judgments from its program graduates, P-12 schools (cooperating teachers, school administrators, mentor teachers, and induction programs). |
| Evidence (or plans) regarding fairness | This observation rubric focuses on the TPEs to help observers and teacher candidates adopt principles of good teaching. We believe that this simple but comprehensive tool will better serve the needs of our teacher candidates, creating opportunities for specific feedback that will be more easily digested and internalized. Fairness is ensured through the inclusion and equitable treatment of all individuals through the equal allocation of time, resources, and materials to all those involved. |
| Evidence regarding Trustworthiness | The TPEs are all areas essential to high-quality instruction. The language used within the rubric is clear and direct and provides effective feedback for teacher candidates. Trustworthiness is established through coursework audit trails. These audit trails highlight every step of data analysis that is made in order to provide a rationale for the programmatic decisions made by the coursework faculty. There are also occasions when the coursework faculty ask another course-like faculty member to perform an inquiry audit in order to ensure that the qualitative findings are consistent and could be repeated. |
| Quantitative Data Measure: CI 161 Curriculum Project Scores | |
| Description of Measure |
For a chosen agricultural course (other than Ag Core Curriculum) students develop a Course Outline, Unit Outline, and at least three consecutive lesson plans.
|
| Evidence (or plans) regarding validity | Scores are calculated based on students’ successful completion and the quality of all the required assignment components. Because the assignment is specifically linked to course content, students’ grades serve as a valid measure of mastery of this content. |
| Evidence (or plans) regarding reliability | This assignment is evaluated and scored by one instructor each semester utilizing the same scoring criteria as used in previous years. In an effort to ensure the reliability of scores, the instructor refers to these benchmark examples prior to scoring the assignment. |
| Evidence (or plans) regarding fairness | To ensure fairness, we will analyze assignment instructions to make sure expectations are clear. Where discrepancies exist between what we intend candidates to do and what they understand, we will revise instructions to ensure they are clear to all. We will also be sure that clear details are provided about how the assignment will be assessed, including specific rubrics and, whenever possible, samples of previous students' successful work. |
| Quantitative Data Measure: Occupational Experience Form (T-14) | |
| Description of Measure | On the Occupational Experience Form (T-14) each candidate is required to record their agricultural occupational experience hours. Candidates are required to have at least 3,000 hours to qualify for the Ag Specialist credential. |
| Evidence (or plans) regarding validity | Each candidate submits their completed form to the California Dept. of Education Ag. Education State Staff representative for the San Joaquin Region who maintains an office on our campus. The Regional State Staff representative reviews the Occupational Experience Forms while interviewing each candidate. The purpose of the interview is to verify the validity and accuracy of the form data. |
| Evidence (or plans) regarding reliability | Each Occupational Experience Form is reviewed and interviews are conducted by the same person who is a qualified representative from the Calif. Department of Education’s Ag. Education Division. |
| Evidence (or plans) regarding fairness | This form includes all the various segments of the agricultural industry, plus candidates are provided an “other” category to ensure that all candidates’ experiences can be recorded and contribute toward the 3,000 hr. requirement. |
| Quantitative Data Measure: CI 161 MicroTeaching Presentation Rubric | |
| Description of Measure | In CI 161, candidates complete a micro-teaching assignment in which they plan a lesson and then teach it to their peers. The rubric used to score the micro-teaching derived from the teaching evaluation rubric which was used in the past to evaluate our student teachers. This form was developed by the Kremen School of Education. The rubric includes criteria related to the lesson presentation, teacher/student interaction, classroom management and the overall quality of the presentation. |
| Evidence (or plans) regarding validity | The instrument/rubric was originally developed by faculty in the Kremen School of Education and later reviewed by faculty in Agricultural Education to ensure it serves as a valid measurement of student’s performance in their MicroTeaching. |
| Evidence (or plans) regarding reliability | A post-hoc analysis of the 2020 data was conducted documenting the reliability of the 19 item rubric. The analysis yielded a Cronbach’s alpha coefficient of .96, leading us to conclude that the instrument is highly reliable. |
| Evidence (or plans) regarding fairness | To ensure fairness, our faculty have reviewed the rubric to ensure it aligns with the course outcomes and fairly measures students’ performance. |
| Quantitative Data Measure: AGRI 281 Project Scores | |
| Description of Measure | Students complete a study on a selected problem in Ag. Ed. either during the student
teaching assignment or otherwise. The special problem should be relevant to the community
and the Ag. Education program. Students define the problem, delimit the scope of the problem and prepare the material, in such a manner that it will be acceptable for publication, partial fulfillment of graduate credit for master's program and/or continued study on a thesis. |
| Evidence (or plans) regarding validity | Scores are calculated based on students’ successful completion and the quality of all the required assignment components. Because the assignment is specifically linked to course content, students’ grades serve as a valid measure of mastery of this content. |
| Evidence (or plans) regarding reliability | This assignment is evaluated and scored by the instructor each semester utilizing the same scoring criteria as used in previous years. In an effort to ensure the reliability of scores, the instructor refers to these benchmark examples prior to scoring the assignment. |
| Evidence (or plans) regarding fairness | To ensure fairness, we will analyze assignment instructions to make sure expectations are clear. Where discrepancies exist between what we intend candidates to do and what they understand, we will revise instructions to ensure they are clear to all. We will also be sure that clear details are provided about how the assignment will be assessed, including specific rubrics and, whenever possible, samples of previous students' successful work. |
| Quantitative Data Measure: Program Completer Follow Up Survey | |
| Description of Measure | Every five years a survey is conducted of program completers from the previous five cohorts to evaluate their perceptions of the program, specifically their level of technical and professional preparation. |
| Evidence (or plans) regarding validity | The instrument is aligned with the CCTC Subject Matter and Professional standards areas. Prior to administering the instrument, it is reviewed by a panel of experts (Ag. Education faculty) to ensure face and content validity. |
| Evidence (or plans) regarding reliability | A post-hoc analysis of the 2021 data was conducted documenting the reliability of the 17 item technical and professional knowledge/skill scale which is the primary measurement tool. The analysis yielded a Cronbach’s alpha coefficient of .88, leading us to conclude that the instrument is highly reliable. |
| Evidence (or plans) regarding fairness | The instrument is aligned with the CCTC Subject Matter and Professional standards areas. Prior to administering the instrument, it is reviewed by a panel of experts (Ag. Education faculty) to ensure fairness and inclusivity of all participants. |
| Quantitative Data Measure: Employer Follow Up Survey | |
| Description of Measure | Every five years a survey is administered to the school site administrators of completers from our previous five cohorts to evaluate the administrator’s perceptions of the completer’s level of technical and professional preparation. |
| Evidence (or plans) regarding validity | The instrument is aligned with the CCTC Subject Matter and Professional standards areas. Prior to administering the instrument, it is reviewed by a panel of experts (Ag. Education faculty) to ensure face and content validity. |
| Evidence (or plans) regarding reliability | A post-hoc analysis of the 2021 data was conducted documenting the reliability of the 15 item technical and professional preparation scale which is the primary measurement tool. The analysis yielded a Cronbach’s alpha coefficient of .96, leading us to conclude that the instrument is highly reliable. |
| Evidence (or plans) regarding fairness | The instrument is aligned with the CCTC Subject Matter and Professional standards areas. Prior to administering the instrument, it is reviewed by a panel of experts (Ag. Education faculty) to ensure fairness and inclusivity of all participants. |
| Quantitative Data Measure: California Agricultural Teachers’ Induction Program (CATIP) Individual Induction Plan (IDP) Self-Assessment | |
| Description of Measure | The California Agricultural Teachers’ Induction Program (CATIP) Individual Induction Plan (IDP) Self-Assessment is administered to new teachers beginning the CATIP Induction program. The assessment serves as a baseline measurement for new teachers to identify their strengths and develop a plan for improvement in weak areas. |
| Evidence (or plans) regarding validity | The instrument is aligned with the technical and professional standards areas in agricultural education. Prior to administering the instrument, it is reviewed by a panel of experts (Ag. Education faculty) to ensure face and content validity. |
| Evidence (or plans) regarding reliability | A post-hoc analysis of the 2018-2020 data was conducted documenting the reliability of the 18 item technical and professional knowledge/skill scale which is the primary measurement tool. The analysis yielded a Cronbach’s alpha coefficient of .92, leading us to conclude that the instrument is highly reliable. |
| Evidence (or plans) regarding fairness | This instrument has been reviewed by a panel of experts representing a variety of agricultural education programs/institutions to ensure that it serves as a valid and fair assessment. |
| Quantitative Data Measure: EHD 155A Professional Competencies & EHD 155B Exit Evaluation of Professional Objectives | |
| Description of Measure | The EHD 155A Professional Competencies & EHD 155B Exit Evaluation of Professional Objectives contain professional competencies required of candidates for the Agriculture Specialist Credential. As each objective is accomplished, the approximate date of accomplishment should be filled in and initialed for verification by someone in a position to evaluate the achievement of that objective. The only people who may verify the accomplishment of these objectives are California State University,Fresno faculty, cooperating master teachers and the administrators of the cooperating schools. |
| Evidence (or plans) regarding validity | The instrument is aligned with the CCTC Ag Specialist Professional standards areas. Prior to administering the instrument, it is reviewed by a panel of experts (Ag. Education faculty and advisory committee members) to ensure face and content validity. |
| Evidence (or plans) regarding reliability | This measurement is a calculation on each of the students completed objectives, making it a completely objective scoring thus removing nearly all threat of error in scoring and making it a highly reliable measurement. |
| Evidence (or plans) regarding fairness | The instrument is aligned with the CCTC Ag Specialist Professional standards areas. Prior to administering the instrument, it is reviewed by a panel of experts (Ag. Education faculty and advisory committee members) to ensure it is a valid and fair measurement. |
| Quantitative Data Measure: Fresno Assessment of Student Teachers II (FAST II) | |
| Description of Measure | FAST II consists of two projects: the Site Visitation Project (SVP) is completed during initial student teaching (EHD 155A) and the Teaching Sample Project (TSP) is completed during final student teaching (EHD 155B). The SVP assesses teacher candidates’ ability to plan, implement, and evaluate instruction. The three parts of the project include (1) Planning: planning documentation for a single lesson incorporating state-adopted content standards and English language development, (2) Implementation: an in-person observation and videotaping of the teaching of the lesson, (3) Reflection: a review of the entire video, selection of a 3- to 5-minute video segment, and a written evaluation of the lesson. (TPE 1.1, 1.3, 1.5, 1.8, 2.2, 2.6, 3.1, 3.2, 3.3, 3.5, 4.1, 4.2, 4.7, 6.1). The Teaching Sample Project assesses teacher candidates’ ability to (a) identify the context of the classroom, (b) plan and teach a series of at least five cohesive lessons with a focus on content knowledge and literacy, (c) assess students’ learning related to the unit, (d) document their teaching and their students’ learning, and (e) reflect on the effectiveness of their teaching. Teacher candidates document how they are addressing the needs of all their students in the planning, teaching, and assessing of the content. (TPE 1.5, 1.6, 1.8, 2.1, 2.3, 2.6, 3.1, 3.2, 3.3, 4.1, 4.3, 4.4, 4.7, 5.1, 5.2, 5.5, 5.8, 6.1, 6.3, 6.5). |
| Evidence (or plans) regarding validity | The SVP assesses the candidate’s ability to plan, implement and reflect upon instruction. Each of these abilities is assessed with performance tasks: the lesson plan (planning), teaching the lesson (implementation) and self-evaluation of the lesson (reflect upon instruction). In order to assess the teaching performance expectations (TPE) the tasks each have a rubric which share the same categories: subject specific pedagogy, applying knowledge of students and student engagement. The categories are rated on a 4-point scale (1-does not meet expectations, 2-meets expectations, 3-meets expectations at a high level, 4-exceeds expectations). The wording in the rubrics is adapted to each of the three specific tasks. Data from the FAST indicate that students are developing the competencies that are essential to effective classroom teaching practice. |
| Evidence (or plans) regarding reliability | Every 2 years, a psychometric analysis of the Site Visitation Project (SVP) is performed. Our most recent analysis found that of the 15% of the SVPs that were double scored, 70% gave the same score and 100% were within +/-1. 94.7% agreed on the determination of whether the SVP should pass or not. |
| Evidence (or plans) regarding fairness | To monitor equity, the three subtests and the final score were examined as part of our psychometric analysis in regards to differences based on students’ ethnicity, gender, whether the student first language was English, the students’ self-rated degree of English language fluency on a 5-point Likert scale, and self-reported disability. In an effort to examine scoring equity, a series of non-parametric statistical tests were calculated to determine whether significant differences in scoring corresponded to students’ demographic characteristics. When examining the three subtests only one comparison showed statistically significant differences, the self-rated degree of English language fluency in the observation task. The statistical analyses for disability were not conducted, because of a very small sample size of 2 students self-reporting a disability. The scores were tabulated and inspected, all scores were passing. |
| Evidence regarding Trustworthiness | Developed over a number of years with the support of the Renaissance Group and a Title II grant, the FAST addresses each of California’s TPEs. Each assessment is scored by at least two faculty members, including the university coach assigned to mentor the teacher candidate. Mandatory calibration sessions are held annually, and all scorers must participate in the norming process each year. The inter-rater reliability is higher than the norm for such assessments. Moreover, students who fail the assessment have the opportunity to revise and resubmit. |
| Quantitative Data Measure: Pre and Post Dispositions Survey | |
| Description of Measure | Complementing the 23 CSU Completer Exit Survey of graduates, the Teacher Candidate Commitment is an instrument administered at the beginning and end of each fieldwork course (EHD 155A and EHD 155B). The California Commission on Teacher Credentialing (CCTC) requires all candidates to demonstrate personality and character traits that satisfy the standards of the teaching profession through a 9-item measure of these traits. Thus, Fresno State developed this commitment statement that students complete as part of their entrance requirements and when they complete their credential program. |
| Evidence (or plans) regarding validity | Items included within the commitment statement align with the dispositions CTC requires of candidates credentialed to teach within the state. It should be noted that commitment statements rely upon self reported dispositions, which can be inaccurately represented. |
| Evidence (or plans) regarding reliability | Reliability of the candidate’s commitment is dependent upon self reported dispositions that can be inaccurately represented. |
| Evidence (or plans) regarding fairness | The commitment statement is intended to reinforce the values of fairness among the candidates as well as an expectation of non-biased dispositions of the candidates toward students of all backgrounds, languages, cultures, and experiences. In the instance when incoming candidates or exiting graduates perform aberrantly on this commitment statement, they are identified, counseled, and advised about their pursuit of the profession. In the rare instance of candidates not agreeing with the necessary commitments, they may resubmit. |
| Quantitative Data Measure: Multicultural/International & Plant 105 Course grades | |
| Description of Measure | Candidates for the Ag. Specialist credential must fulfill the Subject Matter Competency Requirement for program entry, which requires either the state’s CSET exams or the completion of our Subject Matter Waiver option. The Waiver program consists of fulfilling our Ag. Education bachelor’s degree requirements which includes a Multicultural/International course and also General Education Integration area B. Area IB is typically fulfilled by students completing the only agricultural course in that area - Plant 105. |
| Evidence (or plans) regarding validity | Faculty hired to teach the Plant 105 and M/I courses are considered experts in their specific field and have relevant educational training and experience for the content of the courses which they are teaching. Their expertise in the field demonstrates their ability to develop valid assessments to calculate the students’ overall grades. Course assignments and expectations are aligned with the essential course content. |
| Evidence (or plans) regarding reliability | Requirements in courses stay relatively consistent over time since they are aligned with our university’s general education and graduation requirements. |
| Evidence (or plans) regarding fairness | Course grades are required at the end of the course. With few exceptions, all program completers must complete and receive a grade for courses taken. For each course, specific details are provided within course syllabi about requirements for course assignments and to earn a passing grade. |