Appendix E: Evidence of Data Quality

Candidates in each administrative services credential program in California must demonstrate content and skill knowledge on three performance assessments prior to the attainment of a preliminary credential.  Each performance assessment or leadership cycle contains required knowledge and skills found in the California Administrator Performance Expectations (CAPEs).  The CalAPA consists of three leadership cycles that focus candidates on specific content knowledge, a different set of skills, and knowledge to demonstrate new learning relevant to school administration.  Each cycle includes professional dispositions that school administrators should know and be able to demonstrate.  There are 4 steps in each CalAPA performance assessment which include the following actions:  Investigate, Plan, Act, Reflect in a continuous improvement model based on collected data from fieldwork school sites.
Consequential implementation of the CalAPA began in Fall 2019.  Based on this new assessment the Fresno State PASC program has gone through a series of redesign processes to better align student expectations and outcomes to the state standards and assessment.   Each leadership cycle includes either 7 or 8 analytic rubrics that measures knowledge and skills candidates must evidence in narrative writing, document artifacts, and/or video. Each rubric is aligned to specific CAPEs organized by a scale from Level 1 to Level 5.  Based on the overall passage score for each cycle, candidate success or proficiency would be measured at Level 2.  As of 2019, each CalAPA leadership cycle has a summative passing score as follows:

• Cycle 1 (8 rubrics): A final cut score of 14 points.
• Cycle 2 (7 rubrics): A final cut score of 12 points.
• Cycle 3 (7 rubrics): A final cut score of 12 points.

Because this instrument is administered statewide to candidates of all programs (CSU, University of California, private institutions, and local education agencies), we believe in its fairness as an instrument. Additionally, we believe the data the instrument yields are trustworthy because the survey is administered by the CTC, the statewide accrediting agency.

Fresno State has initiated a college-wide data summit to consider the findings of this statewide survey and triangulate them with campus data, including the percentage of First Generation students, access to resources like scholarships, and culture and context of the cohorts in which prospective teachers are placed. Through this triangulation process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to inform the impact on program changes on our own students with respect to the unique diversity of culture and needs in the Central Valley.  

Students were evaluated in their ability to follow instructions, write their ideas clearly, organize material, present content effectively, and engage in self-work. Students were required to incorporate research based resources to improve their understanding of the experience of marginalized groups and to learn about interventions that could improve the lived experiences of these groups. Providing students these clear directions supported with ensuring this assignment’s reliability in supporting students to engage in the specific requirements of this assignment. To analyze the effectiveness of this assignment, we examined the following dataset: COUN201Co.pdf 

Overall, students scored above satisfactory in all areas with scores higher than 12 points total (M= 13.32). It will be important to keep analyzing students’ responses to this assignment to assess its reliability overtime in expanding students’ cultural competence. 

We strive to ensure that the assignment is fair for all students by using the following rubric: Rubric COUN201

Through engaging students in this assignment, our hope was to evaluate students’ understanding of the impact of external and internal forces on marginalized groups, which will be important for strengthening all candidates’ ability to provide counseling to a diverse group of K-12 students, hence making this assignment a fair choice.

Quantitative Data Measure: Mid-Term Exam in COUN 206. Please refer to the following link for syllabus: COUN 206 Syllabus for AAQEP Standard 1b 

Faculty are able to ensure the assignment’s reliability across different semesters by providing clear and consistent directions:

a. Papers will include two parts: A school counselor interview and literature review. Students are required to critically analyze, synthesize, and evaluate research articles on the topic of their interest. Students should be able to suggest their own conclusions and implications for school counseling based on an interview with school counselor and literature review.
b. This paper is intended to be helpful for students’ professional development and will be directly related to the course content.
c. The paper will be a minimum of 5 pages of double-spaced typed text with APA style and formatting. Students must cite at least 3 articles published in referred journals.

Please refer to the following link for a sample of students-work (N = 10):

COUN 241 Sample work for AAQEP Std 1a (Google Doc)

To ensure fairness, faculty provide a rubric along with specific directions for more detailed directions and expectations. Lit Review Rubric 2020.pdf

In future, within the assignment description, we plan to add that students must use the knowledge gained from reading peer-reviewed journal articles and engaging in dialogues with professional school counselors to suggest two evidence-based practices that can promote a positive change within school settings – both at individual and systemic levels. We will also make a comprehensive rubric that focus on applicability of pedagogical content. 

Students are required to create a 20-minute lesson plan based on the ASCA model and present it to the class (in PowerPoint) at the assigned dates. Objective: Develop, present, and evaluate a classroom lesson on school counseling core curriculum, including formative and summative assessments. The assignment includes the following 4 components:

a. Summary of data/information: Students gather data from their school site to support the need for their lesson. Data collection can be any of the following—i. available data from the school, district, and/or state; ii. Observe and gather information from the school Administration/faculty/staff to see what the needs of students are; ii. Or they may collect anonymous surveys from students/parents/teachers/counselors. 
b. Complete a plan using the Lesson Plan template from ASCA and submit to instructor.
c. Create a pre- and post- assessment for the lesson plan.
d. PowerPoint presentation should focus on the lesson and pre/post test.

Assignment aligns with student learning outcomes and CTC standards 23, 26, and 29:

Standard 23: Advocacy
Candidates demonstrate skills and attitudes essential for advocating for the learning and academic success of all pupils. Candidates are able to identify institutional, systemic, interpersonal and intrapersonal barriers to learning, and are able to plan and implement strategies to eliminate those barriers and effectively support positive learning and achievement outcomes for all pupils.

Standard 26: Group Counseling and Facilitation
Candidates understand group dynamics and possess skill in group work including counseling, psychoeducational task and peer helping groups and facilitation of teams to enable pupils to overcome barriers and independence to learning.

Standard 29: Prevention, Education and Training
Candidates know and have skills in the planning, organizing and implementing educational programs designed to promote pupil learning and high academic achievement. Candidates also have knowledge in preventing problems that pose barriers to learning and achievement. Candidates develop knowledge and skills in working with school staff, parents, and family members to enable them to eliminate barriers to learning and achievement.

Please refer to the following link on complete thematic analysis on data that further demonstrates the assignment’s validity. 

Assignment has been constantly revised as per latest school counseling standards and faculty members own growing expertise in school counseling research and school counselor education. For instance, in last two years, faculty members have created a rubric to evaluate the assignment based on extensive discussions and then also added the component on ASCA national model mindsets to further enrich this assignment and ensure its actual reliability in helping school counseling interns translate what they have learned to promote school students academic success, socio-emotional wellbeing, and career development. Specifically, in the case-study, students have to also explain how their case conceptualization and interventions align with American School Counseling Association’s National Model of School Counseling mindsets from ASCA (4th edition):

M 1. Every student can learn, and every student can succeed.

M 2. Every student should have access to and opportunity for a high-quality education.

M 3. Every student should graduate from high school prepared for postsecondary opportunities. M 4. Every student should have access to a school counseling program.

M 5. Effective school counseling is a collaborative process involving school counselors, students, families, teachers, administrators and education stakeholders.

M 6. School counselors are leaders in the school, district, state and nation.

M 7. Comprehensive school counseling programs promote and enhance student academic, career and social/emotional outcomes.

Please refer to the following link on complete thematic analysis on data that also includes links to students’ individual assignments that demonstrate the reliability of this assignment in ensuring all students’ active engagement in the process of case conceptualization, implementing interventions, and evaluating the effectiveness of their interventions in accord with ASCA and CCTC standards.  

Across Standards 1 and 2, we utilized grades and candidate performance on key assignments. While we believe these assignments do reflect valid, reliable, and fair practices, moving forward, as a faculty, we will engage in inquiry to ensure this is indeed the case.

Mean Course Grades: The average GPA candidates achieved on a given assignment.

N186: Special Education Cross Cultural Case Study: The purpose of the assignment is to gain a broader perspective and more in-depth understanding of the Student Study Team and Individual Education Plan assessment team function, while exploring various cultural health beliefs and practices and how it affects the students ability to learn.

N136: Health Assessment:  A review of body systems, conducting health assessments, and how impairments can affect learning and function.

N184: Individual Health Care Plans: Purpose of the assignment is the development and implementation of an individualized health care plan by a school nurse of all the medical care needed to allow a student to attend school safely.  Used for the documentation of care/education of staff and monitoring of student’s health status and the safety of the school environment.

N184: Cultural Book Report: Books are assigned to enhance the cultural competence of school nurse candidates and to gain insight into the diversity of cultural practices and health care beliefs of the school community.

N185: Orientation/Staff Training Design Project: Purpose is to develop the school nurse manager role.  Assignment is to research, design and develop an orientation for health services staff taking into account federal, state, school practice, and law that impacts a school community.  As part of the orientation, the school nurse must design a program to implement and monitor training in health areas.

N186/N187: Leadership Role or Experienced Nurse Project: The purpose of these options is to give candidates additional choices for gaining clinical hours while encouraging them to stretch themselves outside their daily roles. 

N186/N187: Personal Learning Goals and Objectives: The candidate is able to personalize their learning experience by focusing on areas they are least comfortable in or have limited experience in order to have a well-rounded experience in the school.