AAQEP Accreditation 2022
Appendix E: Evidence of Data Quality
Preliminary Administrative Services Credential
Quantitative Data Measure: California Administrative Professionals Assessment | |
Description of Measure |
Candidates in each administrative services credential program in California must demonstrate
content and skill knowledge on three performance assessments prior to the attainment
of a preliminary credential. Each performance assessment or leadership cycle contains
required knowledge and skills found in the California Administrator Performance Expectations
(CAPEs). The CalAPA consists of three leadership cycles that focus candidates on
specific content knowledge, a different set of skills, and knowledge to demonstrate
new learning relevant to school administration. Each cycle includes professional
dispositions that school administrators should know and be able to demonstrate. There
are 4 steps in each CalAPA performance assessment which include the following actions:
Investigate, Plan, Act, Reflect in a continuous improvement model based on collected
data from fieldwork school sites.
|
Evidence (or plans) regarding trustworthiness | The CalAPA was developed and field-tested through a collaboration of the CCTC and
Pearson. School administrators with clear credentials front throughout the state were
brought together in working groups to determine what content would be included on
the assessment. Once items were determined, the assessment then went through a pilot
year during which each cycle was field-tested. Results were analyzed and the cycles
were revised. The assessment then went through a second, nonconsequential year of
testing to ensure validity and reliability of items. Annually, CalAPA administrators seek input from preliminary Administrative Services Credential coordinators throughout the state to learn how students are doing on the exam. This feedback is then used to revise items to ensure the assessment is as valid and fair as possible. |
Evidence (or plans) regarding reliability | All raters are calibrated at least annually to ensure reliable scoring of assessment
measures. Raters must hold a California Clear Administrative Credential. All faculty from within the Fresno State PASC program are fully calibrated scorers who participate in scoring of the state-level exams. |
Evidence (or plans) regarding fairness | Cut-off scores are determined annually to ensure pass rate does not disproportionately
impact administrative credential candidates from diverse backgrounds. Fresno State PASC program faculty use what they learn from their involvement in the scoring of the CalAPA to inform their instruction in the PASC program |
Quantitative Data Measure: President’s Commission of Teacher Education Questionnaire | |
Description of Measure | President’s Commission of Teacher Education Questionnaire. During university and local partners annual face-to-face meetings, the President's Commission on Teacher Education, a pilot 3-item open response feedback questionnaire was used to collect employer perceptions based on observed strengths of the Fresno State PASC graduates. The responses are analyzed for themes in relation to the CAPEs and AAQEP standards. |
Evidence (or plans) regarding trustworthiness | We let completers know their responses are anonymous and only utilized for program improvement to help with trustworthiness. |
Evidence (or plans) regarding fairness | In order to get a true representation of superintendent voices across the central valley, we would like to send this questionnaire or an adapted version of this questionnaire via email too for those who were not in attendance to the President’s Commission of Teacher Education. In addition, we want to also share this questionnaire in the P12 Superintendent Advisory Council as well. |
Measure: CCTC Completer Survey | |
Evidence regarding validity | The survey was designed to align with the California Professional Standards for Education Leaders (CPSEL), which are aligned with the California Administrative Performance Expectations (CAPEs). All survey items have been validated by the administrative bodies to determine their validity and are standardized across the larger systems |
Evidence regarding reliability | All survey items are assessed annually by the CTC to determine their reliability based on the responses received from program completers. |
Evidence regarding Fairness/Trustworthiness |
Because this instrument is administered statewide to candidates of all programs (CSU, University of California, private institutions, and local education agencies), we believe in its fairness as an instrument. Additionally, we believe the data the instrument yields are trustworthy because the survey is administered by the CTC, the statewide accrediting agency. Fresno State has initiated a college-wide data summit to consider the findings of this statewide survey and triangulate them with campus data, including the percentage of First Generation students, access to resources like scholarships, and culture and context of the cohorts in which prospective teachers are placed. Through this triangulation process, we are able to determine the alignment of the finding from the survey with our other measures, further assuring us of the survey’s trustworthiness as an instrument. In the process, we are also able to inform the impact on program changes on our own students with respect to the unique diversity of culture and needs in the Central Valley. |
Quantitative Data Measure: P12 PASC Program AAQEP Candidate Self-Assessment Descriptives Data for Standard 1 by Semester | |
Description of Measure | P12 PASC Program AAQEP Candidate Self-Assessment. The purpose of this self-assessment is to have students reflect on their perceptions of personal growth in the CAPEs and AAQEP standards over time in the PASC program. The information from this self-assessment is utilized to inform both areas of strength and opportunities for growth for students, as well as inform programmatic and instructional decision-making for ongoing continuous improvement. Each semester students are provided the opportunity to grow professionally as an aspiring equity driven school leader. Based on overall program coursework, activities, readings, and discussions, as well as fieldwork experiences through the CalAPA and other supervised activities with your field mentor supervision. This assessment is designed to help students reflect on a wide range of knowledge and experiences essential to their preparation. This self-assessment is divided into three sections: 1) AAQEP Program Standards Assessment, 2) CTC CAPEs Self-Assessment , and 3) Written Narrative Reflective Response. |
Evidence (or plans) regarding validity | Measure items are aligned with CTC and AAQEP standards. We informally tested the content with a pilot group of candidates who were finishing their final semester in the program to ensure understanding of the items. We can conduct a more formal pilot study of the items with a random group of candidates to ensure content clarity and understanding of what the items are asking. |
Evidence (or plans) regarding reliability | We utilize a controlled testing setting where all students are required to complete the assessment within a course structure during course time. Also, as more candidates respond to the survey and the sample size increases, we can consider running some basic statistical analysis on the items. |
Evidence (or plans) regarding fairness | Ensuring all candidates who have completed have an opportunity to share reflections and other needs every semester consistently. |
Reading/Literacy Specialist Credential
Key Data Measure: LEE 213 Inquiry Project | |
Description of Measure | In LEE 213-Teaching the Language Arts K-12, students complete an inquiry project that has three main components. First, students select a topic of inquiry driven by their professional experiences teaching language arts and write a research paper describing the different theoretical perspectives and respective instructional implications. Second, students use the research examined to develop and implement instructional lessons in the classroom setting. Third, students develop a presentation to share with colleagues that presents the theory of their report, the practical applications from their lessons, and a critical reflection on the experiences. |
Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
Key Data Measure: LEE 224 Case Study Report | |
Description of Measure | In LEE 224-Assessing & Developing Reading Abilities, students administer a variety of literacy assessments to an individual struggling reader in K-12, analyze the assessment results, and use the results to develop an individualized instructional plan. The students prepare a case study report that details the assessment tools and results, provides an analysis of the results, and provides instructional recommendations. |
Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
Key Data Measure: LEE 244 Literature Review | |
Description of Measure | In LEE 244-Research for Reading Professionals, students review research from the emergent reading, comprehension, and English Learner fields of literacy and construct a Wiki page. On this wiki page, students provide summaries of the research reviewed, including context, methods, and implications as well as a synthesis comparing the various theoretical perspectives that were examined. |
Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
Key Data Measure: LEE 254 Program Evaluation Report | |
Description of Measure | In LEE 254-Supervised Field Experiences in Reading, students use evaluation tools and school data to construct an evaluation report of their school site’s literacy program. In these reports, students provide analysis of data regarding school instructional procedures and curriculum materials, the strengths and weaknesses of these elements, and conclusions regarding program enhancement and professional development. |
Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
Key Data Measure: LEE 254 Coaching Presentation | |
Description of Measure | In LEE 254-Supervised Field Experiences in Reading, students collaborate with a colleague in 3 peer-coaching cycles, consisting of pre-consultation, observation/modeling, and debriefing consultation. The students prepare presentations for two of the cycles. Presentations include lessons learned about the coaching process, critical reflective insights about professional growth, and plans for future goals. |
Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
Key Data Measure: LEE 230 Discussion Posts | |
Description of Measure | The LEE 230 Supervised Teaching in Reading/Language Arts discussion assignments highlights students’ ability to identify and reflect on the ideological/philosophical underpinnings of teaching reading, to assess their own stances, and to critically reflect on the implications of that assessment. |
Process to Ensure Validity | This assignment invites candidates to make connections between course readings and activities related to literacy instruction and their current beliefs and instructional practices. Because the focus of the course is on supervising reading/language arts instruction, the focus of this assignment is valid. |
Process to Ensure Reliability | The fact that multiple candidates demonstrated similar understandings and were able to reflect on the content in thoughtful and meaningful ways indicates the discussion posts are reliable measures for evaluating their understanding of the focal content. |
Process to Ensure Fairness | All candidates received full credit for the assignment, as long as they demonstrated understanding of the content and made an effort to connect the content to their instruction. These guidelines were provided in the discussion prompt. |
Key Data Measure: LEE 215 Language Issues in Reading Course Grades | |
Description of Measure | The goals of this course are to prepare students to be knowledgeable about literacy
development and instructional practices through the study of theoretical perspectives
and scientific research on literacy processes and language development; and to prepare
students with the capacity to plan, implement, evaluate, and modify literacy instruction
to meet the needs of diverse struggling readers and English Language Learners. Key assignments in the course include reflection papers on course readings, reflections on key course content, and a final research paper on a topic relevant to the course. |
Process to Ensure Validity | Course grades are calculated based on students’ successful completion of all assignments. Because the assignments are specifically linked to course content, the grades serve as a valid measure of mastery of the course content. |
Process to Ensure Reliability | In order to establish reliability, as a program faculty, we will engage in joint scoring of student work using the established rubric to ensure that we are all in agreement on how the assignments should be scored. Where disagreements arise, we will work to come to a common understanding. We will repeat this process on (an annual, biannual, regular) cycle to ensure the ongoing reliability of the measure. |
Process to Ensure Fairness | To ensure fairness, we will analyze assignments instructions to make sure expectations are clear. One way to engage in this process is to have candidates articulate their understanding of what the assignment asks. Where discrepancies exist between what we intend candidates to do and what they understand, we will revise instructions to ensure they are clear to all. We will also be sure that clear details are provided about how the assignment will be assessed, including specific rubrics and, whenever possible, samples of previous students' successful work. |
Key Data Measure: Reading/Literacy Program Comprehensive Exam | |
Description of Measure | The comprehensive exam is one culminating experience option for students in the Reading/Literacy MA program. There is a bank of ten questions addressing key information in five of the required courses (LEE 213, LEE 215, LEE 224, LEE 244, and LEE 278). Students will randomly receive five questions on the exam (one question from each course), and then select three to write a response. |
Process to Ensure Validity | Exam aligns with student learning outcomes |
Process to Ensure Reliability | Exam questions and rubric were created by program faculty. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this exam in the same way. |
Process to Ensure Fairness/ Trustworthiness | All students have access to the exam requirements, question, and rubric ahead of time. |
Key Data Measure: Reading, Language, Literacy Specialist Credential Completer Survey | |
Description of Measure | The Reading, Language, Literacy Specialist Credential Completer Survey was created and piloted in Spring 2021 as a way to gauge completers’ perceptions of the program. Included in the survey are five likert-response items focused on how well the program prepared candidates: with knowledge of varied instructional approaches in the development of oral language, reading, and writing; to understand and use a variety of assessment techniques and instruments; to understand and use a variety of instructional techniques for ELL students; to understand the means of interacting with other professionals, parents, and community members concerning reading instruction; and to effectively meet the reading/language arts needs of their school. |
Process to Ensure Validity | Because each of the areas addressed represented a focal area of the program, we believe
this is a valid measure to evaluate program effectiveness from the perspective of
candidates at the point of completion. Additionally, responders were given the opportunity to provide additional information if they chose to do so. |
Process to Ensure Reliability | This survey was administered as a pilot. As a program faculty, we will evaluate responses to look for overall alignment as a way to ensure it is a reliable measure. |
Process to Ensure Fairness | The survey was sent to all candidates upon their completion of the program, providing all with the same opportunity to respond. Additionally, responses were collected anonymously, giving completers the opportunity to share their honest opinions without fear of retribution. |
Key Data Measure: Reading, Language, Literacy Specialist Credential Year+ Completer Pilot Survey | |
Description of Measure | The Reading, Language, Literacy Specialist Credential Year + Completer Survey was created and piloted in Spring 2021 as a way to gauge completers’ perceptions of the program one to five years after they finished. Included in the survey are items to collect background and demographic information on responders, including type of employment, along with six likert-style items, each of which was aligned with an AAQEP Standard 2 aspect. For each item, respondents were also given the opportunity to provide additional comments. |
Process to Ensure Validity | Because each of the items represented a focal area of the program in line with AAQEP
Standard 2, we believe this is a valid measure to evaluate program effectiveness from
the perspective of candidates at the point of completion. Additionally, responders were given the opportunity to provide additional information if they chose to do so. Moving forward, the program intends to hold focus group discussions with completers as a way to further validate the information provided in the survey responses. |
Process to Ensure Reliability | This survey was administered as a pilot. As a program faculty, we will evaluate responses to look for overall alignment as a way to ensure it is a reliable measure. |
Process to Ensure Fairness | The survey was sent to all individuals who completed the program between 2015 and 2020, providing all with the same opportunity to respond. Additionally, responses were collected anonymously, giving completers the opportunity to share their honest opinions without fear of retribution |
School Counseling
Quantitative Data Measure: Reflective assignments in COUN 201 that require research-based information to understand students’ experiences with intersectionality. | |
Description of Measure | In order to prepare culturally sensitive and responsive candidates, students are required to understand their own social experience in our societal context. Students assess the interaction of race, ethnicity, class, gender identity and expression, sexual identity, language acquisition, disability, learning style among other variables that impact behaviors and roles. |
Process to Ensure Validity | For these assignments, students were expected to evaluate their own identity as well as the intersectionality of multiple factors affecting their own experience and learning. Additionally, students explored the impact of privilege and oppression on their own development and future practice. Since the assignment’s expectations are in line with enhancing students’ cultural competence, it is a valid data source for std. 1 C. |
Process to Ensure Reliability | To assess the reliability of this assignment, we analyzed students’ assignments and found that students scored above satisfactory in all areas with scores higher than 12 points total (M=13.39). Overtime, we need to continue to analyze the data to ensure the assignment’s reliability. Please refer to the following link to access full dataset: COUN201Co.pdf |
Process to Ensure Fairness/ Trustworthiness | We believe this assignment demonstrates students’ understanding about intersectionality and cultural competence. There is also a clear rubric that assesses students’ cultural competence based on this assignment, contributing to the assignment’s fairness. Please see the following link for rubric: Rubric COUN201 |
Quantitative Data Measure: Research based projects in COUN 201 to analyze and understand the experience of marginalized groups and how experiences are shaped by environmental forces. | |
Description of Measure | Through engaging in this assignment, students were expected to explore and understand the experience of marginalized groups to understand their lived experiences. Additionally, students researched the impact of systematic forces on this group as well as ways to enhance the life of these marginalized groups. |
Process to Ensure Validity | In order to prepare culturally sensitive and responsive candidates, students are required to increase their awareness about the struggles and challenges of socially marginalized groups. Since this assignment encourages students to explore, understand, research, and articulate the lived experiences of people belonging to marginzalized groups, it is a valid source for analyzing std. 1 c. |
Process to Ensure Reliability |
Students were evaluated in their ability to follow instructions, write their ideas clearly, organize material, present content effectively, and engage in self-work. Students were required to incorporate research based resources to improve their understanding of the experience of marginalized groups and to learn about interventions that could improve the lived experiences of these groups. Providing students these clear directions supported with ensuring this assignment’s reliability in supporting students to engage in the specific requirements of this assignment. To analyze the effectiveness of this assignment, we examined the following dataset: COUN201Co.pdf Overall, students scored above satisfactory in all areas with scores higher than 12 points total (M= 13.32). It will be important to keep analyzing students’ responses to this assignment to assess its reliability overtime in expanding students’ cultural competence. |
Process to Ensure Fairness/ Trustworthiness |
We strive to ensure that the assignment is fair for all students by using the following rubric: Rubric COUN201 Through engaging students in this assignment, our hope was to evaluate students’ understanding of the impact of external and internal forces on marginalized groups, which will be important for strengthening all candidates’ ability to provide counseling to a diverse group of K-12 students, hence making this assignment a fair choice. |
Quantitative Data Measure: Mid-Term Exam in COUN 206. Please refer to the following link for syllabus: COUN 206 Syllabus for AAQEP Standard 1b |
|
Description of Measure | The Midterm exam consists of a clinical vignette component. The clinical vignette requires students to respond to one clinical vignette in an essay format. Students are expected to demonstrate competency with the relevant material taught to that point in the course. Specifically, students need to be able to connect developmental theory to a case example as well as identify normative and non-normative developmental factors that impact the individual(s) behavior. Students are also asked to discuss how developmental models across lifespan development explain the behavior seen in the vignette. |
Process to Ensure Validity | As indicated in COUN 206 syllabus, the purpose of this course is to provide an overview of various developmental concepts and principles as they relate to the practice of counseling. This course explores topics related to different aspects of development (i.e. social, emotional, moral, cognitive, physical, and other forces of development). This course also covers the elements of the life cycle process and supports students to understand how individual, familial, and social developmental forces interact to create both health and dysfunction. Since this exam assesses students’ ability to apply learning and developmental theories to meet the developmental needs of all individuals including K-12 students, it is a valid measure. |
Process to Ensure Reliability | Requirements to pass in the mid-term exam in COUN 206 stays relatively consistent over time since they are aligned with the CCTC program standards and CACREP standards given that COUN 206 is a core-counseling class in CACREP accredited specializations. In addition, the courses are staffed by the same faculty, on the whole to ensure reliability in content that is covered and assessed. |
Process to Ensure Fairness/ Trustworthiness | To examine our progress in preparing candidates to learn and apply theory--specifically, learning theory including social, emotional, and academic dimensions-- we use a midterm exam in COUN 206. Since the exam is same for all students, it is fair. |
Quantitative Data Measure: COUN 241 (Seminar in Organization of Counseling Services) Assignment titled “Research Paper and School Counselor Interview | |
Description of Measure | “Research Paper and School Counselor Interview” assignment provides students an opportunity to write and submit a research paper on a topic related to school counseling. |
Process to Ensure Validity | This paper directly supports students pursuing their Masters in School Counseling with PPS credential to actively read and reflect upon the research-based, peer-reviewed articles that can help them to gain pedagogical content. In addition, students are required to interview a professional school counselor on the topic of their interest in relation to school counseling and further reflect upon how they can apply the knowledge gained from this interview. Therefore, this paper is helpful for students’ professional development and directly related to the course content and std. 1 objective of the pedagogical, and/or professional knowledge relevant to the credential/degree sought by students, hence it a valid assessment for std. 1 A. |
Process to Ensure Reliability |
Faculty are able to ensure the assignment’s reliability across different semesters by providing clear and consistent directions: a. Papers will include two parts: A school counselor interview and literature review.
Students are required to critically analyze, synthesize, and evaluate research articles
on the topic of their interest. Students should be able to suggest their own conclusions
and implications for school counseling based on an interview with school counselor
and literature review. Please refer to the following link for a sample of students-work (N = 10): |
Process to Ensure Fairness/ Trustworthiness |
To ensure fairness, faculty provide a rubric along with specific directions for more detailed directions and expectations. Lit Review Rubric 2020.pdf In future, within the assignment description, we plan to add that students must use the knowledge gained from reading peer-reviewed journal articles and engaging in dialogues with professional school counselors to suggest two evidence-based practices that can promote a positive change within school settings – both at individual and systemic levels. We will also make a comprehensive rubric that focus on applicability of pedagogical content. |
Quantitative Data Measure: Please refer to the following link on the template that students use to participate and submit the assignment lesson plan: Lesson Plan | |
Description of Measure |
Students are required to create a 20-minute lesson plan based on the ASCA model and present it to the class (in PowerPoint) at the assigned dates. Objective: Develop, present, and evaluate a classroom lesson on school counseling core curriculum, including formative and summative assessments. The assignment includes the following 4 components: a. Summary of data/information: Students gather data from their school site to support
the need for their lesson. Data collection can be any of the following—i. available
data from the school, district, and/or state; ii. Observe and gather information from
the school Administration/faculty/staff to see what the needs of students are; ii.
Or they may collect anonymous surveys from students/parents/teachers/counselors. |
Process to Ensure Validity | The assignment is in direct line with American School Counseling Association’s focus on engaging in data-driven practices through the lesson plan. The content of assignment demonstrates validity in terms of assessing std. 1e objective of creating positive learning and work environments. |
Process to Ensure Reliability | We engaged in qualitative analysis of lesson plan assignments of randomly selected students to demonstrate the trustworthiness of the assignment. Our goal was to see the students provide materials and ideas that would engage the audience, are relevant to the skills/age group, and are collaborative in nature to ensure a positive learning and working environment. Please refer to the following link to access the complete dataset on lesson plans. |
Process to Ensure Fairness/ Trustworthiness | We used COUN 249 assignment lesson plan because it complemented the case study (individual intervention) through engaging in systemic level efforts to promote positive change for a group of students or staff (group intervention). The lesson plan is a way for students to brainstorm how to create a curriculum that teaches and engages the audience (to create a positive learning environment) and to use the ASCA Mindsets and Behavior Standards relevant to a positive learning and working environment. Since the assignment is same for all students, it is fair. Also, since the assignment is in line with ASCA national model, it will help students prepare for theur future school counseling roles. |
Key Data Measure: COUN 249 Case Study Presentation | |
Description of Measure | Students complete a Case Study report on a K12 student-client who they’ve been working with at their internship site. Apart from presenting about their work with a given student-client, students are also required to submit a written assignment based on the following format that explains the elements of this data source: Case Study |
Process to Ensure Validity |
Assignment aligns with student learning outcomes and CTC standards 23, 26, and 29: Standard 23: Advocacy Standard 26: Group Counseling and Facilitation Standard 29: Prevention, Education and Training Please refer to the following link on complete thematic analysis on data that further demonstrates the assignment’s validity. |
Process to Ensure Reliability |
Assignment has been constantly revised as per latest school counseling standards and faculty members own growing expertise in school counseling research and school counselor education. For instance, in last two years, faculty members have created a rubric to evaluate the assignment based on extensive discussions and then also added the component on ASCA national model mindsets to further enrich this assignment and ensure its actual reliability in helping school counseling interns translate what they have learned to promote school students academic success, socio-emotional wellbeing, and career development. Specifically, in the case-study, students have to also explain how their case conceptualization and interventions align with American School Counseling Association’s National Model of School Counseling mindsets from ASCA (4th edition): M 1. Every student can learn, and every student can succeed. M 2. Every student should have access to and opportunity for a high-quality education. M 3. Every student should graduate from high school prepared for postsecondary opportunities. M 4. Every student should have access to a school counseling program. M 5. Effective school counseling is a collaborative process involving school counselors, students, families, teachers, administrators and education stakeholders. M 6. School counselors are leaders in the school, district, state and nation. M 7. Comprehensive school counseling programs promote and enhance student academic, career and social/emotional outcomes. Please refer to the following link on complete thematic analysis on data that also includes links to students’ individual assignments that demonstrate the reliability of this assignment in ensuring all students’ active engagement in the process of case conceptualization, implementing interventions, and evaluating the effectiveness of their interventions in accord with ASCA and CCTC standards. |
Process to Ensure Fairness/ Trustworthiness | All students present their case study in class discussions that enable peers to engage in case consultation and ensure that case study is actually used to resolve the real concerns of K-12 students. Doing so enhances the utility of this assignment in promoting K-12 students’ wellbeing and success. Since all students use the provided format to wrote case study, the assignment is fair and trustworthy for all. Stakeholders such as site supervisors are also aware of this assignment and support students to successfully pursue their case study, further ensuring support for students. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
Key Data Measure: Counselor Disposition Assessment | |
Description of Measure | Students were assessed using the department’s 3-point likert-type Counselor Dispositional Assessment tool during internship (i.e., COUN 249). The likert scale descriptors are listed as evaluation guidelines for each of the dispositions. Included with each disposition are examples of behavioral indicators to help guide the evaluation process. Each indicator is offered as a suggested behavior, and not as a conclusive determining factor. Faculty are able to review data and draw inferences in order to assist with program evaluation methods. On a 3-point likert scale (1 = inadequate 2 = meet expectations 3 =exceeds expectations), supervisors evaluate students in six areas: (a) reflection, (b) critical thinking, (c) professional ethics, (d) valuing diversity, (e) collaboration, and (f) life-long learning. |
Process to Ensure Validity | The current Candidate Dispositions measure plays an important role in evaluating multiple
areas of students’ personal and professional capacities in serving clients from diverse
backgrounds. For example, this tool assesses candidates on the following areas: a)
Demonstrates the ability to work creatively and collaboratively with colleagues, clients,
families, and the community; (b) Values clients as full partners in the counseling/educational
process; (c) Collaborates with community partners and agencies in all phases of intervention
when possible; (d) Works well with others to develop opportunities for peer and student
learning; and (e) Plans and collaborates to ensure that appropriate supports for smooth
transitions are in place. Though the terminology used in this tool ensures content
and face validity, as faculty members committed to constantly reviewing and revising
our assessment tools, we again critically examined the tool in Spring 2021 using a
Candidate Disposition Assessment form that shows faculty members active engagement in critiquing this tool based on their
area of specialization. As a result of this critique and consistent discussions about validity of this tool in assessing students’ dispositions as counselors-in-training, we have decided to replace this tool with other well-established valid and reliable measure of counselor dispositions that is explained in the article. We are in the process of seeking authors permission to start using this tool starting Spring 2022. |
Process to Ensure Reliability | Measuring student dispositions is an important aspect of program assessment and, when combined with systematic progress reviews, constitutes an essential step for continuous counselor education program improvement. The school counseling programs require assessment of student dispositions as a necessary component of the program and student evaluation. Specifically, dispositions are the core values, attitudes, behaviors, and beliefs needed to become an effective and competent professional. Because student dispositions are an integral aspect of counseling skills and suitability for field performance, the measurement of dispositions is a natural component of the school counseling program assessment planning. One way in which we ensure reliability of this assessment is by using the same tool to assess students’ dispositions during practicum class as observed by their COUN 208 instructor and to assess students’ dispositions during COUN 249 as assessed by their internship site-supervisor. |
Process to Ensure Fairness/ Trustworthiness | Since school counselors personal development and professional dispositions play a major role on their work ethics and ability to advance a positive change by working with students at individual and systemic levels, seeking practicum and internship supervisors feedback for students using a candidates dispositions tool is a fair and trustworthy process. This process also ensures that students are assessed comprehensively in terms of not only their knowledge content but also other critical areas such as work ethics, cultural competence, ability to work collaboratively with others. |
Key Data Measure: Evaluation of Field Placement Student (Google form) | |
Description of Measure | As discussed in Standard 1 and Standard 2, students are supervised and evaluated during the 600 hour internship experience. School counselor complete the evaluation form for each student using the following evaluation Google form and providing open-feedback. The evaluations are submitted and the instructor carefully reviews each evaluation. If the supervisor identifies any concerns, or the student receives average ratings or lower, the instructor will contact the supervisor and the student to discuss the evaluation. |
Process to Ensure Validity | Practicum and internship supervisor evaluations, which are carefully revised by program faculty and coordinator, provide an opportunity to assess students in skill development and program outcomes. The data from these experiential courses is collected and analyzed to not only evaluate current students but also to identify areas of improvement and innovation. Additionally, program faculty and coordinator reviews these evaluations for any potential concerns that need to be addressed accordingly. If there are any concerns, the program faculty consults with the site supervisor to develop any potential remediation plan. Site supervisors also attend the semesterly Field Placement Orientation to receive information on our program, expectations for field placement so they can identify opportunities for interns at their sites, as well share information on their sites. Since site supervisors and faculty consistently review and discuss students’ actual progress based on these surveys and expectations laid down by CACREP and ASCA National School Counseling model, this survey undergoes regular content validity check by different stakeholders, ensuring its validity and revising it with changing standards and expectations. |
Process to Ensure Reliability | The reliability of the instrument comes from two principal sources, the number of
responders and the extent that their responses concur with one other. Each year the
data set yields the percent of respondents who gave specified answers to each item
and includes reliability estimates in the form of confidence intervals based on the
number of respondents and the concurrence or homogeneity of responses. All our surveys until last year were in the form of hard-copies collected at the end of semester, which we could not access due to the pandemic. In response to the COVID-19 pandemic and to make the process easier for site-supervisors, we transferred the questions of this survey to a google form, which we started collecting from Spring 2020 onwards. In standard 1, we demonstrated internship supervisor evaluation data for each targeted year, demonstrating a high quality performance and reliability of this measure in ensuring students professional growth as a result of their internship experience. In standard 1 documentation, we have used different items of this survey to demonstrate candidates' strong knowledge in learning theories, cultural competence, and data literacy. On average the analysis of all these items showed our students receiving the rating of “very satisfactory” which aligns with overall feedback that our students also get from their K-12 clients, administration, site supervisors, and program faculty - demonstrating the reliability of these survey results. In future, we want to analyse the data collected over at least 3 academic years to calculate its reliability estimates. |
Process to Ensure Fairness/ Trustworthiness | Faculty and site supervisors work closely to assess students’ performance throughout the program. These collaborative evaluation processes can support students’ plan to effectively work with students in multiple education settings. Given the close collaboration of site supervisors and school counseling faculty in evaluating students’ professional strengths and skill-set as counselors-in-training, the process of using this survey ensures fairness and trustworthiness in evaluating all students’ strengths and areas that need improvement. |
Key Data Measure: Advancement to Candidacy | |
Description of Measure | The Petition of Advancement to Candidacy form serves as a guideline for progress toward and completion of the degree. Students must complete this form by the sixth week of the semester prior to the term in which a student registers for the culminating experience, and/or applies for the graduate degree to be granted. The form can be accessed on this website: |
Process to Ensure Validity | The courses mentioned in Advancement to Candidacy form are in line with the course catalog, which has been approved by the university. Students complete the Advancement to Candidacy form by filling in the courses they have taken and will take by the end of the program. Each completed form is submitted to the office of Graduate Studies for approval. Signatures are obtained from the student, the program director and the student’s advisor. Once the Advancement to Candidacy is completed, our program is under contract with the student that we will not change any course requirements during their time in the program. Since the form is in line with courses mentioned in course catalog and is signed at various department, school, and university levels to ensure students’ steady progress towards degree completion, and the content on Advancement to Candidacy form reflect the important courses that are needed for professional development of counselors-in-training, it is a valid instrument to support students’ progress. |
Process to Ensure Reliability | All students referred to the key role of Advancement to Candidacy form during the internal audit process. Students explained that receiving signature from their advisor on Advancement to candidacy form ensured that they met the key requirements for degree completion. The Advancement to Candidacy form also ensures that our program keeps with the requirements of the Graduate studies and State of California. |
Process to Ensure Fairness/ Trustworthiness | Program faculty work closely with students to support their successful completion of the program and transition into professional careers. Students have an assigned advisor that helps planning their degree plan for graduation within the five years after admission. Advisors complete an advising sheet with students at the beginning of their program. One key role of advisor is to sign students’ Advancement to Candidacy form, which is in alignment with the courses listed in advising form. This ensures that all students receive the same information about courses needed and receive a mid-program check-in to ensure their successful progress towards degree completion. |
Key Data Measure: Comprehensive Exam | |
Description of Measure | A comprehensive examination is an assessment of the student’s ability to integrate the knowledge, show critical thinking and demonstrate mastery of school counseling. Our students have successfully passed the Comprehensive exam suggesting that they have demonstrated high levels of proficiency in the content, pedagogical, and professional knowledge necessary to function as a school counselor. |
Process to Ensure Validity | Comprehensive exam questions are consistently developed, reviewed, and revised by the faculty within the program. Faculty hired to teach within the programs are considered experts in their specific field and all have relevant experience for the content of the courses which they are teaching. The questions developed are in line with counseling standards and well-established texts, thus demonstrating strong content validity. |
Process to Ensure Reliability | We use a standard rubric to analyze students’ responses to comprehensive exam essay. This ensures reliability among the faculty evaluating the surveys. Moreover, two full-time school counseling faculty members evaluate students’ essays and grade them as per rubric. If one faculty member passes a student but other does not, the student’s essay is shared with third faculty member in the program who then evaluates the student’s essay independent and helps in making a reliable decision on passing or failing the student. |
Process to Ensure Fairness/ Trustworthiness | Every year, for a culminating experience, students in the MS in School Counseling may elect to take a Comprehensive Exam during the final stage of their program. For this standard, we are choosing students’ comprehensive exam essay as a data source because the essay requires them to analyze a student’s individual and systemic level concerns based on a vignette and further develop counseling goals and an ethically sound as well as multiculturally competent action plan based on their assessment(s). Hence, these essays can directly demonstrate students’ ability to engage in assessment and use the given data to inform their practice within K-12 educational settings. They also provide the program faculty a trustworthy means to guage students proficiency as counselors-in-training and serve the role as gatekeepers of counseling profession to ensure that only the students with strong knowledge and skill-set graduate and serve future clients. To ensure objectivity in evaluating essays and fairness for all students, the comprehensive exam coordinator removes any identifying information on students’ names or year in program from the responses. |
Quantitative Data Measure: 2021 Survey of Recent School Counseling Graduates | |
Description of Measure | We use the exit survey to assess completer’s perceptions of various aspects of the preparation program including their field placement experience as well as future educational/career plans. |
Process to Ensure Validity | Exit Survey serves as a valid measure of program completers’ perceptions of the school counselors’ preparation program because it asks questions directly aligned with the school counseling program’s learning objectives and CCTC standards of PPS credential. All program completers respond to items asking about their preparation of general pedagogical skills, such as their perception of how well the program prepared them to engage in individual and group counseling skills with specific group of students. In this way, the survey is a valid measure of completers’ perceptions of the program. |
Process to Ensure Reliability | Reliability about evaluation findings comes from two principal sources: the number of evaluation participants and the extent of their concurrence with each other. The evaluation findings become increasingly certain to the extent that the questions are answered by increasing numbers of program completers. We started using this survey during the academic year 2019-2020 limiting our capacity to assess the survey’s reliability. We plan to continue to use the survey every year and engage in analyzing survey’s reliability and then further revising the survey to make it more reliable. |
Process to Ensure Fairness/ Trustworthiness | Given the content of the questions asked, we believe the survey has face validity and content validity. However, in future we plan to triangulate the findings of this survey through engaging in focus group sessions with completers and students near graduation and then analyzing the results of this survey in the context of these discussions. Doing so would help us to ensure this survey’s validity and fairness. |
School Nursing
Key Data Measure: GPA of assignment and course | |
Description of Measure | Earned grades of a course or assignment |
Process to Ensure Validity | Aligned to course objectives and CCTC guidelines of mandated course content for expected
SN outcomes. Aligned to course and assignment rubrics. |
Process to Ensure Reliability | Criteria for grade determination aligned to course rubrics used by all course instructors. Course expectations and rubrics published in the syllabus. |
Process to Ensure Trustworthiness | Grade measures should align with scores from the preceptor skill checklists and other evaluation tools. |
Process to Ensure Fairness | All students have access to the grading rubric ahead of time and are based on criteria stated in the syllabus. Any exceptions that differ from syllabus are offered to all students. Students can access grades over the semester and can access instructor if questions occur. |
Key Data Measure: Preceptor Checklist of Skills (Elementary: Form 105/205 and Secondary: Form 106/206) | |
Description of Measure | Preceptor checklist of skills and dispositions determined by direct supervision of candidate in the field used by preceptors to evaluate candidates in their field work knowledge at two points. Candidates are rated on a 5-point Likert scale from 5 “Excellent” to 1 “Poor” on 40 items clustered into three areas: Providing Health and Wellness Services (Primary Intervention), Providing Direct Client Care Services (Secondary and Tertiary Intervention), and Professional Management Skills. |
Process to Ensure Validity | Criteria scored is clearly defined and stated in multiple areas. Criteria aligns with CCTC requirements of necessary skills for school nurse outcomes. |
Process to Ensure Reliability | Preceptors rate candidates and sign off skills during direct supervision of the candidate. Any discrepancies are discussed and reviewed with instructors of course |
Process to Ensure Trustworthiness | Coordinators individually judge candidate applications and meet to discuss discrepancies. Preceptor scoring should align with GPA/Instructor assessments. |
Process to Ensure Fairness | Preceptor skills check off evaluations are posted in the syllabus and available for
student review. Students receive feedback and review. Instructors review preceptor skill checkoff for relevance and scoring. |
Key Data Measure: Employer/Supervisor Survey of Program Effectiveness | |
Description of Measure | The Employer/Supervisor Survey of Program Effectiveness is administered to District Health Services Directors or Administrators who hire/supervise program candidates one year after candidates have graduated. Candidates are rated on a 4-point Likert scale in 10 areas that align with the CCTC School Nursing Program standards. |
Process to Ensure Validity | Criteria based on SNSC program objectives and goals. |
Process to Ensure Reliability | Coordinators review scoring and comments and look for patterns of results to improve the program. |
Process to Ensure Trustworthiness | Coordinators individually review employee/supervisor evaluations and meet to discuss discrepancies. |
Process to Ensure Fairness | Feedback from employers/supervisors solicited and discussed at SNSC Advisory meetings to determine needed program changes. |
Key Data Measure: Pre-Knowledge-Based Questionnaire & Post-Knowledge-Based Questionnaire | |
Description of Measure | The Knowledge-Based student self-rated questionnaire is administered at the beginning
and end of the program as a way to measure candidates’ perceived knowledge at program
entrance and after program completion. Items on the survey are clustered into 22 core
areas: Coordinated School Health Programs, Standards of School Nursing Practice, Nursing
Models/Theories, Families at Risk, Neurological Screening Tests, Legal/Ethical Issues,
and Child Abuse. Results provide the School Nurse Services Credential (SNSC) Program faculty an idea of the candidates’ knowledge base at entry and areas needing additional assistance to meet the SNSC program competencies. Following the questionnaire, students develop individual goals and objectives based on their learning needs. |
Process to Ensure Validity | The items included on the questionnaire include key items to be learned in the School Nursing Credential program and are aligned with the CCCTC Program Standards. |
Process to Ensure Reliability | Results from the questionnaire, particularly from the completers, are triangulated with student performance on course assignments and preceptor evaluations. Because we find alignment between these different measures, we believe the questionnaire is a reliable instrument. |
Process to Ensure Fairness | All candidates have the same access to the questionnaire. Their results are not counted against them in any way. Rather, the beginning-of-program results provide a way for candidates and program faculty to focus the learning experiences. At the end of the program, the questionnaire provides an opportunity for candidates to reflect on their growth and for faculty to learn candidates’ perceptions of their learning. |
Key Data Measure: CCTC Completer Survey | |
Description of Measure | Beginning in 2018, the CTC has administered a survey that is completed by completers of credential programs other than pre-service teaching. The survey, administered between September 1 and December 31, examines the effectiveness of individual educator preparation programs approved to operate in California. In 2019-2020, 95.6% of the 225 individuals who completed a School Nursing Credential program in California responded to the survey. Of those who responded, 96.7% completed their credential at a California State University. |
Process to Ensure Validity | This instrument has been used to evaluate the effectiveness of School Nursing Credential Programs across the State of California. Items are specifically tailored to completers of School Nursing Credential Programs and are clustered into five general areas: Background Information, Admission Process, Program Faculty and Coursework, Field Experience, and Assessment and Feedback/Program Guidance and Support. |
Process to Ensure Reliability | The reliability of the instrument comes from two principal sources, the number of responders and the extent that their responses concur with one other. The instrument becomes increasingly reliable to the extent that the items are answered in similar ways by increasing numbers of completers of School Nursing Credential Programs. Each year the data set yields the percent of respondents who gave specified answers to each item and includes reliability estimates in the form of confidence intervals based on the number of respondents and the concurrence or homogeneity of responses. |
Process to Ensure Fairness | Data were not constructed with bias. The existence of this statewide instrument allows us to examine trends in perceptions of individuals who were prepared as School Nurses across the system and compare them with perceptions of individuals who completed our program. Because the survey is administered statewide, we believe it is a fair and trustworthy instrument. |
Key Data Measure: Professional Dispositions | |
Description of Measure | This instrument is used by preceptors to assess the candidate’s ability to competently carry out required skills and competencies in clinical practice. Items address behavior needed for successful practice, including ability to engage in reflection, critical thinking, ethical judgements, valuing diversity, collaboration, and lifelong learning. Candidates are evaluated in each area on a 5 point Likert scale as observed by the preceptor in clinical practice. |
Process to Ensure Validity | The dispositions listed on the rubric align with disposition critical for a successful
school nurse. Moving forward, we will continue to review the instrument to ensure that the disposition rubric definitions are current and clearly stated. |
Process to Ensure Reliability | Moving forward, we will review the assessment forms with preceptors to ensure that all are in agreement with the rubric and how candidates are to be scored. |
Process to Ensure Fairness | Although this is an observational tool, the data is compared to additional assessment tools by which a candidate is scored for a composite picture of a candidate. Data can be compared between different preceptors for discrepancies to ensure fairness. |
Key Data Measure: Key Course Assignments | |
Across Standards 1 and 2, we utilized grades and candidate performance on key assignments. While we believe these assignments do reflect valid, reliable, and fair practices, moving forward, as a faculty, we will engage in inquiry to ensure this is indeed the case. Mean Course Grades: The average GPA candidates achieved on a given assignment. N186: Special Education Cross Cultural Case Study: The purpose of the assignment is to gain a broader perspective and more in-depth understanding of the Student Study Team and Individual Education Plan assessment team function, while exploring various cultural health beliefs and practices and how it affects the students ability to learn. N136: Health Assessment: A review of body systems, conducting health assessments, and how impairments can affect learning and function. N184: Individual Health Care Plans: Purpose of the assignment is the development and implementation of an individualized health care plan by a school nurse of all the medical care needed to allow a student to attend school safely. Used for the documentation of care/education of staff and monitoring of student’s health status and the safety of the school environment. N184: Cultural Book Report: Books are assigned to enhance the cultural competence of school nurse candidates and to gain insight into the diversity of cultural practices and health care beliefs of the school community. N185: Orientation/Staff Training Design Project: Purpose is to develop the school nurse manager role. Assignment is to research, design and develop an orientation for health services staff taking into account federal, state, school practice, and law that impacts a school community. As part of the orientation, the school nurse must design a program to implement and monitor training in health areas. N186/N187: Leadership Role or Experienced Nurse Project: The purpose of these options is to give candidates additional choices for gaining clinical hours while encouraging them to stretch themselves outside their daily roles. N186/N187: Personal Learning Goals and Objectives: The candidate is able to personalize their learning experience by focusing on areas they are least comfortable in or have limited experience in order to have a well-rounded experience in the school. |
|
Process to Ensure Validity | In order to establish content validity, we will analyze the key assignments to ensure they focus on content that aligns with the intended outcome of the course. Additionally, we will review rubrics to ensure that they in fact measure the key content areas the assignments are intended to assess. |
Process to Ensure Reliability | In order to establish reliability, as a program faculty, we will engage in joint scoring of student work using the established rubric to ensure that we are all in agreement on how the assignments should be scored. Where disagreements arise, we will work to come to a common understanding. We will repeat this process on (an annual, biannual, regular) cycle to ensure the ongoing reliability of the measure. |
Process to Ensure Fairness | To ensure fairness, we will analyze assignment instructions to make sure expectations are clear. One way to engage in this process is to have candidates articulate their understanding of what the assignment asks. Where discrepancies exist between what we intend candidates to do and what they understand, we will revise instructions to ensure they are clear to all. We will also be sure that clear details are provided about how the assignment will be assessed, including specific rubrics and, whenever possible, samples of previous students' successful work. |