Standard 2 Aspect A

Standard 2a. Analysis of Completers Preparedness
To measure whether completers understand and engage local school and cultural communities, and communicate and foster relationships with families/guardians/caregivers in a variety of communities, these instruments were used: Comp Exam, Counselor Disposition, and Evaluation of Field Placement Student.

---

Data Sources & Analysis:

Data Source 1

Comprehensive Exam (Data Set) 

Perspective Captured from Data Source:
Three School Counseling Faculty members grade the exam. 

Rationale for Using Data Source:
The multiple choice (MC) section measures students’ knowledge of course contents that prepares them to work in school and local communities.  Questions for the MC section derives from knowledge gained from the following courses: Counseling Techniques; Multicultural Aspects of Counseling; Group Counseling; Assessment in Counseling; Counseling Through the Lifespan; Career Counseling; Practicum; Counseling of Exceptional Children and their Parents; Organization of Counseling Services; and Seminar on Parent Education, Pupil Advocacy, and Consulting.  The vignette section measures how students conceptualize the case in the vignette and how they apply the knowledge gained from the program.  As seen in the Comp Exam Vignette Rubric, we assess students' knowledge and application of the American School Counselor Association’s National Model and the school counseling consultation model in the vignette, which focus on working with all the systems and professionals involved in each student case.  We also assess how they set goals at the micro and macro levels at the school system, who they’d consult with, and what multicultural/legal/ethical issues need to be addressed.

Specific Elements of Data Source:
Since all elements of the courses are needed to be effective counselors, both the Multiple Choice and the written vignette sections will be used as data sources. Data from 2018, 2019, 2020, and 2021 are used.  Since the pandemic impacted the students’ learning the most in 2020, we included two data sets for that year to review. Completers are asked to identify counseling skills and knowledge needed for building rapport and work with diverse communities in the MC sections.  Examples of multiple choice questions included approaches to empower parents, who to include in IEP meetings, who to consider when building family alliances, and strategies to building alliances.  They are also asked to respond to the vignette on how they’d work with the individual student and in collaboration with the school on a certain issue the student is facing.  

Definition of Success for Each Element:
Students who achieve 70% or more correct responses on their MC and Vignette portion are considered successful in grasping and applying knowledge obtained from the program and knowledge to use various resources to meet the needs of diverse students and families. Students are allowed to retake any portion of the exam they do not pass up to three times. 

Displays of Analyzed Data:
As seen in the bar graph below, most of the students pass both the MC and Vignette portions of the COMP Exam on their first attempt, and from 2018 to 2021, all students passed their COMP Exam (culminating experience) during their second try. Every semester, we have between 10 and 15 students taking the Comp Exam.  Note that each fall semester, we only admit about 18 to 20 students into the school counseling program. We believe that the pandemic may have impacted students’ scores in Fall 2020 for the written vignette portion, resulting in five fails. However, upon closer inspection, those five passed the Vignette portion in Spring 2021 on their second try.  

Interpretation of Data and Moving Forward:
The Comprehensive Exam (COMP Exam) provided an overview of how much knowledge students have retained or learned from their courses in the MC section and provided an idea of how students apply their knowledge to a case vignette in the Vignette portion.  By passing the exam, they have enough knowledge to approach cases from a multicultural, ethical, and legal stance to foster and engage with students and communities to create goals and create individualized  interventions at the individual level and at the systemic level. Moving forward, if we do not have enough data from the other assessments, we may need to do a qualitative analysis of the students’ vignette responses to look for strengths and weaknesses to improve curriculums and the program in the area of engaging and fostering relationships with diverse communities.  We may also need to do an item analysis to see which areas students are the strongest in and which ones they are not so we could strengthen our program and curriculums.

Data Source 2

Counselor Disposition Assessment (Full Survey, Data Set, n=44): 
Field practice site supervisors rate completers based on their field practice at different school districts or programs serving primary and secondary pupils. Site supervisors rate completers on the following items based on their observations and by using the following ratings: 0= not observed; 1=inadequate; 2=meet expectations; 3=exceed expectations.

Perspective Captured from Data Source: Field practice site supervisors

Rationale for using Data Source:
Although these evaluations take place while candidates are still in the credential program, we believe that assessing them in these key areas prepares them to draw on the knowledge and skills they develop once they enter the field. 

Specific Elements of Data Source:
The following items demonstrate candidates’ ability to work creatively and collaboratively with colleagues, clients, families, and the community. 

• Collaborates with community partners and agencies in all phases of intervention when possible.
• Works well with others to develop opportunities for peer and student learning.
• Plans and collaborates to ensure that appropriate supports for smooth transitions are in place.
  Spring 2021 (Data Set, n=44)

Definition of Success for Each Element:
Candidates are considered successful if they receive ratings of 2 or 3.  If they receive a 0, that means the site supervisor didn’t get a chance to observe their abilities in that category.

Display of Analyzed Data:

Analysis of Data Source 2:
Out of 44 students assessed, only one didn’t meet expectations  for three of the items analyzed.  Forty-three students met expectations in all areas on the survey, most of them exceeding expectations in collaborating with community partners and agencies, working well with others, and planning and collaborating to ensure support of students.  For the one student who didn’t meet expectations, the instructor reached out to the site supervisor and students for consultation on how to help the student improve.  The instructor, Dr. Sharma, reported that the student is an English learner, which impacted understanding and communication with the site supervisor.  The remediation plan was for the student to present a case study in addition to specific details on the activities engaged to support their learning at the internship site. The student also articulated specific instances of miscommunication and apologized for not engaging in direct communication with the supervisor. Since these instances highlighted language and cultural differences, the result of the meeting was for the student to continue engaging in internship experiences and submit assignments in a timely manner and to engage in ongoing communication with current and future supervisors..

Data Source 3

Evaluation of Field Placement Student (Data Set for Spring 2021)

Perspective: Site supervisors rate the completers on their knowledge and skills.

Rationale for using Data Source:
Although these evaluations take place while candidates are still in the credential program, we believe that assessing them in these key areas prepares them to draw on the knowledge and skills they develop once they enter the field. 

Specific Elements of Data Source: 
Six items relevant to Standard 2a were selected. These include:

• Item 5) Understand and use tests and measures used in assessing student learning and achievement, development of school, family, and community partnerships. 
• Item 8)Have knowledge and ability to work with students of diverse backgrounds. 
• Item 11) Able to learn about the duties that a school counselor will perform; adapted well to conditions. 
• Item 13) Have knowledge of relevant resources and connects with students to resources needed to assist in academic achievement; 
• Item 14) Have knowledge and abide by policies and processes relating to academic development at the school, district, state, and federal levels; 
• Item 25) Demonstrate Knowledge of and skills in consulting with and educating school staff on social emotional needs of students.  These items ask for knowledge and abilities to work in local schools and cultural communities.

Definition of Success for Each Element:
Completers are deemed successful if they’ve achieved a 3 or 4 rating satisfied and most satisfied ratings).  

Display of Analyzed Data:
See below for a bar graph of all items (Rating colors: 4=green, very satisfactory; 3=yellow moderately satisfactory: 2=red, moderately unsatisfactory; 1=blue, very unsatisfactory.)

Interpretation of Data Source 3.  We believe all items of knowledge in this assessment are needed to understand and engage local school and cultural communities, and communicate and foster relationships with families/guardians/caregivers in a variety of communities. However, the most relevant are items 5, 8, 11, 13, 14, and 25. As seen above, almost all of our students (96%; 50/52) received satisfactory remarks, with more than half receiving the highest remarks (the green bars) on all 31 items on tasks and roles of a school counseling, which involves working with other professionals, organizations, and the diverse students and their families.

Note that there were two students that received unsatisfactory remarks out of 52 students.  In both cases, comments from site supervisors remarked that the dissatisfaction resulted from lack of communication.  One of the students' lack of communication was due to language barrier and the issue was resolved through an alternative assessment.  The other student’s low rating was due to unprofessional behaviors such as not completing tasks and not communicating needs.  The internship coordinator has reached out to the instructor and student and will proceed as needed in remediation plans or the student can be referred to our Clinical Review process. There is a process in place and clearly stated in the syllabus and student handbook for working with students who fall below satisfactory remarks.

Next Step:
The COMP Exam seems to indicate that our students are learning and retaining the information needed to be a counselor who engages with other individuals and agencies to meet the needs of K12 pupils.  The Counselor Disposition and Evaluation of Field Placement Students provided us how their site supervisors view their knowledge and abilities in engaging and building relationships with others.  Although the scales show that completers have knowledge and work well with others, it’s not telling how they are engaging with diverse communities.  The structure of our school counseling is based on the humanistic approach, using the core conditions to build relationships with others.  However, we do not have a measurement of how completers are doing in engaging with students, families, other professionals, and other communities or agencies. 

As our program moves forward, our next step is to collect and analyze data from the perception of pupils and families whom completers serve. Additionally, because data are now collected electronically, we will be able to engage in more systematic analysis of results, including disaggregating comp exam performance by specific item in order to better understand specific areas where candidates may have challenges.

Aspect B →