AAQEP Accreditation 2022
Appendix E: Evidence of Data Quality
Reading/Literacy Specialist Credential
| Key Data Measure: LEE 213 Inquiry Project | |
| Description of Measure | In LEE 213-Teaching the Language Arts K-12, students complete an inquiry project that has three main components. First, students select a topic of inquiry driven by their professional experiences teaching language arts and write a research paper describing the different theoretical perspectives and respective instructional implications. Second, students use the research examined to develop and implement instructional lessons in the classroom setting. Third, students develop a presentation to share with colleagues that presents the theory of their report, the practical applications from their lessons, and a critical reflection on the experiences. |
| Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
| Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
| Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
| Key Data Measure: LEE 224 Case Study Report | |
| Description of Measure | In LEE 224-Assessing & Developing Reading Abilities, students administer a variety of literacy assessments to an individual struggling reader in K-12, analyze the assessment results, and use the results to develop an individualized instructional plan. The students prepare a case study report that details the assessment tools and results, provides an analysis of the results, and provides instructional recommendations. |
| Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
| Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
| Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
| Key Data Measure: LEE 244 Literature Review | |
| Description of Measure | In LEE 244-Research for Reading Professionals, students review research from the emergent reading, comprehension, and English Learner fields of literacy and construct a Wiki page. On this wiki page, students provide summaries of the research reviewed, including context, methods, and implications as well as a synthesis comparing the various theoretical perspectives that were examined. |
| Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
| Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
| Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
| Key Data Measure: LEE 254 Program Evaluation Report | |
| Description of Measure | In LEE 254-Supervised Field Experiences in Reading, students use evaluation tools and school data to construct an evaluation report of their school site’s literacy program. In these reports, students provide analysis of data regarding school instructional procedures and curriculum materials, the strengths and weaknesses of these elements, and conclusions regarding program enhancement and professional development. |
| Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
| Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
| Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
| Key Data Measure: LEE 254 Coaching Presentation | |
| Description of Measure | In LEE 254-Supervised Field Experiences in Reading, students collaborate with a colleague in 3 peer-coaching cycles, consisting of pre-consultation, observation/modeling, and debriefing consultation. The students prepare presentations for two of the cycles. Presentations include lessons learned about the coaching process, critical reflective insights about professional growth, and plans for future goals. |
| Process to Ensure Validity | Assignment aligns with student learning outcomes and CTC standards (see Reading and Literacy Standards course Matrix) |
| Process to Ensure Reliability | Assignment and rubric were created by program faculty. All program faculty who teach
this course use this assignment and rubric. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this assignment in the same way. |
| Process to Ensure Fairness/ Trustworthiness | All students have access to the assignment requirements and rubric ahead of time. |
| Key Data Measure: LEE 230 Discussion Posts | |
| Description of Measure | The LEE 230 Supervised Teaching in Reading/Language Arts discussion assignments highlights students’ ability to identify and reflect on the ideological/philosophical underpinnings of teaching reading, to assess their own stances, and to critically reflect on the implications of that assessment. |
| Process to Ensure Validity | This assignment invites candidates to make connections between course readings and activities related to literacy instruction and their current beliefs and instructional practices. Because the focus of the course is on supervising reading/language arts instruction, the focus of this assignment is valid. |
| Process to Ensure Reliability | The fact that multiple candidates demonstrated similar understandings and were able to reflect on the content in thoughtful and meaningful ways indicates the discussion posts are reliable measures for evaluating their understanding of the focal content. |
| Process to Ensure Fairness | All candidates received full credit for the assignment, as long as they demonstrated understanding of the content and made an effort to connect the content to their instruction. These guidelines were provided in the discussion prompt. |
| Key Data Measure: LEE 215 Language Issues in Reading Course Grades | |
| Description of Measure | The goals of this course are to prepare students to be knowledgeable about literacy
development and instructional practices through the study of theoretical perspectives
and scientific research on literacy processes and language development; and to prepare
students with the capacity to plan, implement, evaluate, and modify literacy instruction
to meet the needs of diverse struggling readers and English Language Learners. Key assignments in the course include reflection papers on course readings, reflections on key course content, and a final research paper on a topic relevant to the course. |
| Process to Ensure Validity | Course grades are calculated based on students’ successful completion of all assignments. Because the assignments are specifically linked to course content, the grades serve as a valid measure of mastery of the course content. |
| Process to Ensure Reliability | In order to establish reliability, as a program faculty, we will engage in joint scoring of student work using the established rubric to ensure that we are all in agreement on how the assignments should be scored. Where disagreements arise, we will work to come to a common understanding. We will repeat this process on (an annual, biannual, regular) cycle to ensure the ongoing reliability of the measure. |
| Process to Ensure Fairness | To ensure fairness, we will analyze assignments instructions to make sure expectations are clear. One way to engage in this process is to have candidates articulate their understanding of what the assignment asks. Where discrepancies exist between what we intend candidates to do and what they understand, we will revise instructions to ensure they are clear to all. We will also be sure that clear details are provided about how the assignment will be assessed, including specific rubrics and, whenever possible, samples of previous students' successful work. |
| Key Data Measure: Reading/Literacy Program Comprehensive Exam | |
| Description of Measure | The comprehensive exam is one culminating experience option for students in the Reading/Literacy MA program. There is a bank of ten questions addressing key information in five of the required courses (LEE 213, LEE 215, LEE 224, LEE 244, and LEE 278). Students will randomly receive five questions on the exam (one question from each course), and then select three to write a response. |
| Process to Ensure Validity | Exam aligns with student learning outcomes |
| Process to Ensure Reliability | Exam questions and rubric were created by program faculty. Moving forward, program faculty will meet to calibrate the rubric to ensure all faculty are using the rubric to assess this exam in the same way. |
| Process to Ensure Fairness/ Trustworthiness | All students have access to the exam requirements, question, and rubric ahead of time. |
| Key Data Measure: Reading, Language, Literacy Specialist Credential Completer Survey | |
| Description of Measure | The Reading, Language, Literacy Specialist Credential Completer Survey was created and piloted in Spring 2021 as a way to gauge completers’ perceptions of the program. Included in the survey are five likert-response items focused on how well the program prepared candidates: with knowledge of varied instructional approaches in the development of oral language, reading, and writing; to understand and use a variety of assessment techniques and instruments; to understand and use a variety of instructional techniques for ELL students; to understand the means of interacting with other professionals, parents, and community members concerning reading instruction; and to effectively meet the reading/language arts needs of their school. |
| Process to Ensure Validity | Because each of the areas addressed represented a focal area of the program, we believe
this is a valid measure to evaluate program effectiveness from the perspective of
candidates at the point of completion. Additionally, responders were given the opportunity to provide additional information if they chose to do so. |
| Process to Ensure Reliability | This survey was administered as a pilot. As a program faculty, we will evaluate responses to look for overall alignment as a way to ensure it is a reliable measure. |
| Process to Ensure Fairness | The survey was sent to all candidates upon their completion of the program, providing all with the same opportunity to respond. Additionally, responses were collected anonymously, giving completers the opportunity to share their honest opinions without fear of retribution. |
| Key Data Measure: Reading, Language, Literacy Specialist Credential Year+ Completer Pilot Survey | |
| Description of Measure | The Reading, Language, Literacy Specialist Credential Year + Completer Survey was created and piloted in Spring 2021 as a way to gauge completers’ perceptions of the program one to five years after they finished. Included in the survey are items to collect background and demographic information on responders, including type of employment, along with six likert-style items, each of which was aligned with an AAQEP Standard 2 aspect. For each item, respondents were also given the opportunity to provide additional comments. |
| Process to Ensure Validity | Because each of the items represented a focal area of the program in line with AAQEP
Standard 2, we believe this is a valid measure to evaluate program effectiveness from
the perspective of candidates at the point of completion. Additionally, responders were given the opportunity to provide additional information if they chose to do so. Moving forward, the program intends to hold focus group discussions with completers as a way to further validate the information provided in the survey responses. |
| Process to Ensure Reliability | This survey was administered as a pilot. As a program faculty, we will evaluate responses to look for overall alignment as a way to ensure it is a reliable measure. |
| Process to Ensure Fairness | The survey was sent to all individuals who completed the program between 2015 and 2020, providing all with the same opportunity to respond. Additionally, responses were collected anonymously, giving completers the opportunity to share their honest opinions without fear of retribution |