Fresno State CREATe Observation Pilot Study: Summary and Major Takeaways

Please direct all further inquiries to Dr. Cathy Yun and Dr. Heather Horsley, Fresno State University.

Summary

In Spring 2019, The Fresno State New Generation of Educators Initiative Team, in partnership with WestEd, The New Teacher Project, and Sanger Unified School District, conducted a set of teacher observations of graduates and non-graduates of the Fresno State residency program. The purpose of this study was to analyze the use of the Continuum of Reflective, Engaging, and Accessible Teaching (CREATe) rubric in the field in an effort to continuously improve the teacher preparation program. Overall findings suggest that the CREATe rubric is successfully differentiating performance trends across three of its four dimensions (internal consistency/reliability), and that overall performance trends on CREATe are consistent across different observation instruments (convergent construct validity).

Activities

A strong partnership with the Sanger Unified School District New Teacher Induction program was the foundation to initiating this pilot study. A sample of 28 teachers who were in the first year of teaching in the district were pre-selected from the Induction cohort. Observations were scheduled via direct contact with coordinators. Each observation was conducted by a pair of observers, present in the classroom together: one observer who was calibrated on the CREATe rubric, the other observer was calibrated on the TNTP CORE rubric. 24 teachers were successfully observed during the observation window, with 4 teachers excluded due to scheduling/observation conflicts or content-areas unobservable by TNTP CORE (e.g.; physical education). Observer data was compiled and merged to enable data analysis.

Findings

Three major takeaways from the observation data are summarized below. Given the small sample, these takeaways do not represent generalizable findings; however they do provide key insights on which action steps may be designed:

1) Both CREATe and CORE are measuring performance similarly:

Average performance on the CORE rubric was correlated with average performance on the CREATe rubric (Pearson r=0.35, Spearman p=0.38). These rubrics are comprised of different dimensions and possess distinct scoring scales, but the aggregate performance trends were consistent across the two instruments. Given that CORE is a widely-used field instrument, this suggests a convergent construct validity for the newer CREATe rubric.

2) Confirmatory Factor Analysis of the CREATe rubric suggests that the four proposed dimensions of the rubric hold up relatively well, with the exception of the 12th and 13th indicator:

The CREATe rubric places teachers on a continuum of development on fourteen indicators, distributed across four dimensions. With the exception of the third dimension (which includes indicator #12 and #13), a confirmatory factor analysis (CFA) and analysis of internal consistency both suggest that the other dimensions of the rubric hold together as outlined in the original design of the rubric (for CFA, p<0.01 for factor loadings excluding #12/#13; a>0.8).

3) Generally speaking, qualitative feedback for the CREATe observations provides more evidence about the lesson, while qualitative feedback for the CORE observations provides more evidence about the scoring decision:

The style of offering qualitative feedback on each instrument is distinct. This is not a referendum on either approach, but a noted difference worth exploring and considering, in an effort to ensure that teachers are able to link their placement on the CREATe with the associated evidence for each placement.

Next Steps

The findings from this pilot study provide evidence to support the continued use of CREATe as a field-tested observation instrument. Further examination of the protocols for, and calibration training of, indicators #12 and #13 on CREATe would follow from this analysis; however, if prior analyses suggest that these indicators are ordinarily loading consistently with other indicators on the “content” dimension, this may be an unsubstantiated anomaly. Lastly, additional review of qualitative scoring protocols from the viewpoint of teachers is advised, to ensure that evidence for placing teachers on the CREATE continuum is easily discernible for teachers receiving written feedback from observers.

Download the PDF

This HTML version was generated with AI assistance and may contain errors or omissions. The original approved PDF remains the official document of record. An alternative accessible format is available upon request from the document creator in accordance with Title II of the ADA.