Using Custom Diagnostic Reports to Explore the Influence of Academic Programs on Students’ Growth
The Custom Diagnostic report provides growth measures for groups of students at varying achievement levels and allows users to choose which students to include. LEAs/districts and schools can use this report to determine and identify patterns or trends of academic growth among students with similar educational opportunities and experiences.
CAUTION: The Custom Diagnostic reports are designed for diagnostic purposes, not accountability purposes. This report should be used for the purposes of continuous improvement of professional practice. Any attempt to create, interpret or include a Custom Diagnostic report for the purposes of a teacher’s summative evaluation would be an inappropriate and invalid use of the PVAAS data.
When using the Custom Diagnostic report, it is important to consider the students you are selecting. The criteria for the selection of students are defined by you as the user and should be established before selecting the students. These criteria will assist you in interpreting the resulting report.
Remember, this report only serves as one indicator; additional indicators should also be used when exploring the influence of educational programs on student performance. Additionally, care should be taken in using this report, as it does not imply direct causal relationships between any educational variables and student growth. For example, you may have two Math classes – one class in the morning and one class in the afternoon. We may see from generating Custom Diagnostic reports that the students in the morning class tend to have more growth than students in the afternoon class. We cannot say, however, that taking the class in the morning “causes” better growth. There are many possible root causes for student outcomes. We can say, however, that there appears to be a relationship between academic growth and when students take the class - morning or afternoon. We would then need to look at the many factors that may contribute to that relationship. Were different instructional strategies used earlier in the day versus later in the day? Was instruction adapted later in the day based upon what was learned from teaching students earlier in the day? Are students being pulled out from classes earlier in the day for various activities? These, as well as other questions, would need to be investigated to get at all possible root causes.
When choosing the criteria for selecting students, you are encouraged to carefully think about the educational programming or opportunities that are being investigated. The clearer the criteria are defined, the clearer the interpretation of the results.
Consider the example of a school that offers tutoring in Math and wishes to select all 39 students who received tutoring services during the school year. The resulting report is seen below. If all 39 students received tutoring support and had similar educational experiences within the classroom, this report indicates that the top third of students in regards to achievement exceeded the growth standard (i.e., gained ground), while the other two-thirds with middle to low achievement in relation to the whole group of 39 students met the growth standard (i.e., maintained their achievement).
The educators in this school may wish to look at the list of 13 students in the High group, and ask themselves questions such as:
- Did those students have a different tutoring experience than the other 26 students? Or, did all students receive the same tutoring experience that varied in its impact on student growth, depending upon the entering achievement of the student?
- Were different instructional strategies used with the 13 students in the High group than with the other 26 students? Or, were the same instructional strategies used with all students, producing different amounts of growth, depending upon the entering achievement of the student?
- Did the 13 students in the High group have a need in a particular assessment anchor or in standard(s) that were different from the other 26 students? Or, were the standards or assessment anchors that were taught in the tutoring program more appropriate to the needs of the High group of students than to the Middle or Low groups of students?
- How many hours of tutoring did students receive? Was there large variability in those hours?
Depending on the answers to those questions and the number of students selected, the educators may wish to refine their list and create a new Custom Diagnostic report with further defined criteria. For example, they may decide to look further at only students receiving 45 hours or more of tutoring instruction. As long as a minimum of 15 students meet this additional criterion, a Custom Diagnostic report can be produced.
Following the example above, the school now creates a Custom Diagnostic report for only the 24 students who received a minimum of 45 hours of tutoring. The resulting report can be seen below. From this report, we can clearly see that all groups of students (Low, Middle, and High) exceeded the growth standard (i.e., gained ground).
These two reports combined would provide additional evidence to the educators in this school that not only is the Math tutoring benefitting students, but also the largest impact for growth is occurring for those students who participate in tutoring services for a minimum of 45 hours during the school year.
REMEMBER: When interpreting this report, NO causal relationship between growth and educational variables can be inferred from this report. This report is intended to serve as one indicator.