This first Module is designed to help anyone interested in student achievement, school performance, and accountability to gain a better understanding of the various assessment and data systems in Ohio. Some of these have been around for a while, and some are new, such as "value-added" approaches to interpreting student test results. In this introductory Module, you will find information and activities to help you learn about:
- The concept of "value-added" and "growth model" approaches, and how these are related to other school performance measures such as School and District "Report Cards," performance ratings, and "Adequate Yearly Progress" required by No Child Left Behind.
- How value-added assessment will be introduced in Ohio, how it works, and what it can mean for students, teachers, school leaders, and parents.
- Unique aspects of value-added accountability measures that focus on student progress and growth as opposed to simply looking at test results at one point in time.
- How value-added results relate to curriculum, instruction, and assessment choices that teachers make for their classrooms.
- How value-added approaches should and shouldn't be used in making decisions about school quality and improvement.
- Context
- Various Accountability Measures and What They Capture.
With the addition of "value-added" growth results, the state of Ohio will have three distinct types of measures of school and district quality, each focused on a different aspect of student achievement. All of these will be available to the public on the annual "Report Cards" for schools and districts. The three types of measures are:
- "Status" reporting, which summarizes overall performance on achievement tests, attendance, and graduation rates for the district and school (as applicable—for example, elementary schools do not have a "graduation rate" calculated into their performance, and high schools do not have test scores for elementary or middle grades).
- "Adequate Yearly Progress" reporting, which indicates how the school and district are helping each specific sub-group of children progress academically.
- "Value-added Progress" reporting, which will look at the academic growth of each individual student to better understand the support that schools and districts are providing for student progress.
Taken together, these three types of reported measures will give teachers, school leaders, and the public information they need to have a more well-rounded understanding of the work that schools are doing to support student achievement. While all of these measures analyze standardized test results and other data that are common across schools (such as attendance and student demographic information like race/ethnicity, gender, and poverty status), they each analyze the data in a different way in order to shed light on different aspects of the work that we want our schools to accomplish. It is, of course, important that they rely on standardized measures of student achievement and other common data. While these sources of data do have some limitations (which we'll discuss in detail a bit later), the fact that they are collected and "scored" the same way across all schools is what lets us compare results across the state—otherwise, we would be attempting to compare "apples and oranges."
We'll now look at each of these three types of achievement and progress measures in order to better understand what each one does and does not do, and how they relate to each other.
- School and District Performance Designations and Performance IndexScores
Performance Designations and Performance Index Scores are two ways in which standardized test scores, attendance rates, and (for high schools and districts) graduation rates are summarized across all students to get an overall "snapshot in time" of how well students in a particular school or district are doing.
General student performance and participation overall, for the year in question.
Both of these measures summarize current (or recent) achievement status. Because they are looking at all kids in a school or district, some things may not be obvious. For example, if a population of students changes significantly in a school or district, we may see these indicators change without the change being due to the work that teachers or schools are doing. Also, because scores are grouped together, it can be difficult to see if particular sub-groups of students are doing well or performing poorly. The Adequate Yearly Progress measure is designed to help address this problem.
Performance Designations are assigned depending on the percentage of how many indicators are satisfactorily met, out of the number possible. For districts, there are 30 indicators. These indicators are: (#1 through 28) the percentage of students achieving a proficient score on every test offered in grades 3 through 11 in math, reading, writing, social studies, and science (28 tests in all); (#29) overall attendance rate; and (#30) the graduation rate of high school students. For the highest designation, Excellent, a district must meet 29 or 30 of the indicators, meaning that at least 75% of students passed each of the tests in all grades and that the district reached an overall attendance rate of at least 93% and an overall graduation rate of at least 90%.
At the individual school level, only some of the 30 indicators will be applicable—only the tests in the grades that the school has, plus attendance rate. The graduation rate indicator also is counted for high schools.
Performance Designations are: Excellent (29 or 30 indicators met); Effective (23 to 28 met); Continuous Improvement (15 to 22 met); Academic Watch (10 to 14 met); and Academic Emergency (0 to 9 met).
The Performance Index Score is a special analysis that summarizes only the tests given in grades 3, 4, 5, 6, 7, 8, and 10 in math, reading, writing, social studies, and science (23 total tests in all). These scores are "weighted," or averaged together in a certain way, in order to give a score that can be compared across schools regardless of the fact that one school may offer a different group of tests than another school. Different schools will offer different tests depending on the grade levels in the school. For example, an elementary school with grades Kindergarten through 6 would offer a total of 11 tests—reading and math each year in grades 3 through 6, plus writing in grade 4, and social studies and science in grade 5. However, a high school with grades 9 through 12 would only offer 5 tests—reading, writing, math, social studies, and science at the 10th grade. The Performance Index Score is calculated so that the score for the elementary school can be compared to the score for the high school, giving a quick summary of the academic performance of the students for each school.
- Adequate Yearly Progress ("AYP")
The annual check on how much progress particular groups of students in a school or district have made. The federal No Child Left Behind legislation requires schools and districts to pay attention not only to their overall student academic scores (as with the Performance Designations and Indicators above), but also to special groups. This is intended to avoid the possibility that smaller groups of students in the minority who are having trouble are overlooked because the majority of students are doing well.
A summary of standardized test results for each sub-group of students. These are calculated annually for each school and district, and the progress of the school or district on moving each student group toward 100% proficiency is noted.
The average results and the progress for each of the sub-groups of students that are followed. NOTE, however, that even though the progress is compared annually, this is NOT a measure that follows any individual student's progress year to year. This is because the measure looks at, for example, "all African American students' scores this year" and compares that result to "all African American students' scores last year"—these are NOT the same students in each group. Therefore, AYP is designed to tell us about a school's or district's progress in working with these groups year to year, not necessarily how well each individual student is progressing. To meet that need, we must turn to the Value-Added calculations.
Like the Performance ratings, AYP uses standarized test scores. There are two differences though. The first is that the scores are first grouped by student types so that the performance of sub-groups is summarized. These sub-groups are: 1) African American, 2) American Indian/ Native Alaskan, 3) Asian/Pacific Islander, 4) Hispanic, 5) Multi-racial, 6) White, 7) Economically Disadvantaged, 8) Limited English Proficient, 9) Students with Disabilities. Note that some students may fall in more than one of these groups. The second difference is that instead of just summarizing scores for the one year—a "snapshot in time"—AYP calculations look at the growth of the group over time. In order to meet annual AYP goals, a group's performance must increase sufficiently in that year. The annual growth requirements are set by each state, based on a formula that is intended to ensure that a majority of students in every group are passing the state tests by the year 2014.
- Value-added Progress Scores
A calculation of the growth, or progress, that each individual student makes on academic tests from year to year. The progress of each student (as opposed to a one-time "snapshot" score or the progress of groups) is what is used to assess the effectiveness of a school or district.
Value-added scores summarize the growth and progress of individual students in a way that permits making inferences about the teacher, school, and district work related to the students' progress. In this way, the "value" part of the "value-added" terminology is referring to the student growth that is due to the "value" that the school adds—how much value did the student get from attending the school? It is a specialized version of more simplistic "growth" or "gain score" models which only look at the difference in scores over time to judge student growth. Such simple growth models do not try to specify why the growth did (or did not) occur.
Value-added analysis is designed to highlight the support from the school for individual student achievement and growth. Because value-added results use the progress of individual students as the basis for all of the results for whatever level is being considered (e.g., grade level, school, or district), we can better understand the actual individual progress that schools and teachers are helping students to achieve. Unlike AYP, value-added results are not dependent on special groupings, and because value-added calculations follow individual kids, the grouped results that are obtained are potentially more reliable. Unlike the Performance Indicators, value-added results use multiple tests over time. This permits us to extend our view from a "snapshot" to a "movie" about the progress that students are making. Such a "movie of progress" is extremely valuable in allowing us to credit the work of teachers and schools where good work might otherwise be missed. Consider, for example, a classroom of students that are well behind grade level when beginning the year with an outstanding teacher. The teacher brings the kids up a full grade and a half—well beyond what might have been expected. If one only considers the end-of-year Performance Indicators, then this classroom of kids is still performing below proficient; but by understanding their growth through value-added analysis we can also learn that the teacher did an outstanding job in spite of the final test results.
There are two ways in which value-added progress scores will be calculated in Ohio. These will be discussed and compared in more detail in later modules. In both cases, the growth over time for an individual student is estimated by looking at multiple test score results. These results can then be aggregated, or summarized, for any grouping of students that might be of interest—grade levels, whole schools, districts, or special sub-groups of students (for example, previously low-achieving or gifted students). All individual student progress scores (for example, growth from last year's math test to this year's math test for all the kids tested in the school) are brought together to get a "big picture" of the effectiveness of the grade level group, the school or the district. These scores are also adjusted statistically in order to estimate how much of that growth and learning was due to the work of the teachers, school, and district in supporting students.
- Value-Added Expectations Related to State and District Work
- K-12
Legislation House Bill 3 Sec. 3302.021 (A) states: "Not earlier than July 1, 2005, and not later than July 1, 2007, the Department of Education shall implement a value-added progress dimension for school districts and buildings and shall incorporate the value-added progress dimension into the report cards and performance ratings issued for districts and buildings under section 3302.03 of the Revised Code.
The state board of education shall adopt rules, pursuant to Chapter 119, of the Revised Code, for the implementation of the value-added progress dimension. In adopting rules, the state board shall consult with the Ohio accountability task force established under division (D) of this section.
Such a task force was formed and created the following schedule for implementation for K-12 schools:
Value Added will be gradually implemented within K-12 schools. The following is a draft plan to be followed by the Ohio Department of Education:
DRAFT Ohio Value-Added Scale-up Plan
Analysis
(2003 - 2004) | SOAR III
2004-2005 | SOAR IV
2005-2006 | ODE 1
2006-2007 | ODE 2
2007-2008 | ODE 3 | | Districts | 80 | 100 | 612 | 612 | 612 | | 3rd | R M Sc SS | R M Sc SS | | | | | 4th | R M Sc SS | R M Sc SS | R M | R M | R M | | 5th | R M Sc SS | R M Sc SS | | R M | R M | | 6th | R M Sc SS | R M Sc SS | | R M | R M | | 7th | R M Sc SS | R M Sc SS | | R M | R M | | 8th | R M Sc SS | R M Sc SS | | R M | R M |
R = Reading Sc = Science M = Math SS = Social Studies
Yellow Highlight = Value-added is part of the state’s accountability system
How will the addition of a value-added result impact report card ratings?
If a district or building demonstrates two consecutive years "above" expected progress, their report card rating will improve. In the case where a district or building which was already designated "Excellent" demonstrates two consecutive years of "above expected" value-added progress, the rating will be revised to "Excellent with distinction." If a district or building demonstrates three consecutive years of "below" expected progress, their report card rating will be lowered. The exception to this is for buildings and districts already in "academic emergency" designation.
The purpose of the initiative is not to be an evaluation for the individual teacher but rather to serve as a means to monitor the continuous growth for each student, regardless of their achievement level in comparison to grade level norms (e.g., a 3rd grader who tested at the 2nd grade level should then be expected to test at a 3rd grade level or greater the following year).
- Higher Education
In January 2006, House Bill 107 was signed to amend section 3319.23 of the Revised Code to require the State Board of Education to adopt standards that require the curricula of teacher preparation programs to be aligned with the value-added progress dimension developed by the Department of Education.
House Bill 107 states: "Within one hundred eighty days after the Department of Education implements the value-added progress dimension, as required under section 3302.021 of the Revised Code, the curricula, including methods of interpreting data, are aligned with that value-added progress dimension."
In response to this legislation, two initiatives occurred:
(1) The Ohio Board of Regents initiated a task force from the State University Education Deans (SUED), with representatives from private colleges, Battelle for Kids, and the Ohio Resource Center, to create on-line modules for institutions of higher education (IHEs) to use within their teacher preparation programs; and,
(2) The Ohio Department of Education initiated a task force to work on policy implementation, thus the compliance mechanisms IHEs would utilize to gain approval for the inclusion of value-added within their teacher preparation programs.
- D³A2 - Data Driven Decisions for Academic Achievement
Data Driven Decisions for Academic Achievement (D³A2) is an Ohio Department of Education long-term initiative to provide a systematic approach for Ohio educators to access and analyze valuable data.
Begun in November 2005, it is a three-year $5.7 million dollar grant to improve data exchange. Funds are being used to develop the project as well as train educators in the use of data for decision-making.
- Value-Added Basics
In this section of the Module 1 resources, we will explore some of the fundamental concepts behind value-added measures, and how they connect with—and sometimes change—traditional perspectives on teaching, curriculum, and assessment.
We’ll first discuss some the impacts of shifting from the traditions of an “achievement status” measure to considering students’ growth and progress in learning. Some of the implications for how students, teachers, and school leaders work are obvious, and some are more subtle.
We will then discuss the sources of data used for value-added analysis—standardized tests—and how these test results are used differently to provide a growth perspective on student learning. (In Module 2, we will take the connection to testing a step further, and look at how teachers can relate value-added measures and standardized tests to their day-to-day instruction and assessments with students in the classroom.)
In our final section of Module 1, we’ll look at ways in which value-added information and reports are likely to affect various students, teachers, and schools, both from the “internal perspective” of teachers and school leaders working to improve their efforts with students, and from the “external perspective” of the impact of public reporting of value-added results.
- Changing from achievement status to growth and progress.
Since very early in our traditions of education in the United States, schools have been asked to provide as many children as possible with the opportunity to learn important knowledge and skills, in a system that would be as efficient as possible. This system has worked very well for some students, and less well for others. In addition to providing the opportunity to learn (with the results of taking advantage of that opportunity largely up to the students), schools have also served to rank order the results of learning and teaching. Arguably, the results of participating in education, taken across the entire population of students, has been less important than ensuring access to education, and then understanding which students successfully navigated the educational experience. Some learned a lot and went on to higher education; some were not as successful; still others dropped out of schooling entirely. Whether or not this result was intentional or acceptable in the past, there is little doubt that we cannot afford this result today. The advancement of knowledge, technology, and skill-levels needed for what used to be basic jobs, combined with global competition and the fact that the rest of the world is working diligently to catch up with the educational advantage which we have traditionally enjoyed, all combine to radically change expectations held for schools. While we still need to understand achievement status—where students are at a given point in time—it has become more important for schools who wish to successfully support all students to begin to look at individual students’ growth and progress.
This is no trivial change. Not only does a move from “status” to “growth” bring many challenges to schools regarding accurate measurement of students’ growth (hence the value-added approaches), but the expectations for day-to-day work in schools change when we begin to think “progress” instead of just “achievement.” As Dennis Palmer Wolf of the Annenberg Institute puts it, we are shifting from understanding students’ “intellectual address”—an address from which the student has already moved by the time we get the information—to “signposts on a journey” of learning. The status measure implies some fixed result of past efforts, over which current teachers have little influence, whereas the progress measure implies understanding the past in order for teachers and students to work together to impact current work and future results. This journey also requires a map, and thus presents expectations for teachers, school leaders, and the community at large to come to some agreements about what is important for students to learn—standards for both content and quality of work.
Finally, when progress is measured over time, it becomes possible to better understand how work that students and teachers do together in schools is contributing to learning. This is in marked contrast to status measures which, because they are only one point in time, make it impossible to decide how much learning is due to education and how much is due to factors beyond the control of the school. This aspect of understanding growth over time also requires a subtle but extremely important change in the way that students, teachers, and school leaders look at test results—the goal is to change what is being measured, not to just passively receive results after the fact. Students and the teachers and leaders who support them in their learning journey must become proactive and find ways to understand progress as it is happening. If they wait until the “final exam,” they will have missed their window of opportunity to impact the result.
Suggested Activities:
- Review and discuss the “scatterplot” charts that show the strong relationship of student demographic variables with traditional status achievement (one-time) test results. How can looking at growth change teachers’ understandings of student results?
- Review the “General FAQs” document and consider these answers to “Frequently Asked Questions.” What other questions do you have about value-added and/or status measures? What do you want to focus on as you learn more about connecting value-added measures with classroom teaching?
- Tests and how they are used in value-added.
Before reviewing the Powerpoint presentations included in this section of the Module 1 resources, we want to present some general background on standardized testing and the concept of testing in general. These features of tests and why we test students will provide a framework for viewing the Understanding Value-Added, Understanding Value-Added Calculations, and Value-Added Primer presentations.
What makes a test or assessment “standardized?”
A number of things must be present in order for an assessment activity to qualify as “standardized.” The actual type of activity itself does not matter very much—a standardized assessment can be in the form of a multiple choice “bubble sheet,” an open-ended essay, or even a creative dance. What matters is the context of the development, administration, and scoring of the assessment activity. These include:
- Ensuring that the activity is free of bias, both in content and in what is required of students who must complete the assessment.
- Having a common (and commonly understood, by those involved) set of rules for administering the assessment so that all students are participating in an equivalent fashion—e.g., students receive the same instructions, the same amount of time, the same materials.
- Creating a clear and specific set of rules for how student performance on the assessment will be interpreted and scored.
What is an assessment actually assessing?
While we tend to focus on the discrete “correct answers” that make up a test, when we create an assessment we are actually measuring some selected portion of a much larger intangible concept that we REALLY want to know about – “student learning.” So to ask a few selected questions about mathematics, for example, will give us some sense of whether the student “understands and can do” mathematics. It follows that asking more questions, with more variety, with different applications, and over time will give us a better, more well rounded view of the invisible underlying “truth” about a student’s understanding of mathematics. When one thinks of assessment in this way, then any assessment scenario, formal or informal, is really an opportunity for teacher and student together to uncover the student’s learning progress. This learning is affected by many things – not just ONE teacher, but all of them, as well as the full school and home experience. The goal is to understand how any given test connects with the “big picture” of student learning that is behind the scenes, so that expectations for the next stages in the teaching and learning that a student needs can be identified. When this perspective on understanding student learning is combined with a good standards-based “domain” of learning goals and appropriately designed assessments that help uncover progress towards these goals, then we have an environment that works for our value-added progress measures.
Can standardized tests really measure growth for all kids?
Yes, within some limits. Tests that work well for value-added analysis are designed in such a way that they have “curricular stretch,” meaning that they test not only the specific knowledge and skills representative of a particular “point” in a curriculum—for example, a specific grade level—but also some of the skills and knowledge a bit above and below the standard. In addition, when tests are “vertically aligned” with a curriculum that is based on standards and an appropriate progression of student learning from early knowledge and skills content to more advanced, the whole group of tests grade-level to grade-level gain additional “stretch” in capacity to measure student growth. Within a fairly broad range, even students who are very gifted or very far behind the standard can be assessed adequately for the purposes of value-added analysis.
Suggested Activities:
- Review some or all of the Powerpoint presentations (as desired or assigned), and then take a few minutes to review the “Key Points” document from Battelle for Kids. Work through the “Checking for Understanding” questions with colleagues or classmates.
- Consider again the 6th slide in the “Understanding Value-Added” Powerpoint—titled “A View of Student Growth,” below. Discuss the implications for schools of realizing that Student B is on a decreasing learning trajectory. What are the implications for schools in realizing that while Student A is moving upward, he or she is still unlikely to meet the standard?
- Application - what this means for teachers and schools.
As we noted above, it is most important for teachers, school leaders, and students to approach value-added measures as another useful tool in the toolbox. Test results, whether status or growth and progress measures, are most useful in identifying areas where students have either mastered an identified learning goal, or where they are having difficulty. What to do NEXT is something that teachers and students must address together. No test result that is divorced from additional information about the teaching and learning that are taking place in the actual classroom can be very helpful to student learning, beyond simply identifying “good” or “not so good” schools for accountability reporting. Tests can show where there are problems or successes, and value-added methods can add another dimension to the picture of student learning. However, teachers, students, and school leaders must “own” these data results, and must make use of them alongside other assessments and evaluations of programs and school work to be effective. The people in the school and the classrooms are the only ones with good access to the real story of what is really going on with student learning.
Value-added progress measures are all about supporting student learning over time. By helping focus test results on each individual student and the growth that each student can and should make from wherever he or she started, we are helping to focus our attention on what really matters—student learning.
Suggested Activity:
- Review the “Gifted_FAQs” and “Implementation Stories” documents from Battelle for Kids. What ideas can you and your colleagues brainstorm related to making proactive use of assessment data, rather than just waiting for the results?
- Conclusions and preparation for Module 2.
A final important note before moving into more hands-on work with value-added results, assessments, and working together in the school: value-added measures are NOT being recommended for use in high-stakes decisions about teacher or school performance. This is in large part because of the underlying concepts as described above: value-added approaches are intended to focus on improving learning, not ranking and judging. But it is also because there are a number of value-added models in use, and because none have been in practice long enough for them to be considered “absolute” in accuracy. Our recommendation is that policy leaders, community members and parents, and most importantly, school leaders, teachers, and students consider value-added alongside the array of other information available about schools and student learning. These include status measures and AYP results, but also formative assessments, benchmark assessments, and informal classroom activities designed to help teachers differentiate instruction to meet student needs. Teachers and school leaders in various positions will work with value-added results in different ways, and in combination with many other sources of data to make decisions about curriculum, instruction, school policies, and communicating with parents and the community. Module 2 will focus in greater detail on how school personnel work with value-added measures.
Suggested Activities:
- Two documents, “Checking for Understanding” and “Introduction to Value-Added Group Activities” are included to give you some options and ideas for reviewing the ideas in this Module’s materials.
|
|