Constructing Standards-Based Assessment Questions

by Carol Brown Dodson

As an Ohio educator, you probably wonder at times how the questions for the Ohio Graduation Test are developed. You've likely written many such questions for your classes, but like most teachers, you probably struggle to find four plausible choices for multiple-choice questions. You may find, after students respond to the questions on your tests, that they totally misunderstood at least one of the questions you asked. Other times, you might find that a correct response to a question doesn't necessarily mean that a student has achieved mastery of a benchmark or grade-level indicator.
 

How do you develop questions that indicate whether or not students achieve a benchmark?

Each year, following the spring administration of the OGT, the Ohio Department of Education posts the test on its website. The reading OGT , including passages and questions as well as a scoring guide and results, is available as a PDF file from the department. A close review of previous OGT questions is useful. Learning to write standards-based assessment questions, however, requires more than just looking at questions. By deconstructing test questions, we can begin to understand how the questions are constructed.

Let's look at some of the passages and questions from the Spring 2006 Ohio Graduation Test. The nonfiction passage "Wheelchair Flying" by Carrie Dearborn provides an opportunity for measuring students' achievement in reaching the goals set by the informational text standard. Students are required to read the paragraph quoted below:

Wheelchair flying takes place on an asphalt path around a pond. I quite startle people as I zoom by them at full-out 7 m.p.h. Part of the high of this sport is that people get a new slant on wheelchairs and wheelchair users. One time, a little kid pointed at my chair and said, "Look, Mom, can you get me one?"
    Ohio Graduation Test, March 2006, Ohio Department of Education

Students are expected to refer to the paragraph to answer a question that assesses them on informational text Benchmark B. The benchmark reads this way:

Identify examples of rhetorical devices and valid and invalid inferences, and explain how authors use these devices to achieve their purposes and reach their intended audiences.
    Ohio Academic Content Standards, English Language Arts, Ohio Department of Education

The question (below) requires students to infer meaning and to understand why the author uses a quote from a small boy.

4.   Dearborn includes in the passage the quotation from the little boy in order to show

1. that people of all ages can wheelchair fly.
2. how people's attitudes can change.
3. that people can sometimes be rude or impolite.
4. how dangerous people think wheelchair flying can be.
       Ohio Graduation Test, March 2006, Ohio Department of Education

Answer B was correctly chosen by 72 percent of the students who responded to this question. They understood that the boy's plea for a "flying wheelchair" suggests a different view of wheelchair users than is commonly held by children. The boy might not have said this if he had watched someone going by slowly in an electric wheelchair, but seeing the author fly by at about 7 miles per hour changed his attitude from what might have revealed negative feelings toward a disability to a desire to have the same experience that the disabled person has.

Let's look at the stem for the above question.

Dearborn includes in the passage the quotation from the little boy in order to show...

Notice that the question requires students to think not only about the meaning of the quotation, but also about the author's purpose for including the quotation from the little boy. As a result, students must look beyond the text itself and consider why a writer includes certain information and details. That addition to the question forces it into a higher level of questioning than it might have reached if the question simply asked why the boy wanted a wheelchair. But even more importantly, the question assesses the benchmark expectation that students should be able to explain how authors use rhetorical devices and inferences to achieve their purpose.

View the results

Another consideration in constructing standards-based assessment questions is to recognize the kind of information that is available about students who select the incorrect answer. Ohio's tenth grade students generally do extremely well when answering questions that test reading process, but only 48 percent of the students responded with the correct answer for the question given below. After reading the passage "The Grandfather" by Gary Soto, students were asked to select the correct answer to the following question:

33.  Why did the grandfather view the avocado tree as a symbol of hope and promise?

1. For the grandfather, the avocado tree is a reminder of his prosperous life in Mexico.
2. The grandfather had begun growing the avocado tree from a small seed in a jam jar.
3. The avocado tree needs more love and care than the lemon, orange and apricot trees.
4. For the grandfather, life is like an avocado tree, which bears fruit slowly and patiently.
       Ohio Graduation Test, March 2006, Ohio Department of Education

The above question tests students on their achievement of the reading process standard, Benchmark B:

Demonstrate comprehension of print and electronic text by responding to questions (e.g., literal, inferential, evaluative and synthesizing).
    Ohio Academic Content Standards, English Language Arts, Ohio Department of Education

Literal-level questions are easy for the majority of students, but when the question causes students to infer, evaluate, or synthesize to get the correct answer, the students frequently select the incorrect answer. In the case of this question about how the grandfather viewed the avocado tree, students had to infer what the text merely implied. Many tenth graders selected the wrong response, with a full 33 percent of them choosing response B, "The grandfather had begun growing the avocado tree from a small seed in a jam jar."

Notice that it's not just the response, but the overall development of both the stem and the possible answers, that leads to information about students' ability to respond to inferential questions. The benchmark drives the question and is always the focus of the question.

Some questions cause students to read critically and fully

Another item that assesses students on Benchmark B in the reading process standard is the following short-answer question about the "Wheelchair Flying" passage.

In your Answer Document, explain what the author means when she says, "Wheelchair flying gives me the freedom to, well, stretch my 'legs,'" and give a detail or example from the passage to support your idea. Write your answer in the Answer Document. (2 points)
    Ohio Graduation Test, March 2006, Ohio Department of Education

To receive 2 points for their answer to the question, students had to provide an explanation and support the explanation by giving a single detail or example from the passage. Yet only 39 percent of the students responding to the question received 2 points. Let's look at some student responses to gain insight into the question and its intent. The following answer received a score of 2.

According to the scoring guide:

The response provides a reasonable explanation for what the author means by the expression (it gives her a chance to do something challenging ... to make it seem that she can do as much as someone who isn't in a wheelchair) and supports this example with an example from the text (Now I indulge in a sport few people have tried―flooring my electric wheelchair).
    Ohio Graduation Test, March 2006, Ohio Department of Education

Another student received a score of 0 because this student failed to look at the expression beyond the literal meaning. Notice the answer given by the student.

This time the scoring guide states that "interpreting the author's legs as her wheelchair is a literal understanding of the expression and does not adequately describe the author's intended meaning." The use of the phrase "intended meaning" rather than a literal understanding is an important distinction being made by the question. The student is not being tested on getting the exact meaning of the words, but rather is being tested on what is being conveyed by the entire phrase. The student must interpret or infer the meaning based on an understanding of the overall passage.

How can I tell if they've achieved the benchmark?

An extended-response question for "The Grandfather" assesses students on Benchmark E of the literary text standard. Here is Benchmark E:

Analyze the use of a genre to express a theme or topic.
    Ohio Academic Content Standards, English Language Arts, Ohio Department of Education

The question requires students to have an understanding of theme and its relationship to the structure of the story. Notice that the question does not ask students to identify the theme. Instead, it asks them to explain its development and to support their answer with specific details or examples from the story.

35. Explain how the concept of growth is developed in the story. Use three details or examples from the story to support your answer in the Answer Document. (4 points)
    Ohio Graduation Test, March 2006, Ohio Department of Education

A review of the percentage of students at each score point reveals that a majority of students have not yet achieved the benchmark. A mere 28 percent of students who responded to the question received a score of 4.

In order to receive a score of 4, students had to give a plausible explanation of the theme's development and then provide three details or examples to support the explanation. The following answer earned 4 points.

According to the detailed description of the above response in the scoring guide:

The response provides a plausible explanation (the tree grew over the years as the boy did and as the grandfather grew older) and supports it with three details from the passage (the boy was able to jump over it at first, then it was as tall as he was, finally ... all the men on top of each other weren't as tall).
    Ohio Graduation Test, March 2006, Ohio Department of Education

The student who wrote this response seems to have an understanding of theme and a grasp of how an author uses details to develop the theme. The responses to the 4-point question reveal clearly whether or not students have achieved the benchmark.

What are some elements of questions that assess benchmarks?

The questions that effectively assess benchmarks are stated in such a way that students must go beyond simple recall of details and information. Students are asked to explain why authors use some details or rhetorical devices. The same is true when students are tested on literary elements and techniques.

By following some of these same patterns for test development in your classroom, you can determine how well your students are doing in meeting standards and benchmarks. You've seen a few examples here, but be sure to go to the Ohio Department of Education's website, where you can deconstruct other OGT questions and analyze students' responses to them.

The Ohio Resource Center collection includes released OGT items which can be accessed by browsing Ohio Standards or by conducting an advanced search for assessment items, grade 10, reading. Also included in the ORC collection are questions from the National Assessment of Educational Progress (NAEP). The NAEP questions are extremely useful in assessing benchmarks, whether the questions are used as they were written or as models of good standards-based assessment questions.

Some other ORC resources may be used to help you as you develop standards-based questions and rubrics.

Authentic Assessment Toolbox

The Authentic Assessment Toolbox is a how-to hypertext about creating authentic tasks, rubrics, and standards for measuring and improving student learning. Author Jon Mueller includes detailed, practical application chapters: (1) What is it? (2) Why do it? (3) How do you do it? (4) Standards, (5) Tasks, (6) Rubrics, (7) Portfolios, (8) Examples, (9) Glossary, and (10) Workshops. The author discusses rubrics, both analytic and holistic, and portfolio assessment in detail at this professional development site, and examples draw from all areas of the curriculum. (author/bcbrown)
 

Assessing to Inform Instruction

The March/April 2005 issue of Adolescent Literacy In Perspective, "Assessing to Inform Instruction," focuses on assessing literacy in the middle and high school classroom. In the feature article, "Living with Hope, Rather Than Doubt: Classroom Assessment of Adolescent Literacy Learning," author Gwynne Ellen Ash poses and then answers specific questions about literacy assessment: (1) Classroom assessment of literacy learning: What do I want to measure? (2) How do I assess these elements and still have time to teach? (3) Students and teachers as partners, and (4) Assessment stress: Remembering your purpose. Additionally, two classroom vignettes support the practical application of assessment. The first vignette, "'Will This Be on the Test?' Literacy and Assessment in High School English," provides author Colleen Ruggieri's insights and strategies for assessment: Positive Results with Pre-Assessment, K-W-L-R Charts, Anticipation Guides, Making Meaning by Marking the Text, Double-Entry Diaries, and From Study Guides to Comprehension Construction. The second vignette, by authors Darla Wagner, Joe Paris, Judy Mikita, Jay Falls, and Pat Tallman, looks at how to tell if our students are learning the material. The focus in this vignette concentrates on addressing and responding to questions the authors deem significant in literacy assessment: (1) Are students aware of the daily learning goal? (2) Do students understand the connection between the class activities and assignments and the learning goal? and (3) When students leave the room, how do we know which ones "got it"? This month's "Look at the OGT" focuses on improving students' writing, and author Carol Brown Dodson looks at specific ways to improve students' writing through "Writing Suggestions." (author/bcbrown)

Effective Reader Scoring Guide

Developed by educators at the Northwest Regional Education Lab (NWREL), this reading assessment provides a thorough, yet practical, way to evaluate students' oral and written responses to texts. Understanding of the text is assessed based on six observable traits of effective readers: conventions, comprehension, context, interpretation, synthesis, and evaluation. The scoring guide may be used to outline strengths and weaknesses of students and to set clear performance goals. For additional information about this scoring guide, refer to The Traits of an Effective Reader. (author/ncl)

Carol Brown Dodson is the outreach specialist for the Ohio Resource Center. Dodson was an English language arts consultant for the Ohio Department of Education and is past president of OCTELA (Ohio Council of Teachers of English Language Arts). Dodson, formerly a high school English teacher, department chair, and supervisor of English language arts in Columbus Public Schools, serves on the Ohio Graduation Test Reading Content Committee.

Return to top