\doc\web\98\06\terrnov.txt
Teaching & Learning: Theory, Methods,
Technology, Content - Terra Nova Reviewed
Back to Teaching & Learning: Theory, Methods,
Technology, Content
Posted by cassidy@pacifier.com on 1998-06-16
22:21 (11590 bytes):
Evaluation of the California Test of Basic
Skills (CTBS) 5: Terra Nova 9th Grade Reading and Math
Dianne Cassidy May 2, 1998
After my son's experience with the Terra Nova
exam on Monday, April 13, (Gr. 9) I decided I should see the test for
myself. I went in to the school district office and was able to view
the actual 9th grade exam my son took. From my point of view, it is a
case of good news and bad news. Since I am not experienced in test
analysis, what follows is simply my impressions of what I saw. I was
able to take notes, but not copy any of the questions. I did a bit of
follow-up by posing some questions to our district’s director of
assessment, and Caleb Burns, Ph.D., who has been doing some
investigation of the validity of the Oregon State Assessments and the
Terra Nova tests.
The Terra Nova in Lake Oswego
The Terra Nova test was first introduced in
1997, and this is the second year the Lake Oswego School District has
administered this test to its students. All students at the 4th, 6th,
7th and 9th grades were asked to take part in this assessment.
Students in the 3rd, 5th, 8th and 10th grades were administered the
Oregon Statewide Assessments in Reading & Literature and Math.
McGraw-Hill created the Terra Nova (CTBS 5) in
1997 to update their standard test to reflect new educational trends
and to include problem-solving and thinking skills in their test
format. McGraw-Hill has created special parts within the Reading &
Literature and Math sections for its Oregon clients, the ESD
Consortium. The ESD Consortium consists of the Clackamas Co. ESD,
Multnomah Co. ESD, and the Northwest Regional Education Service
District. (Lake Oswego is in Clackamas Co.) These special sections
are specifically designed for Oregon educators to complement the
Oregon Statewide Assessments and are meant to be given at grades NOT
covered by the state assessments. The Consortium Off-Grade (COG)
tests are expected to yield information abut student progress in
relation to the Oregon content and performance standards and to a
national norm-referenced group.
Lake Oswego did not have students take this
special COG test this year as there was not sufficient time to do so.
Next year the 10th graders will be taking the Statewide Assessments
in order to qualify for the Certificate of Initial Mastery. Off-grade
students will be assessed with the Terra Nova to determine if they
are meeting the benchmarks established for their grade. Students who
score 1 standard deviation below the cut score will need remediation
to prepare them for the CIM by 10th grade. I did not look at or
review these Oregon-only sections.
The first technical manual for the Terra Nova
does not indicate comparisons to other achievement tests. Standard
practice for test creation and use is to have external validation of
results, and the Terra Nova does not meet this standard. Further, in
April, 1998, a representative of McGraw-Hill said that the publishers
have no intention of doing any such comparisons in the future.
McGraw-Hill also does not list the names and expertise of its test
creators.
Note: The Oregon State Assessments (Reading and
Literature, and Math, grades 3, 5, 8, and 9) also have no external
validation, and no published literature with the names and expertise
of its test creators as required for content validity. The fact that
the Terra Nova is correlating its test results with the Oregon State
Assessments is troubling to say the least. Two tests, neither of
which has followed the standard test-making procedures outlined in
Standards for Educational and Psychological Testing, are now being
used to judge the academic achievement of a large number of students
in Lake Oswego, and in the state of Oregon.
Test Format
The Terra Nova test is divided into 2 sections,
Reading & Literature, and Math. Each section has two parts, Part A
and Part B. For members of the Oregon ESD Consortium, a third part in
each section was also created (see above).
Reading & Literature Section: Grade 9
I found the Reading & Literature section to be
very poorly constructed. I really don’t know what the test makers
were trying to test, and the district director of assessment could
not explain either what this section of the test was trying to
measure nor how the questions were valid. Only about 40% of the test
appeared to have anything to do with reading comprehension or
literature analysis. The other 60% was a mishmash of charts, concept
maps, questions that involved prior knowledge of little-used terms or
cultural experiences, and questions that asked students to respond as
to what they thought (not based on information given). There was
little here to suggest that a high or low score on this section would
have any relevance to one’s academic or thinking abilities.
As a literate adult, I was hard-pressed to
answer some of these questions! I was often searching the text for
clues as to what the expected, correct answer was, but could find no
clue. I have never been instructed as to the use of concept maps, and
wonder if there is an accepted method that ALL students are now
expected to know. My own children had no idea what a question of this
type “should” look like unless specific instructions were given. What
about kids (and adults) who have never seen a concept map? Are they
determined to be illiterate? Are all students 9th grade now expected
to analyze and interpret federal legal codes when no other supporting
or explanatory information is given? Is filling out a form or reading
numbers on a chart now considered “reading and literature”? I have no
faith that any of the scores resulting from this test have a solid
relationship to reading or literature.
Types of questions:
· A sample question gave the definitions of the
term, “zone” as it is used in basketball. Given a diagram of two
teams playing basketball, students are asked to determine the intent
of the player with the ball. The possible answers use the terms,
“offensive” and “defensive.” Students would be handicapped if they
did not understand the rudiments of basketball and how the game is
played. This question might favor boys and certain cultural groups. ·
Given a reading passage, students were asked to interpret what a
character meant by a certain phrase. I could find no other clues as
to what the specific character meant in the given context. Students
are then expected to apply their own personal interpretation (not
deduce any from the reading) in answering the question. This type of
“tell us what you think, but we aren’t going to tell you what to
think” ploy was used in several questions in this section when
students were asked to determine motivation or reasoning where none
was implied or explicit within the reading. · Students were asked to
interpret a legal passage. The questions that were asked had no
reference to the passage given. · 2-3 sections of vocabulary fill-ins
were given. Students were asked to select definitions from a list
(4-5, multiple choice) of options. · A couple of reading sections
followed a traditional comprehension format (similar to the SAT I
verbal test). A reading selection was presented, and the questions
that followed asked students to extract factual information from the
text and to summarize the main idea. · Students were asked questions
relating to a given chart. · Students were given a reading selection
and asked to select the literary form (terms given) that it
represented. · Given a short reading passage (about 2-3 paragraphs),
students were asked to fill the empty bubbles in a concept map. No
specific instructions of how the information was organized was given.
· Students were asked to fill in an application to participate in s
“Buddy” program. (E.g.: What response belongs on line #5, Parent or
Guardian occupation?) · Students were asked to “read” a timeline and
answer questions associated with the time events took place.
Math Section: Grade 9
Overall I found the math test to representative
of what average 9th graders should know (having completed Algebra I).
My principle complaint is that the test is focused on a narrow range
of an ability level does not accommodate the lower end or more
advanced students, so that if there are a large number of these
students in a testing group, the results for a district would be
skewed toward the either end. Individual test scores are given as a
percentage of how a student compares to the national norms for the
grade tested. Individual scores for advanced students would not
reveal a grade-level achievement beyond the 9th, earning a 99%, and
low-achieving students would score at the lower ends of the scale.
Below are the types of questions asked in the
test. All questions were fairly basic in that very simple numbers
were used when asking students to determine processes or
computations. It was obvious that the test required the students to
know how to apply the correct process rather than do complicated
mathematical calculations. All problems were given as story problems
and/or referred to accompanying charts or graphs. The math section
had 40 questions total. Of the 40, I concluded that: 1 was a reading
comprehension question (not math at all), 1 had two correct answers
if the question was taken literally (one has to deduce what the
test-maker wanted to get a most correct answer), and 1 had no correct
answer (I will follow up on this).
Types of questions:
· measure a triangle, given a scale to convert
dimensions, determine its perimeter · addition of decimal numbers:
XX.XX + X.X = ? · addition of positive and negative integers ·
multiplication of fractions · find the square root of the addition of
2 squared numbers · convert a fraction to a decimal · understand the
meaning and use of the repeat bar above a set of numbers · how to set
up and figure the missing element in a proportion: X:X as X:? · read
and understand information given in a bar graph · determine and
compare relative values of a set of numbers (used in pricing) when
stated as: X for $X, X% OFF of $X, $X OFF of $X, X/X OFF of $X ·
definition of a geometric shape -- select a diagram that fits · when
should you estimate -- select from examples given · figure the
pattern of increase in a set of numbers and find the next in the
series · given similar shapes, determine length of a given side:
requires proportion · questions relating to rounding · questions
requiring estimation · given a geometric shape and a length of a
side, determine length of another side · determine equality of two
algebraic equations · must know different ways to express numbers in
fractions, decimals, percent, parts of.. · simple factors of a set of
numbers · given a table of X and Y values, determine the equation
used to create the table · probability · scientific notation · number
lines, line sections · exponential growth: determine pattern in a
sequence of numbers and figure what comes next (or later in the
series)
Resources
For a discussion of the Oregon State
Assessments, see: http://www.teleport.com/~calebb/cim.html
To see sample tests for the Oregon State
Assessments, see: http://www.ode.state.or.us/ Select Assessment, and
look for sample tests.
Post a Followup to This Message