\doc\web\99\17\itbsnote.txt


Reviewing the ITBS
Arthur Hu arthurhu@halcyon.com http://www.leconsulting.com/arthurhu/washtest
.htm
Dec 23, 1999


1. ITBS
2. Catherine Taylor's WASL defence


1.


I visited the Lake Washington School District testing office to take
a look at the ITBS test (Iowa Test of Basic Skills) to figure out why
my 2nd grader "only" got 75th percentile when he's good for 95 in
reading and he seems to know everything there is in the 2nd grade
math textbook I have at home. It is a high security test, I could not
write down notes, but some things did strike me.


This is what I noticed:


- Money problems requires discriminating between the unmarked side of
black and white coin photographs distinguished only by the heads of

Washington (25), Lincoln (1), and whoever is on the side of the
nickel or dime (can somebody help me, who ARE these guys??) US money
is NOT marked as to value on either side in numbers.


- Nothing more difficult than addition or subtraction, the end of 2nd
grade has some regrouping (borrow, carry) required. Some problems
also use the term half of or doubling, which is technically
multiplication or division that can be solved by counting or
duplicating techniques.


- I did not see any probability problems, but there were charts.


- Most problems do not give any instructions on the page, verbal instruction
s
require the recognition of symbols such as "find the problem with the
feather" or the key or the house.


- Very few problems are as simple as 2 + 11 = [  ]. Most are story problems.




- Most story problems are given verbally, and only repeated twice. If
I have five cats and 2 run away, how many are left. If you do not
understand or did not write down the information quickly enough, you
will not be able to set up the equation.


- Although Asian and Hispanic minorities evidently do OK on the ITBS
relative to the WASL, I suspect non-English speaking kids would do
MUCH better if given a sheet of math equations to solve instead of
this test which appears to be 80% story problems of one kind or
another. It is especially ironic that given the amount of space that
the Dale Seymour series spends explaining how it is important to be
inclusive of non-native English speakers that the very non-English
speakers who are right a home with 12 + 33 = ? are up a creek without
a paddle when confronted with Jack and Jill had a bake sale, they
baked 120 cookies, but broke one of every four cookies, what is the
probability they will make a profit if they sell 6 cookies per hour
if Jill sells twice as many cookies as Jack, and Jack sells as many
as his age, 2 years ago he was 6?


- Another type of problem is "what number is closest to 27" 20, 30,
28 you'd have to turn this into 3 subtraction problems to solve, it
is not direct application of subtraction traditionally taught, so
even the multiple choice problems have been transformed into "problem
solving".


- Some problems also include "N" for none of the above, this may
confuse some kids.



- Graph problems require reading bar charts, pie charts, and doing
math on quantities read from the chart. For example, if a pie chart
has numbers of cats, dogs, and canaries, which animal is only half as
many as the canaries?


- According to the conversion charts, if you get only 2 problems
wrong in one section, that's good enough to take you down from 99 to
75th percentile, so these tests aren't very accurate above the 70th
percentile.


- There is no proficiency scale, only a percentile ranking. I suspect
that with a test like this, even students at the 10th percentile can
peform addition and subtraction and some basic level, but it is easy
for educrats to label any student that cannot ace this test (which is
evidently most kids) as a failure. I suspect this is a major problem in
education that no one wants to accept a 15th percentile or D- average
as acceptable, and everybody wants to be a whiz kid.




2. I also got a copy of a handout by Catherine Taylor of the
University of Washington entitled "Evidence for the Validity and
Reliability of the Washington Assessment of Student Learning".
presented at the Washington State Assessment Conference Dec 9, 1999.
If there is interest, I will post this as an efax document where
everybody can look at it on my web site.


Patty Henson Crawford says that she was invited to attend this
meeting, where some other professors from Washington State University
claimed that Don Orlich's conclusion that the G4 WASL math was
developmentally inappropriate was based on flawed assumptions.  The
document gives the old spiel about how the test has multiple checks
and safeguards so that it is impossible to produce a test that is bad
after all the commitees, etc.


Of course it doesn't say a thing that this Arthur Hu guy noticed that
area = height x width is clearly G7 on the EALRs, but in the G4 test,
and it gets worse from there.


It gives "high correlations" between the CTBS and WASL math scores,
(.60-.80) and judge agreement of 79-96% exact agreement and 95-99%
within 1 point. (Remember on a 5 point scale, exact agreement is off
by as much as 20%, or 40% even if you are "adjacent" off by 1 point,
is being off by 40% 5% of the time acceptable??) This appears to be
along the lines as every other state that says their test is valid
even though any one with any sense knows that any test with a 50% to
98% flunk rate designed to be a _minimal" standard is invalid no
matter how many of these numbers they can come up with.






From: "George K. Cunningham"


The ITBS is not really very secure.  A friend of mine, an involved parent,
wanted a copy of it.  Riverside was happy to send her a complete set of
items in the form of an examination copy.  The only caveat was they had to
send it to a university office.  She asked me if I would oblige which I did.


The Riverside people understood what she was doing and it was fine with
them.


You also point out a problem endemic to all standardized achievement tests.
These companies have but one group loyalty and that is to their share
holders. They want to keep the buyers happy and these tests are constantly
hammered because they supposedly measure recall of facts and do not measure
high level thinking skills.  They overcompensate by including indirect
measures of basic skills turning the task into something closer to an
intelligence test.  This also makes it more difficult to prepare students
for the tests.  Knowing basic math skills is not enough, but isn't that the
whole point of why the tests are given.


George K. Cunningham
University of Louisville