z43\doc\web\2000\07\anghoff.txt


Carol,


The key decision that every state that is implementing SBER must make, after
setting standards, and assessing students is where to set the cut points
that designate passing or failing or place students or schools into the
various categories that are being used such as basic, proficient, or
advanced (NAEP categories).  The problem is that performance is continuous
and exists on a straight line and this makes the designation of cut-points
entirely arbitrary.  It as though you were trying to decide whether a person
was tall.  Is six foot the right cut point.  Would this mean that someone
five foot eleven was not tall?  The problem is that there is simply no good
way of doing this.  Of course if you are going to do standards-based testing
you have to have performance standards, which is what the setting of
cut-points is called.


The most widely used method of doing this is the Angoff  Method.   It is
used by many states and one form of it is used by the NAGB board in setting
the NAEP standards.  When you read about how 40 percent of fourth grade
students are "below basic" on the fourth grade NAEP reading assessment, it
is the Angoff method that was used to set the cut-points that led to that
statistic.  To implement this method, you have a group of judges examine
each item and estimate the probability that a minimally qualified student
would get it correct.  The probabilities are pooled to set the cut-point.


Even though this is the most widely accepted approach it has been widely
criticized.  It is thoroughly subjective and different judges and/or groups
of judges will come up with wildly different cut-points.  In its purest form
the judges are given no information about actual student performance.  As a
result, they often come up with cut-points that make no sense.  For example
they might result in virtually every student failing.    Then we have the
Modified Angoff Method which permits information about actual student
performance to be given to the judges.  The more of this information that is
given, the more the process begins to resemble norm-referenced testing.
This is ironic because the purpose of  SBER is to avoid norm-referenced
comparisons.


A good source of information about methods of setting cut-scores and NAEP
can be found in a book about NAEP published by the National Research
Council, titled _Grading the Nations Report Card_.  The authors are highly
critical of the use of the Angoff method, and when it comes time for them to
describing what they would use to replace it, they come pretty close to
advocating   norm-referenced testing.


What states really do is determine the percentage of students that they want
to pass and then use obscure statistical methods like Angoff or other
similar methodologies to come up with a cut-score.  If they don't get the
results they want they use a different method or they "modify" the methods
until they get what they want.  The most pathetic states are those that
stumble around with standard setting methods that they don't understand and
get cut-points that are absurd and stick with them out of some misguided
belief that the cut-points were somehow scientific and can't be changed.


Whatever you want to say about Texas, they didn't fall into that trap.  They
wanted everyone to pass, in particular minority students, and they have been
highly successful.  This is easily accomplished if you set your cut-points
low enough.


George K. Cunningham
University of Louisville


> -----Original Message-----
> From: Assessment Reform Network Mailing List
> [mailto:ARN-L@LISTS.CUA.EDU]On Behalf Of kceh
> Sent: Saturday, July 01, 2000 9:53 AM
> To: ARN-L@LISTS.CUA.EDU
> Subject: Re: exit tests are a scam
>
>
> What is the Angoff method and where can I read about it? I have
> thousands of pages of public record documents from Harcourt and NCS. It
> would be nice to be able to decifer all of the information.
>
> Most kids don't have too much trouble with the TAAS test itself because

> it's not that difficult (not the ones I've looked at, anyway).  The
> problem here is that so much test prep is crammed down their throats and
> there is so much pressure to pass it.
>
> Carol


--------------------------------------------------------------------------
To unsubscribe from the ARN-L list, send command SIGNOFF ARN-L
to LISTSERV@LISTS.CUA.EDU.




> > Every state is using exactly the same procedure to create "standards",
> > then set standards based on bogus tests using a committee to "bookmark"
> > where they think "everybody should know and be able to do". That's why
> > every state has a test that flunks from 60% to 80% of students,
>
> Except of course Texas.  They use the modified Angoff method for setting
> cut-scores.  In fact it is modified enough to make it look a lot like
> norm-referenced testing.  That makes it easy to set the cut-point so that
> there are few failures.  Bizarrely they call  there test
> "criterion-referenced,"  I suppose because that sounds better.  Their test
> is far from that.
>
> George Cunningham