z63\doc\web\2003\01\bench1.txt
George Cunningham U Louisville:
"items are placed in order of difficulty.  Judges then determine
where the cut-points should placed.  Generally, the judges are give
impact data (in California this is called the empirical approach),
which allows the judges to take into account the ratio of passes to
failures associated with a given cut-score. [but not in Washington]
The judge's scores are combined using methods of achieving consensus
or they are averaged in some way."

In the washington state process, judges were not shown what
percentage were answered correctly, so the level could be set even if
0% answered correctly. Also judges were not given the opportunity to
check the problem against the standards, thus, they could and did
approve problems which were not compliant with grade level standards.

-----Original Message-----
William Cala
Sent: Tuesday, January 21, 2003 5:54 PM
To: arn-l@interversity.org
Subject: Re: [arn-l] "cut scores"


George,

If interested, please read my analysis of  what New York did to the
bookmarking process in the case of Regents' Physics.  Not a pretty picture.

----- Original Message -----
From: "George K Cunningham" 
To: <arn-l@interversity.org
Sent: Tuesday, January 21, 2003 7:19 PM
Subject: Re: [arn-l] "cut scores"


 Lee,

 McGraw-Hill developed this standard setting method and prefer it be
 called CTB Bookmarking because they consider it proprietary.  Since
 Indiana is a McGraw-Hill state, it makes sense that they would use this
 method.  Most of the traditional standard setting  methods are based on
 the use of absolute standards.  The Angoff method is the most often used
 of these.  The CTB Bookmarking method is different in that it is a
 norm-referenced method of standard setting.  It also has the advantage
 of being able to include both multiple-choice and constructed response
 items.

 This is not a complex method although it has its complexities.  Using
 the 3-parameter item response theory favored by McGraw-Hill, items are
 placed in order of difficulty.  If constructed response items are placed
 into one of four categories (novice, apprentice, proficient, or
 distinguished as is done in Kentucky), the categories are interspersed
 among the multiple-choice items according to difficulties.  Judges then
 determine where the cut-points should placed.  Generally, the judges are
 give impact data (in California this is called the empirical approach),
 which allows the judges to take into account the ratio of passes to
 failures associated with a given cut-score.  The judge's scores are
 combined using methods of achieving consensus or they are averaged in
 some way.

 George K. Cunningham
 University of Louisville

  lheffern@indiana.edu 01/21/03 07:00PM 
 ARN'ers,
     I plan to respond to an article in the paper today regarding the
 re-defining of "cut scores" for ISTEP (our hst in Indiana).  The brief
 article mentioned the "Bookmark
 Standard-Setting Procedure" that has been  used by "28 states to set
 passing
 levels on student assessments."  Does anyone on the list have info
 about
 what's involved in this procedure?
 Thanks,
 Lee

 ------------------------------------------------
 Direct list questions to listmom@interversity.net