Date sent:        Sun, 8 Feb 1998 18:59:16 -0500
From:             "Richard G. Innes" <70224.434@compuserve.com>
Subject:          RE:  Portfolio Assessment
To:               ed-consumers main listing <education-consumers@tricon.net>


Dan, 


I live in Kentucky, which has had a high stakes portfolio program since
1992.  


In 1995 a Panel of nationally known experts conducted a review of our KIRIS
portfolios as part of an overall review of KIRIS.  Here is what they said:




"For numerous reasons, the Panel concluded that the KIRIS portfolio
assessments are currently inapporpriate for use in the KIRIS accountability
system.  Despite improvements in rater consistency over the first biennium,
the scoring of portfolios remains too flawed for use in a high stakes
system."  (Page 6, Review of Measurement Quality....)


The report goes on a few sentences later: "Numerous aspects of the
operation of the program cast doubt on the validity of the portfolio
scores, and the evidence pertaining to the validity is both very limited
and unpersuasive."   (Page 6, Review of Measurement Quality....)




The above are from:  REVIEW OF THE MEASUREMENT QUALITY OF THE KENTUCKY
INSTRUCTIONAL RESULTS INFORMATION SYSTEM, 1991-1994, By Ronald K.
Hambleton, Richard M. Jaeger, Daniel Koretz, Robert L. Linn, Jason Millman,
and Susan E. Phillips.  This is a "dream team" where analysis of testing
and assessment is concerned.  You can get the full report from the Kentucky
Office of Education Accountability, Capitol Annex, Frankfort, KY  40601. 
Phone 502-564-8167.




A brand new report dated January, 1998, agrees with the earlier 1995 study.




"Early audits of portfolio scores in 1992-1994 showed systematic and often
dramatic differences between teacher scores and external expert scores. 
The 1996 writing portfolio audit suggests that the gap is narrowing to
reasonable levels of agreement between teacher scores and expert scores,
even though an inflationary bias ranging between 5 to 10 percent in the
writing index scores remains.  A 1997 writing portfolio audit scoring audit
showed gaps in a similar range."  (Page 16, KIRIS, A Technical Review...)




In other words, the best accuracy is varies by about a letter grade.


This report indicates the team was divided on a recommendation for use of
portfolios.  That isn't an overwhelming endorsement.




These comments are in KENTUCKY INSTRUCTIONAL RESULTS INFORMATION SYSTEM: A
TECHNICAL REVIEW, by James S. Catterall, William A Mehrens, Joseph M. Ryan,
Eugene J. Flores, and Pamela Rubin.  It is also available from the KY OEA
at the address and phone number above.


Overall, the Kentucky portfolio program has been quite contentious due to
the exhorbitant amount of time required and the questionable validity and
reliability of scores.  


Richard Innes  
EDUCATION CONSUMERS CLEARINGHOUSE