<a href="../98/02/kirisbad.txt">KIRIS was supposed to be set to
NAEP</a> but KIRIS goes up while NAEP scores are stagnant

Date sent:        Thu, 22 Jan 1998 12:57:50 -0500
From:             "Richard G. Innes" <70224.434@compuserve.com>
Subject:          RE:States Compare Testing Standards -- From Ed Week
To:               ed-consumers main listing <education-consumers@tricon.net>

To ALL:

Dave Shearon wrote about the new group, Achieve, and other plans to make
state tests comparable to NAEP.  ECC and others in education need to know a
few things about attempts that have already been made to link NAEP to state
standards.

The first example is the Kentucky Instructional Results Information System
(KIRIS).  KIRIS has been in development since 1991, and the assessments
have been administered continually since 1992.  By law, KIRIS was supposed
to be the same as, or similar to, the National Assessment of Educational
Progress (NAEP) during at least the development phase.  The KIRIS creators
even indicated they employed NAEP "Frameworks" to create the tests.  

Considering the legal requirement and the employment of similar frameworks,
one would have expected a high degree of agreement between KIRIS and NAEP
results, *but* that has not been the case.  KIRIS has shown steady and
impressive growth, while Kentucky's performance on NAEP tests has been
essentially stagnant.  Thus, a testing program which may well be the
closest parallel to the NAEP that any state has created has not correlated
well to the national test.  Whether the reason is an inherent inability to
create stable tests with open response questions or ineptitude on the part
of either Kentucky or NAEP (here I would certainly place blame with
Kentucky), no-one currently knows.

The second point of interest is Kentucky's tests are not alone in poor
comparability to NAEP.  A rather complex, empirically based paper, "Linking
Statewide Tests to the National Assessment of Educational Progress:
Stability of Results," by Robert Linn and Vonda L. Kiplinger, points out
that there are a number of problems with attempts to make a rigorous link
from state testing based on standardized achievement tests to the NAEP. 
This paper points out: "Tests or assessments constructed for different
purposes using different content frameworks or specifications will almost
surely violate the conditions required for strict equating."  The main
problem surfaces with the very best kids and the very weakest.  Those
scores diverge considerably from one test to the other.  There were also
gender related differences.  The report concludes: 

"No matter what the substantive explanation for the lack of stability of
the equating function from 1990 to 1992, it seems clear that there is
substantial uncertainty in the estimates.  The lack of stability suggest
that linking standardized tests to NAEP using equipercentile equating
procedures isnot sufficiently trustworthy to use for other than rough
approximations."

So, both the leading example of a state implementation of a NAEP-like test
and recent testing with standardized achievement tests have failed to link
to NAEP.  Again, whether that is due to inherent problems with linking all
tests, or something else, I don't know.  

I suspect the *only* way state tests can be made to link to each other or
to any national tests will be if they all become essentially identical.

If that happens, we will have created a national curriculum that will be
under the control of one, national entity.

Richard G. Innes  
EDUCATION CONSUMERS CLEARINGHOUSE