+OK 16805 octets Received: from smtp06.nwnexus.com (smtp06.nwnexus.com [206.63.63.45]) by mail3.halcyon.com (8.8.8/8.8.8) with ESMTP id QAA15151 for ; Thu, 7 Oct 1999 16:17:27 -0700 (PDT) Received: from minglewood.dundee.net (minglewood.dundee.net [206.249.104.16]) by smtp06.nwnexus.com (8.8.8/8.8.8) with SMTP id QAA07306 for ; Thu, 7 Oct 1999 16:17:23 -0700 (PDT) Date: Thu, 7 Oct 1999 19:15:38 -0400 From: "Richard G. Innes" <70224.434@compuserve.com> Subject: [education-consumers] NAEP Mess Update, Part 1 Sender: "Richard G. Innes" <70224.434@compuserve.com> To: "ClearingHouse" Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Disposition: inline Reply-To: "Richard G. Innes" <70224.434@compuserve.com> Precedence: bulk Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by mail3.halcyon.com id QAA15151 Status: ===================================================================== This is part 1 of a two part message. This is a text-only version of a more extensive document in Microsoft Word. The author will be happy to provide the full version, which includes graphs and tables, upon request. Richard G. Innes 2836 Deerfield Drive Villa Hills, KY 41017 INTERNET:70224.434@Compuserve.com Phone: 606-344-0406 October 4, 1999 The Troubling Situation With The 1998 National Assessment of Educational Progress (NAEP) 4th Grade Reading Assessment When the 1998 NAEP Reading Report Card for the Nation and The States [1] was released on March 4th, 1999, a problem with the state-level data was immediately apparent. Some states showed a considerable increase in the number of students with learning disabilities (SD) who were excluded from testing due to conflicts between requirements in their individual education plans (IEP) and NAEP testing guidelines. Apparently, provisions in the Individuals with Disabilities Education Act of 1997 made it impossible to conduct uncorrupted and meaningful tests of many SD children for reading ability. Because of IDEA, many more children had to be excluded from the NAEP in 1998. The higher exclusion rates raised questions about whether some state level NAEP scores accurately reflected real performance and could be fairly compared to other states. Kentucky offers one of the most dramatic examples of the problem. Kentucky's six point score increase between 1994 and 1998 was one of the best improvements of any state. But, it was matched by an equally large increase in the percentage of SD who were excluded from taking the test. The 1994 to 1998 change contrasts sharply with the state's change in scores and rates of exclusion between 1992 and 1994. In addition, from 1992 to 1998 the state's SD population skyrocketed from 7 to 13 percent, an increase of 86 percent which moved the state from two points below to two points above the national averages which also increased in this interval. Table 1. Kentucky's NAEP 4th Grade Reading Scores, (Exclusion Rates for Students with Disabilities As Percent of Raw NAEP Sample), And [Total Percent of Students with Disabilities As Percent of Raw NAEP Sample], By Year 1992 213 (4%) [7%] 1994 212 (4%) [8%] 1998 218 (10%) [13%] Note: The term IEP rather than SD was used to describe students with disabilities in 1992 and 1994. Notice that both the NAEP 4th Grade reading assessments in 1992 and 1994 tested 96 percent of the raw Kentucky sample. Just 4 percent of the raw sample was excluded. That changed very dramatically in 1998 when every one of the corresponding SD kids from the earlier years was excluded. In addition, some students who would have been classified in prior years as completely non-disabled were also excluded. Thus, in 1998 only 90 percent of the raw Kentucky sample was tested on NAEP. Clearly, the 1998 NAEP totally ignored a significant part of the most educationally challenged in Kentucky's student population. The foregoing made it seem fairly obvious that Kentucky's 1998 NAEP score might have risen simply because more weak students were not allowed to participate. The question was, what was the real impact of the increased exclusion on the NAEP scores? For an answer, the National Center for Educational Statistics (NCES) first turned to the experts at Educational Testing Service (ETS). ETS creates NAEP and provides technical administration and scoring services, too. A preliminary ETS report was issued in memo format by NCES on May 13, 1999 [4]. The ETS Memo offers a list of questions posed by NCES along with initial answers to those questions. The most critical question of all is number 4. In the memo, this reads: 4. How would gains in State NAEP reading scores have been affected if exclusion rates had been equal across years? The highly significant answer from ETS: The real answer to this question can never be known. ETS lists a number of reasons why this is so, but the bottom line is that data collected during the 1998 NAEP reading assessment is insufficient to answer this absolutely crucial question. In other words, the excluded students were not tested in any way on NAEP, so their reading abilities are unknown. Although they said that an exact answer will remain a mystery, ETS also engaged in some rather controversial "what-if" analysis to try to gauge the possible range of scoring error that had been introduced by the increase in exclusions. Their analysis indicated possibilities ranging from no statistically significant improvement for Kentucky to a statistically valid rise in the state's score. However, ETS did not publish the possible range of scores from their analysis, and ETS did not defend the full, six point rise in the Kentucky scores between 1994 and 1998, either. As a side comment, it is easy to show that other, entirely plausible assumptions about the 1998 NAEP provide much more pessimistic results than the ETS scenarios (Attachment 1, not part of this electronic version, has examples). The ETS report was a tremendous disappointment to the Kentucky Commissioner of Education. Eager for evidence that Kentucky's decade old reform had improved educational performance, he requested another study. Not too surprisingly, considering Kentucky's Commissioner sits on the board that governs the NAEP, he got his request. This second study was performed by Dr. Lauress Wise of the Human Resources Research Organization (HumRRO). HumRRO has a contract with the Kentucky Department of Education to conduct research on Kentucky education. Dr. Wise's report, released on September 27th, asserts that Kentucky made statistically significant improvement on the NAEP and that the impact of the exclusions was almost negligible [5]. It is important to examine how Dr. Wise generated this result which differed quite sharply from the ETS findings. Wise develops 'equivalent' NAEP scores for Kentucky's excluded SD by using their test results from a state-wide assessment used only in Kentucky, the Kentucky Instructional Results Information System (KIRIS). 1. There are a number of questions about the validity of this approach. One problem: The validity of the KIRIS became so suspect in Kentucky that this assessment was totally abandoned after 1998. Beginning in 1999, a new and quite different Kentucky assessment was launched which totally ignores the old KIRIS scores. Thus, Dr. Wise's work is predicated on results from an obsolete assessment that was abandoned for cause. 2. Another problem: Strong evidence from research on Kentucky's SD performance on KIRIS by RAND researcher Dr. Dan Koretz implies that virtually all of the excluded kids really didn't take a reading test at all with KIRIS [6], [7]. As shown in Table 2, RAND data for the years 1995 and 1997 shows that during this period about 3 out of 4 Kentucky SD consistently had an "oral presentation," or reading accommodation, in their IEP. In other words, these students had all their tests read to them by proctors. And, many SD received some of the other accommodations in Table 2, as well. >From the data in Table 1, it can be calculated that 10/13, or 77 percent of Kentucky's total SD cohort was excluded from 1998 NAEP testing. That is remarkably similar to the percentages of the SD population that Koretz found had the reading accommodation on KIRIS in both 1995 and 1997 (72%). Thus, assuming no dramatic demographic shifts between 1997 and 1998, it is highly likely that almost all of the Kentucky children excluded from NAEP in 1998 had the KIRIS 'reading' test read to them. If so, these kids did not take a real reading test in KIRIS. They actually took a spoken language comprehension test. Absent a detailed look at the exact demographics for KIRIS in 1998, we face the very real possibility that the 1998 KIRIS results don't tell us much about whether the Kentucky SD excluded from NAEP can read printed text. If this is actually correct, comparing these 'spoken word' KIRIS scores to real reading results for kids who did take the NAEP would be misrepresentation of the highest order. Dr. Wise's report is silent on the issue of whether students with learning disabilities tested on KIRIS really were evaluated for printed text reading. That is very unfortunate because the potential problem had been identified to NCES months before Wise issued his findings. For now, this makes Dr. Wise's report incomplete. It also makes NCES acceptance of the Wise report as the final word on this matter premature. Table 2. Percentage of Students With Disabilities Receiving KIRIS Assessment Accommodations, Grade 4, by Year Accommodation 1995 1997 None 19 19 Oral Presentation 72 72 Paraphrasing 49 48 Dictation 50 55 Cuing 10 10 Technological Aid 3 34 Interpreter 2 1 Other 8 9 Sources: 1995 Data ([6], Pg. 13); 1997 Data, ([7], Pg. 12) 3. Still more problems: Some Kentuckians with mild disabilities did take the 1998 NAEP on their own. Wise indicates their average score was 176.7 on NAEP's 500 point scale ([5], Table 1). When Wise converts KIRIS scores for the excluded kids to NAEP equivalents, the averages work out to something between 200.1 and 206.5 9 ([5], Table 2). So, if we accept Wise's findings, we have to accept the notion that Kentucky's strongest students with disabilities, those who can read on their own, were significantly outscored by other students with more severe learning problems who possibly might not be able to read printed text at all. Table 3. Dr. Wise's Calculated Scores for Kentucky 4th Graders with Learning Disabilities Who Took the NAEP and for Two Different Modelings of Those Who Were Excluded Average Score for SD Who Took NAEP Unaided 176.7 Average 'NAEP Equivalent' Score for Dr. Wise's Model 1 176.7 Average 'NAEP Equivalent' Score for Dr. Wise's Model 2 206.5 Source: Table 1, Subsample 2, Stratum B, and Table 2, NAEP Equivalent, Model 1 and Model 2 [5] Table 4 NAEP Scores for Kentucky Students With Disabilities, by Year Year Kentucky SD NAEP Score 1992 185 1994 168 1998 176.7 Sources: 1992 and 1994, (Attach 2), 1998, ([5], Table 1) 4. Another disturbing conclusion also follows from the data in Table 3. While NCES declined to publish scores for the 1998 Kentucky SD who took the NAEP due to inadequate sampling, there are scores available for the Kentucky SD who took NAEP 4th Grade Reading in 1992 and 1994 when more Kentucky SD children did take the NAEP (Attach. 2). These are summarized in Table 4. Notice in Table 4 that the SD cohort tested in 1998 did not score as well as the 1992 SD cohort. But, the 1992 SD group would reasonably be expected to include many more weak students than was true in 1998 (recall discussions about Figure 1). So, comparing Dr. Wise's findings to data for 1992 seems to indicate that Kentucky isn't being terribly successful with SD children. That finding is very different from the flavor in Dr. Wise's report. By the way, inspection of the test data in the 1998 NAEP Supplemental Data Tables indicates that Kentucky is far from alone in its disturbing performance with students with learning disabilities over the years (Full information on this is in the full version of this report). For now, it should be noted that virtually every state that took the NAEP 4th grade reading assessment in 1992 and again in either 1994 or 1998 shows declines in scores for their learning disabled population. Just one state per each year grouping (1992 to 1994, or 1992 to 1998), indicates improvement. This is for samples that include 31 and 21 states, respectively. By the way, a number of states had so many SD excluded in 1998 that their remaining SD samples were insufficient. Thus, 1998 SD scores were not reported for these states. Aside from Kentucky, states with missing 1998 SD scores include several where highly aggressive education reforms are in place such as Maryland and North Carolina. In addition, the state with the overall best NAEP improvement from 1994 to 1998, Connecticut, also had an insufficient SD sample in 1998 and did not receive SD scores. Connecticut did have a 9 point decline in SD 4th grade reading scores between 1992 and 1994. That is why the 1992 to 1998 figure cited above only included 21 states. Other states had so many SD eliminated from testing that their 1998 samples of SD kids was simply too small! Thus, while NAEP may not be providing accurate information about whether these states are making progress, it does offer disturbing clues that educational failure with SD students could be hiding behind rapidly increasing exclusion of the learning disabled from meaningful assessment participation. This entire situation makes it very inappropriate to gloss over what is happening with the growing number of students with learning disabilities in Kentucky and elsewhere. 5. One other point on the technical issues should be made. NAEP elementary school reading tests deal with children who are more than half way through their primary school years. The idea that a growing number of 4th grade children in any state have not been taught to read well enough to cope with a reading assessment is very disturbing, especially when the NAEP data shows we were accomplishing this task better with a much larger percentage of our learning challenged children in the early years of the decade. It is difficult not to believe that the rapid increase in the number of students who are being labeled as learning disabled and saddled with test-corrupting accommodations in some states is more a result of schools trying to cover up failure rather than a result of a real shift in the demographic makeup of school populations. To be blunt, if the 86 percent rise in students with learning problems in Kentucky between 1992 and 1998 is real, then the Centers for Disease Control and many other agencies should descend upon the Bluegrass State in droves to find out what is causing this epidemic of mental deterioration. ===================================================================== EDUCATION CONSUMERS CLEARINGHOUSE networking and information for parents and taxpayers on the internet Website & Archives: http://education-consumers.com You are currently subscribed to education-consumers as: arthurhu@halcyon.com TO UNSUBSCRIBE: Send a blank email to leave-education-consumers-989462S@lists.dundee.net ===================================================================== For less mail, use the following link and choose 1) a daily digest, 2) a daily list of subjects, or 3) no mail (read postings on Web) http://lists.dundee.net/scripts/lyris.pl?enter=education-consumers For more help & info: http://www.lyris.com/help or . +OK 8657 octets Received: from smtp06.nwnexus.com (smtp06.nwnexus.com [206.63.63.45]) by mail3.halcyon.com (8.8.8/8.8.8) with ESMTP id QAA14852 for ; Thu, 7 Oct 1999 16:16:39 -0700 (PDT) Received: from minglewood.dundee.net (minglewood.dundee.net [206.249.104.16]) by smtp06.nwnexus.com (8.8.8/8.8.8) with SMTP id QAA06113 for ; Thu, 7 Oct 1999 16:16:26 -0700 (PDT) Date: Thu, 7 Oct 1999 19:15:30 -0400 From: "Richard G. Innes" <70224.434@compuserve.com> Subject: [education-consumers] NAEP Mess Update, Part 2 Sender: "Richard G. Innes" <70224.434@compuserve.com> To: "ClearingHouse" Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Disposition: inline Reply-To: "Richard G. Innes" <70224.434@compuserve.com> Precedence: bulk Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by mail3.halcyon.com id QAA14852 Status: ===================================================================== Part 2 of a two part message on NAEP. Closing Comments At best, this entire matter is rather extraordinary. It is worth noting that the former federal Commissioner of Education Statistics resigned recently, and issues surrounding the 1998 NAEP have already been the subject of one quite heated Congressional inquiry. While that inquiry focused on political issues, which are most certainly troubling, the Congress did not discuss any of the equally disturbing technical problems with NAEP SD sampling such as outlined above. Hopefully, this report will stir NCES to reopen this matter to determine whether Dr. Wise's study can withstand close scrutiny by the technical education community. Absent NCES action, it may prove necessary for the protection of the interests of the nation's students with disabilities for Congress to make farther inquires. In any event, the controversial impacts of the 1997 Individuals with Disabilities Education Act of 1997 need revisiting. As things stand, IDEA forms a barrier both at the federal and state level that separates parents, the general public and state and federal leaders from the truth about school performance. Without modification, IDEA creates a climate that is hostile towards accountability assessment programs. Coupled with current accountability trends, IDEA creates powerful pressure to label children as learning disabled when that may not really be the case. Richard G. Innes ATTACHMENT ONE brief summary 1. Includes an Alternate Analysis of Kentucky's NAEP Exclusion Impacts which assumes that Kentucky's eliminated students could properly be scored a zero for reading because their own teachers have declared them such poor readers that all test questions must be read by proctors. The analysis shows that Kentucky's score would drop to no higher than 204 in 1998. Compare to the 218 the state actually recieved. 2. Also includes a linear regression analysis of the change in NAEP exclusion from 1994 to 1998 for the states that participated in both years compared to the changes in their scores. The slope of the regression line, 0.54, implies that for each one percent increase in SD exclusions, there was approximately a half a point increase in score. For Kentucky's 6 point increase in exclusion, that would mean 6 times 0.54 or an error of about 3.2 points due to the effect of exclusion. The Y intercept of the regression line (2.11) implies the overall average improvement for all states on NAEP 4th grade reading between 1994 and 1998 was closer to 2 points rather than the 3 points actually posted. That isn't a terribly strong improvement on a 500 point scale test, especially since most of it can be explained by statistical sampling error alone. This also raises questions about the possible corruption of scores for other states besides Kentucky. One problem with linear regression is that it assumes a straight-line relationship exists between the data for all points in the database. Analysis in the full report shows evidence that the linear model isn't accurate across the entire spectrum of changed exclusions. Potential non-linearity was explored by doing a piecewise regression analysis of only those states that had an exclusion change of plus 2 percent or more from 1994 to 1998. The slope of this piecewise line is 1.1, with a Y intercept of -0.43. That implies virtually all of the score increase for states at the top of the listing in Table 1 represents no real improvement between 1994 to 1998. Real performance for these states may have even declined a bit. As a result, it seems fair to say that: Regression analysis of published NAEP 4th grade reading score changes and exclusion rate changes indicates no less than 3.2 points of Kentucky's 6 point score increase might be solely caused by increased exclusion of students with disabilities. And, piecewise linear regression analysis implies a still higher inflation occurred due to the high rate of exclusion in Kentucky. Either of these situations would mean Kentucky's score change was not statistically significant. It must be noted that this is not hypothetical "what-if" modeling. This regression analysis is based on hard data including actual NAEP scores and actual exclusion trends. ATTACHMENT TWO contains an extract from NAEP 1998, 1994 And 1992 National And State Reading Summary Data Tables For Grade 4 Student Data, pages 63 to 71. This is too extensive for electronic posting but is online at the NCES web site, http://nces.ed.gov. This shows the SD scores for all states in all years of NAEP 4th grade reading. Bibliography [1] Patricia Donahue Et. Al., NAEP 1998 Reading Report Card for the Nation and the States, National Center for Educational Statistics Report NCES 1999-500, Washington, DC, March 1999. Downloadable from NCES web site at http://nces.ed.gov [2] Table B.7, NAEP 1992 Reading Report Card for the Nation and the States, Part of Faxed letter from Lawrence Feinberg, National Assessment Governing Board, Washington, DC, 17 May 1999. [3] Jay R. Campbell Et. Al., NAEP 1994 Reading Report Card for the Nation and the States, National Center for Educational Statistics, Washington, DC, January 1996. [4] John Masseo, John Donoghue and Catherine Hombo, A Summary of Initial Analysis of 1998 NAEP Exclusion Rates, (A published memo), Educational Testing Service, Princeton, NJ, 13 May 1999. Online at: http://nces.ed.gov/pressrelease/naep599ets.html [5] Dr. Lauress Wise, Impact of Exclusion Rates on NAEP 1994 to 1998 Grade 4 Reading Gains in Kentucky, National Center for Education Statistics, Washington, DC, 27 September 1999. URL: http://nces.ed.gov/commissioner/remarks99/9_27_99.asp#approach [6] Daniel Koretz, The Assessment of Students with Disabilities in Kentucky, CSE Technical Report 431, National Center for Research on Evaluation, Standards, and Student Testing, UCLA, Los Angeles, CA, July 1997. [7] Daniel Koretz and Laura Hamilton, Assessing Students with Disabilities in Kentucky: The Effects of Accommodations, Format, and Subject, CSE Technical Report 498, National Center for Research on Evaluation, Standards, and Student Testing, UCLA, Los Angeles, CA, January 1999. [8] NAEP 1998, 1994 and 1992 National and State Reading Summary Data Tables for Grade 4 Student Data, Weighted Percentages and Average Composite Scale Scores, National Center for Educational Statistics, Washington, DC, 19 February 1999. Downloadable from NCES web site at http://nces.ed.gov ===================================================================== EDUCATION CONSUMERS CLEARINGHOUSE networking and information for parents and taxpayers on the internet Website & Archives: http://education-consumers.com You are currently subscribed to education-consumers as: arthurhu@halcyon.com TO UNSUBSCRIBE: Send a blank email to leave-education-consumers-989462S@lists.dundee.net ===================================================================== For less mail, use the following link and choose 1) a daily digest, 2) a daily list of subjects, or 3) no mail (read postings on Web) http://lists.dundee.net/scripts/lyris.pl?enter=education-consumers For more help & info: http://www.lyris.com/help or .