\doc\web\99\02\firsworl.txt FIRST WORLD, THIRD WORLD, ALL RIGHT HERE AT HOME: HOW INTERNATIONAL COMPARISONS OBSCURE THE TRUE CONDITION OF EDUCATION TABLE 1 HIGH SCORERS ON NAEP AND THE TIMSS MATHEMATICS TEST NAEP % CORRECT SINGAPORE 328 79 KOREA 313 73 JAPAN 313 72 HONG KONG 306 70 BELGIUM (Flemish) 297 66 CZECH REP. 297 66 TOP THIRD, U. S. SCHOOLS 293 63 NEXT NINE NATIONS AVG. 287 60 TABLE 1 HIGH SCORERS ON NAEP AND THE TIMSS MATHEMATICS TEST NAEP % CORRECT SINGAPORE 328 79 KOREA 313 73 JAPAN 313 72 IOWA 284 59 WHITE STUDENTS, U.S. 282 57 ASIAN STUDENTS, U.S. 274, 282 TIMSS 8TH GRADE TIMSS-NAEP MATH RESULTS: LOW SCORERS HISPANIC STUDENTS, U. S.251 42 BOTTOM 1/3 U.S. SCHOOLS 244 40 BLACK STUDENTS, U.S. 243 39 DISTRICT OF COLUMBIA 233 33 SOUTH AFRICA 218 24 TABLE 4. TOP SCORERS IN TIMSS-NAEP COMPARISON: 8TH GRADE SCIENCE NAEP % Correct Singapore 172 70 Korea 166 66 Japan 164 65 Maine 163 64 Wisconsin 160 63 White Students, U. S. 159 62 Not eligible for free lunch, U. S. 156 61 Asian Students, U. S. 152 59 Gerald W. Bracey Gerald W. Bracey is an independent researcher and writer living in Alexandria, VA. He is author of the annual, "Bracey Report on the Condition of Public Education," (Phi Delta Kappan, October). His most recent book is Put to the Test: An Educator's and Consumer's Guide to Standardized Testing. A profile appeared in the Virginia Weekly section of the Washington Post, March 26, 1998). The view that all American public schools are in crisis is wide spread. Consider four representative quotes: 1. "Our schools are failing.." 2. "The public school system as we know it has proved that it cannot fix itself. It is an ossified government monopoly. 3. "American public education continues to be in crisis." 4. ".people don't learn only at school. If they did, we'd be doomed." 5. The authors of these quotes are, respectively, IBM CEO, Louis V. Gerstner, Jr. (New York Times, 1994); former assistant secretary of education, Chester E. Finn, Jr., (Wall Street Journal, 1998); education consultant Denis P. Doyle (Heritage Foundation publication, 1996); and Robert J. Samuelson, Newsweek and Washington Post pundit, 1998. Those who believe that our schools are "broken" are in for a surprise if they bother to examine the results of the Third International Mathematics and Science Study (TIMSS) and what it shows when linked to the National Assessment of Educational Progress (NAEP). The U. S. Department of Education commissioned Educational Testing Service to conduct such a linking and has been curiously quiet about the results. For each of the 40 states that participated in a NAEP state-by-state comparison, ETS reported how many of the 41 countries in TIMSS scored statistically higher, the same and lower than that state at the 8th grade level . I went further. I obtained conversion formulas from ETS and, using them, obtained a precise number for each country. It turns out that among the 41 countries in TIMSS, the distance between the highest scoring nations and the lowest is not much greater than the distance between the highest states and the lowest states. And our best states compare pretty well to the top countries. In math. Where we are such putative dolts. As we shall see later, when it comes to science our best states are virtually unequaled. The figures for the high scorers in math are given in Table 1. The numbers represent scores on the NAEP mathematics scale followed by simple percent correct. These numbers are for public schools only. TABLE 1 HIGH SCORERS ON NAEP AND THE TIMSS MATHEMATICS TEST NAEP % CORRECT SINGAPORE 328 79 KOREA 313 73 JAPAN 313 72 HONG KONG 306 70 BELGIUM (Flemish) 297 66 CZECH REP. 297 66 TOP THIRD, U. S. SCHOOLS 293 63 NEXT NINE NATIONS AVG. 287 60 IOWA 284 59 MAINE 284 59 MINNESOTA 284 59 NORTH DAKOTA 284 59 MONTANA 283 58 WISCONSIN 283 58 AUSTRALIA 283 58 NEBRASKA 282 57 WHITE STUDENTS, U.S. 282 57 NOT ELIG. FOR FREE LUNCH, U.S. 280 56 ASIAN STUDENTS, U.S., 274, 282 Readers might find it strange that Japan and Korea have the same NAEP scale score but different percent correct. This is possible because the scaled score takes into account the margin of error. Different countries have different margins of error because of the degree to which they met the TIMSS sampling criteria. Readers might find it equally strange that Asian students have two scores, neither of which is larger than the scores for white students. The two scores come from two different ways of estimating the results for Asian students. The U. S. Department of Education thinks that the differences might come from sampling error, which can be quite large for Asians, because nationally they are a small proportion of the population. It is also possible, however, that the results come from changes who is immigrating to the United States from Asia. As a group, Asians are much better educated than the population as a whole and have a substantially higher income. Recent waves of immigrants from Southeast Asia, though, have not conformed to this profile of earlier Asian immigrants. They include many more uneducated rural people. Table 1 shows that only six nations, all small and homogeneous, score higher than fully the top third of American schools (whose homogeneity it would be interesting to know). And before anyone swoons over the scores in top-ranked Singapore, consider that the Singapore Minister of Education has visited this nation several times in recent years, trying to figure out how to make his kids more like ours. They can take tests, he says, but they can't think and they aren't creative. Similarly, in The Enigma of Japanese Power Dutch Journalist Karel van Wolferen, a Japan resident of some 25 years, reported the case of the Japanese scientist who won a Nobel Prize. Opinion in Japan was universal, says van Wolferen: the scientist won only because he had lived in Europe and the United States for many years. Had he remained in Japan, the pressure for group-think and consensus would have prevented him from engaging in the individualistic, high-risk research that often leads nowhere but which is also essential to the kinds of discoveries that lead to in Nobels. Clearly there is much more to school outcomes than can be measured by tests. Still, it is interesting to note that a third of our schools are up there with the top nations. It is also the case that a cluster of states scored as high as a second cluster of nine nations. These states once again confirm the earlier conclusion of Senator Daniel Patrick Moynihan that high test scores in this country are associated with proximity to Canada. We can get an idea of how the range of nations and states looks by examining table 2. Table 2 TIMSS 8TH GRADE TIMSS-NAEP MATH RESULTS: LOW SCORERS NAEP % CORRECT LITHUANIA 262 48 SOUTH CAROLINA 261 48 CYPRUS 260 48 ALABAMA 257 46 LOUISIANA 252 43 PORTUGAL 252 43 FREE LUNCH ELIGIBLE, U. S. 252 43 HISPANIC STUDENTS, U. S. 251 42 MISSISSIPPI 250 42 BOTTOM 1/3 U.S. SCHOOLS 244 40 BLACK STUDENTS, U.S. 243 39 IRAN 242 38 DISTRICT OF COLUMBIA 233 33 KUWAIT 228 30 COLOMBIA 225 29 SOUTH AFRICA 218 24 UNITED STATES 271 VIRGINIA 270 UNITED Only four nations of the 41 scored lower than the bottom third of American schools, and only three outdid the District in sinking low. ETS also conducted another kind of analysis. Its researchers said, "Suppose we take the kids in all 41 countries and lump them all together. Then let's find their average score (50th percentile) and the score of the their top 10% (90th percentile). How many students in the various states would score as high or higher?" The answer is shown in Table 3. TABLE 3 What percentage of students score as well as the top 10% and top 50% of all 41 TIMSS nations combined? Top 10% Top 50% Iowa 4% 63% Minnesota 6% 62% Maine 6% 62% Montana 6% 62% Nebraska 5% 61% VIRGINIA 3% 45% Alabama 1% 32% Louisiana 1% 25% Mississippi 1% 23% The picture painted by Table 3 is that our highest scoring states produce more above average students than the 41 nations combined but not quite so many that might be considered outstanding. Fully 63% of Iowa's students would be above the average score of all 41 nations. Who are these 41 nations? The results wouldn't mean much if our kids were beating up on underdeveloped countries in Africa and South America, but fully 27 of the 41 countries are European. Four are Asian. There are, in addition, Australia, New Zealand, Canada and Israel. These countries, in other words, are mostly not mere educational also-rans. At the bottom things look pretty grim. I was once chastised for saying that some science textbooks in Alabama predict that man might one day walk on the moon, a statement I had heard on National Public Radio's "All Things Considered." When I checked into the situation I found that it was even worse: some Alabama schools didn't even have textbooks. This top 10% and top 50% analysis is available only for states, not other NAEP categories shown earlier. Things look better in science than in mathematics. The top and bottom third data are not available for science. Given that the overall performance of the U. S. in science was better than in math, one would expect a slightly improved outcome. In mathematics at the 8th grade level, American students achieved 53% correct compared to a 41-country average of 55. In science, American students outscored the international average, 58% correct to 56%. TABLE 4. TOP SCORERS IN TIMSS-NAEP COMPARISON: 8TH GRADE SCIENCE NAEP % Correct Singapore 172 70 Korea 166 66 Japan 164 65 Maine 163 64 North Dakota 162 64 Montana 162 64 Czech Republic 162 64 Wisconsin 160 63 White Students, U. S. 159 62 Minnesota 159 62 Bulgaria 159 62 Iowa 158 62 Nebraska 157 62 Vermont 157 62 Massachusetts 157 62 Netherlands 157 62 Slovenia 157 62 Austria 157 61 Not eligible for free lunch, U. S. 156 61 Utah 156 61 Hungary 156 61 Connecticut 155 60 Oregon 155 60 Belgium (Flemish) 154 60 Australia 153 60 Asian Students, U. S. 152 59 (The NAEP Science Scale is different from the NAEP Math Scale) Only Singapore has a statistically significantly higher score than our highest states. It also should be noted that the top scorers are much closer together in science than in math. "Statistically significant" means that if the tests were given again, Singapore would in all likelihood finish #1 again, but Japan and Korea might fall below the top states. At the bottom, the picture is similar, but there are more countries packed in with the lowest scoring states. Again, the differences among countries are smaller. Iceland, the first country presented in the low-scoring group ranked 30th among the 41 nations and got 53% correct, just 17% fewer than top ranked Singapore. Still, the lowest states in this country are ranked among a group of mostly Third World nations. TABLE 5. TIMSS NAEP COMPARISON: LOW SCORERS IN SCIENCE NAEP % CORRECT Iceland 136 52 Hawaii 135 51 Romania 134 50 Latvia 134 50 Free lunch eligible, U. S. 133 50 Mississippi 133 50 Louisiana 132 50 Denmark 131 51 Lithuania 130 49 Hispanic Students, U. S. 129 50 Belgium (French) 129 50 Iran 129 47 Cyprus 126 47 Black Students, U. S. Schools 120 45 Kuwait 115 43 District of Columbia 113 42 Colombia 109 39 South Africa 81 27 As tempting as it might be to crow about the science results, I would caution modesty. My guess, and it is really no more than that right now, is that some other nations do not emphasize science as much as we do until students enter programs at the high school level that are focused on science and math or science and technology. My single piece of evidence for this is France. France scored well in TIMSS mathematics and very well in an earlier international comparison of reading (American kids finished second in the world, something the Bush Department of Education tried to ignore). But in science, France scored only one point better then Iceland, the first country in the low scoring group in Table 6. It doesn't seem reasonable to me that French kids' brains can absorb reading and math but not science. When we turn to how many of our states' students score above the 50th and 90th percentile of all 41 nations combined we obtain another warming portrait: TABLE 6. PERCENTAGE OF STUDENTS IN VARIOUS STATES SCORING AS HIGH IN SCIENCE AS THE TOP 10% AND TOP 50% OF THE 41 COUNTRIES COMBINED Top 10% Top 50% Connecticut 16% 65% Iowa 15% 69% Maine 18% 75% Massachusetts 16% 67% Minnesota 20% 67% Montana 16% 75% Nebraska 15% 68% North Dakota 17% 75% Oregon 13% 65% Utah 12% 67% Vermont 14% 67% ------------------- Alabama 6% 45% Louisiana 5% 38% Mississippi 4% 37% If our top 11 highest scoring states competed against all 41nations, some two thirds of each state's students would be above the average score for the 41 nations. Minnesota would have twice as many students scoring as high as the 90th percentile as the nations themselves did. Not bad, guys. Even at the bottom, there is considerable improvement over the results in mathematics. I am aware that there are racist interpretations available to explain these data. Indeed, Archie Lapointe, former executive director at NAEP told me that they were about to abolish the ethnic categories when Mary Hatwood Futrell, now the Dean of the School of Education and George Washington University, but then the President of the National Education Association counseled against it. Futrell thought we needed to keep the categories to mark progress-or lack of it-for the groups represented. In any case, I have much evidence to indict poverty as the principal culprit in the low performance of ethnicities, but to detail it would be another article. We should also note a technicality: For all states except Minnesota, these results are estimates constructed with statistical formulas. One could wonder about the accuracy. In the case of Minnesota we also have actual data to go along with the estimates. It is comforting to note that the estimated and actual are virtually identical. The variability of performance among states and various ethnic and socioeconomic groups is so wide as to give us first world and third world, all right here at home. As long as people persist in thinking of American schools as an undifferentiated mass and mess, we cannot take steps to correct the real problems that do exist. There is also a TIMSS "Final Year" report but is meaningless. It purported to compare students in their final year of secondary school. However, the systems in various countries differ so much in their focus, their duration, and the age of the students in the terminal year that it's not even an apples-to-oranges comparison, more of an apples-to-iron ore. There are also so many technical problems that only 5 of the 26 nations in the study met the TIMSS criteria for exclusion and participation rates. I described the various problems in the May and September, 1998 issues of Phi Delta Kappan. This is an estimate. The last time this statistic was reported was in 1992 when the figure stood at 289. Between 1992 and 1996, overall growth was four points, so four points were added to the growth of the top third of U.S. schools. This addition is reasonable and might well be a conservative estimate: Between 1990 and 1992, overall growth was five points but growth for the top third schools was nine points. 15 2