zip74\doc\web\2003\09\wasl2003.txt Revised Sept 15, 2003 See Arthur Hu's Powerpoint on the 2002 WASL Math study here: http://www.arthurhu.com/2002/10/mathstud/mathstud_files/frame.htm To: ghall@ospi.wednet.edu Cc: bneitzel@ospi.wednet.edu Subject: Review of WASL and answer key as classwork (Note I got no response to this message) Hello Greg, Arthur Hu here. I saw this year's 4th grade WASL, near as I can tell, it's still about 50% wrong, either obviously at the wrong grade level, or requires significant work beyond straightforward application of proficient 4th grade knowledge and skills. Even though some of the problem items were removed, some items I complained about are still in the current test. I'd also like to ask your opinion about some classwork that I've been shown - every item is effectively an answer key which shows you exactly how to solve an actual WASL item, obviously after inspecting the actual test. I've spoken to principals who believe that this is entirely appropriate. It's about in the middle of the review. This might be very widespread. Since I'm still the only person in the state that can figure this stuff out, doesn't that make me qualified to be the next assessment director?? Kirkland Parent reviews WASL 4/5, 50% problems wrong, WASL answers issued as classwork May 5, 2003 Highlights - Most parents don't know they can inspect the test - Mr. Hu was one of the original complainers leading to 2000 study - Mr. Hu concluded that revised 2002 test was also flawed and that test difficulty, passing cut score, specifications, and standards were adjusted, making comparison impossible - The 2003 math test still has over 50% problems not compliant with 4th grade standards or cognitive levels, includes some problems he had complained about in 2002. - One 4th grade problem was already declared faulty after printing, and not allowed to be given to students. - Some schools are issuing class worksheets that are virtual answer sheets to WASL problems with only a few details changed. - Some students still being assigned old WASL problems which have since been determined to be not compliant with grade-level EALRs as "problem of the week". - The science test tests hardly any facts, and does not use standard terminology. Most is high school or college level lab skills. - Scores won't even be returned since nobody knows what a reasonable "standard" is. PARENT EXERCISES RIGHT TO INSPECT 4TH MATH AND 5TH SCIENCE TEST I finally got a look at the 2003 4th grade WASL, and 5th grade science WASL. It's a shame that the press have never notified the public of their right to inspect the test if their child is taking the test, let alone that this parent has done so and gone to OSPI with specific complaints of problems with the WASL test. 5TH GRADE SCIENCE TEST WON'T EVEN BE GRADED The 5th grade science test won't even be graded, so it's a real waste of time. The least they could do is return a % correct, and how a student did ranked compared to everybody else. If everybody gets only 20% right (which might be what comes back), that's something the public should know. This only shows that the concept of "standard" is competely invalid if it depends on the acceptance of a a valid test and acceptable cut score. (Note - Sept 2003, the superintendent says he got a look at the science scores, but for some reason they are not being made public) The 4th grade math test was the one that was so bad that the state legislature forced the OSPI to look at it. The OSPI concluded it was perfectly fine except for the 10% of problems that were competely wrong and 25% that had other problems. NOT JUST TEST, 2002 STANDARDS WERE ALSO MOVED AROUND The 2002 test was the first "fixed" test reflecting not only the revised test, but what I revealed were also revised standards and item specifications, so when I had complained that items like mean, median and mode were in the 7th grade, they moved them down into the 4th grade. 2002 TEST STILL OUTRAGEOUSLY BROKEN AFTER 2001 "FIX" What I found was the 2002 test was still outrageously broken, even though it was less blatant about putting 10th and 7th grade items into 4th grade. About half of items were clearly not compliant with 4th grade specifications or EALR standards, or had standards changed from their original 1997 level without notice. Even when mean/average was moved down to 4th grade, one problem asked students to chose the proper method of computing an average when the specification clearly states that students are not to be asked for a definition, and that the definition of average was its method. Other problems, such as picking out the winning square of a 10 x 10 tic tac toe game required game and IQ skills clearly outside the scope of merely picking out a row and column from a grid. Many of the spinners were not clearly marked to allow students to easily determine which areas were larger than others. No protractor is supplied, nor are 4th graders instructed how to use protractors to measure the angles neccesary to solve these problems. In contrast, the 10th grade spinners were simple to compare without the use of protractors. Moreover, computing and comparing probability as ratios, percentages, or areas is a 7th grade level skill, not 4th grade. 4th grade only specifies certain vs uncertain. Another problem asked which square would deliver a win in the next move in the Japanese game of GO. Specifying a letter and number for the square might be fair game, but testing on strategy is clearly out of scope for a 4th grade proficiency test. It appears that nearly all of the problems are designed to guarantee that if all you know are basic 1 step 4th grade math skills, you won't be able to solve these problems, which is also true of nearly all NCTM (National Committee of Teachers of Mathematics ) standards based problems. OSPI ASSESSMENT DIRECTOR ADMITS THEY GOOFED BIG TIME, BUT ONLY TO ME Greg Hall, assessment director at OSPI at first said that none of my complaints were valid. After meeting with him in September, he still refused to talk about items which were not released, but fortunately, half of the items were released. Then he admitted that most of my complaints were valid, and they reviewed and removed many items from the 2003 test and item bank. Furthermore I observed that the test seemed a lot easier than the sample test, and that items had moved around in the 2001 revision of the standards. OSPI AGREES TO MASSIVE FIXES FOR 2003 TEST Hall admitted that the test was deliberately easier. They indeed had changed the cut score, and they had even changed the standards and item specifications without any notice. This makes any valid comparison between years impossible 2003 TEST IS STILL MASSIVELY BROKEN OVER HALF OF WASL PROBLEMS ARE DEFINITELY OVER GRADE LEVEL, OR DEVLOPMENTALLY INAPPROPRIATE FOR 4TH GRADE In the 2003 test, I still come up with about 20 of 40, or 50% of problems that are either completely beyond the 1997, 2001 standard, or way beyond simple 1 or 2 step application of 4th grade skills w/o excessive construction of a complex method. In other words, a student who knew 4th grade arithmetic, geometry, and statistics backwards and forwards of average IQ could still get over half of the problems wrong. I still recognize 2 or 3 problems I had complained about in the 2002 test, such as requiring students to compute an average without being told the method for computing an average, or extrapolating a timetable from rush hour to the middle afternoon. PROBLEMS IN VIOLATION OF 2002 EALRS STANDARDS MANUAL Some of state standards that are violated: The Essential Academic Learning Manual is on the web at: http://www.k12.wa.us/curriculuminstruct/documents/EALRs-Technical-Manual .doc - rate is grade 7 Grade 4: blank Grade 7: Understand and apply the concepts of ratio and direct proportion. - Compute the fraction of a number in millions is grade 7 Grade 4: Add, subtract, multiply, and divide whole numbers. Grade 7: Add, subtract, multiply, and divide nonnegative fractions - Computing average or mode is grade 7 (2 problems) Grade 4: Understand.... such as mean, median, and mode Grade 7: **Calculate** and appropriately use ... measures of central tendency - Creating a survey method r (1 problem) 2001 study concluded that constructing a method was not allowable. - Computing or comparing probability as a ratio other than certain or uncertain is grade 7. Several spinner problems violate this. Grade 4: Understand the difference between a *certain and uncertain* event. Grade 7: Know how to *calculate numerical measures* of chance for simple events. - Very complex "slice" problem "how many slices per person" is a fraction word problem that really require an algebraic solution Grade 4: Evaluate simple expressions using blocks, sticks, beans, pictures Grade 7: Solve simple equations using blocks, sticks, beans, pictures - one problem using stacked bar charts was so botched it was crossed out entirely and not given to students - Sorting triangles without a clue as to what to sort them by Grade 4: triangle isn't even mentioned Grade 7 standard: describe...including..triangles, isosceles, equilateral [there is not mention of right triangle at all] - Figure out without a clue that a frequency histogram (sometimes line plot) would be the best way to solve a problem. The way the EALR standard seems to allow this, but this is really beyond the developmentally appropriate level for a 4th grader without a hint. - Create a fully chart as fully labled as Excel or Powerpoint from a nearly blank page. Now the EALR standard does mention drawing charts, but it's pretty taxing to expect every 4th grader to be able to whip out a fully labeled chart complete with axes just like Excel. As a graphics programmer who has created software to do this at the age of 20, this is no mean feat. Filling in the blanks or maybe part of the data would be reasonable for an assessment. Grade 4:Organize and display data in numerical and graphical forms such as tables, charts, pictographs, and bar graphs. Grade 4: Draw and build simple shapes and figures using appropriate tools, such as a straightedge, ruler, protractor, or nets. [Now this is more reasonable] WASL ANSWERS AS WORKSHEETS Very interesting that someone has sent me 4th grade hand-made worksheets that were VERY similar to many of the new WASL problems, except that they were closer to being appropriate - they gave hints. Example - Organize this data on a line plot so you can answer the questions: 12 14 11 6 15 10 6 12 13 9 8 7 9 12 11 9 10 10 12 12 11 x x x x -------------- 6 7 8 How many students could do more than 11 sit ups? Could more than half do at least 10 sit ups? The WASL doesn't even hint as to what sort of graph would be helpful. Here's another worksheet problem: Match the triangle right obtuse acute The actual WASL wants you to match triangles, but gives no hint as to what categories items should be sorted by. According to the standards, such categorization is grade 7. Standards: Grade 4: triangle isn't even mentioned Grade 7 standard: describe...including..triangles, isosceles, equilateral [there is not mention of right triangle at all] I have these sheets avaiable to fax to anyone who is interested. I have no idea how widespread this is, but this gives a whole new meaning to teaching to the test. This probably explains why a few schools did VERY well on WASL, too well. There is nothing to keep teachers from giving out what amount to solution methods before the test. For that matter, there isn't anything to keep parents from passing on hints as well. LAKE WASHINGTON ADMINISTRATION SAYS "IMPOSSIBLE" A letter that I got from my principal and Superintendent said that there was absolutely no relationship between these worksheets and unreleased test problems, even though my son brought these home shortly before he took the WASL test. Perhaps it is not surprising that AG Bell's 4th grade math pass rate just about doubled to 70 percent, above state average, as did a couple of other area elementary schools which I suspect may have done the same thing. 90th PERCENTILE STUDENT FAILS WASL STANDARD My son Peter, who scores about the 90th percentile on standardized ITBS math, and well over 60 on verbal nevertheless failed to pass the listening test. You only have to fail one test to fall "below standard". You can see my comments on the listening test below, which goes way beyond simple listening skills, but high level analysis of plot and characters. PROBLEMS IN VIOLATION OF COMMON SENSE The 2001 revision of the EALRs includes Mean, Median and Mode. Mean is the average, Median is the halfway point, Mode is the most common data point. 3 or 4 problems out of 4 is a huge proportion, are devoted to these concepts. However, one New York professor of mathematics noted that Median and Mode are inappropriate even for inclusion on the high school level Regents examination as most mathematicians are even very familiar with the concepts. Also, neither the college entrance SAT or ACT tests include either median or mode. Even when they appear on the EALR standards, COMPUATATION of these figures is pegged at grade 7, yet the 2002 or 2003 WASL tests require computing the mean, median, and mode. VERY LITTLE 4TH GRADE MATH There are almost no examples of straightforward 4th grade level arithmetic - subtracting, adding, multiplying or dividing 3 or 2 digit whole numbers. They're multiplying a number in the millions by a fraction which has absolutely nothing to do with 4th grade level math by the state's own standard. READING PROBLEMS Some of the reading was based on a book I saw used in some classrooms. Students who had used that book would be at an advantage. I was bothered by the sort of question - the BEST reason or the BEST conclusion would be, when 2 or all 3 choices were in some degree true. LISTENING I was appalled by the number of subjective questions without even seeing the passages being read. Cindy was a) close b) distant c) indifferent to Jack. WRITING A TOTAL JOKE The writing prompt is a total joke. It consists of two lines, and is just like "You walk by a hole in the ground with a ladder leading down. Write 2 pages about what happens next". If you don't have a very vivid imagination, you'll be sunk. It's completely creative process. If you're an inner city deprived kid, it's hard to imagine them putting much into this. On the other hand, if you've been outdoors, or seen a lot of movies or read a lot of books, you might do fine. There is no way the students are even writing about the same thing. If they were doing a book report, or writing about a specific historic event, it would be more fair. The "standard" must be completely subjective. How you'd take a piece of writing and declare it a 0, 1, 2, 3 or 4 is completely nuts since writing ability of 4th graders is all over the place. Science Most of the science is interpreting experiments, or asking studnets to write how they would set up an experiment. I never had this this stuff in 5th grade, and 5th grade classrooms don't have science labs. They're asking kids how many 100ml beakers and how many ml of solution to use, and which variable are changed, or which ones to change. Stuff I, as a kid who went to engineering school by age 18 didn't need to worry about until high school. None of the lab scenarios required any knowlege of science fact. The few questions that required knowledge of facts covered topics such as physics that weren't typically covered in K-12 in my days until high school physics. Even then, standard terms such as "potential" and "kinetic" energy aren't even used by WASL. This is a completely useless test since science isn't really a K-6 topic at all. Even at the high school level, traditionally, only college prep students even took science, and not all student took the same number of the standard biology, physics and chemistry classes. This goes back to the traditional reason for sticking with verbal and math aptitude tests, because this is something that all kids do, and all kids are taught at some level. It's useless to test kids on material they haven't been taught, then expect the schools to align their curriculum to what amounts to an invalid test / standards. NOT TOO LATE TO STOP GRADUATION REQUIREMENT The statement that the WASL will be a graduation requirement isn't really true. The State Board must find the test to be valid as a graduation test, or the legislature could ban the State Board from making such a requirement. Every testing manual, even the WASL technical notes say that no single test score should be used as a condition that can deny a student a diploma or grade-level promotion. Every politician and educator who promotes this idea is in violation of this very simple rule of education. A standard isn't a test or a passing score. A content standard can't fairly set a passing performance standard, especially in a school system whose goal and mission is to educate every student at every age and ability level, up to the age of 18. Arthur Hu arthurhu @ comcast.net 425-891-2619 Kirkland / Lake Wash School Dist.