Date sent: Sat, 28 Mar 1998 14:20:01 -0800 (PST) From: Caleb Burns Send reply to: Caleb Burns To: education-consumers@tricon.net Subject: Attacking CIM/CAM on test construction basis The CIM/CAM tests -- at least in this state -- were not constructed in keeping with the basic test construction standards found in the '85 "Standards of Educational and Psychological Test Construction." The only kind of validity claimed by the test-makers is "content validity" and when I Sueng Choi, Ph.D., what he meant by that, he said that a group of experts made up the questions. I asked him about other kinds of validity and he asked what other kinds of validity I might be concerned about. I said external validity, such as predictive, concurrent, etc. (What other measures do the CIM results predict to, etc.) He said, no, all the test people needed was content validity. I called him up a day later and asked whether the State or Oregon's Education Department had to follow the '85 standards, and he said no, that those standards had been replaced by '87 (date?) "Fair Testing Standards" and so I asked him where I could get a copy of those standards and he directed me to an internet site. But when I accessed the internet site, I found that the first paragraph of the "Fair Testing Standards" said that those standards were intended to be consistent with the '85 guidelines. I gave Choi a call back and he said yes, he had talked to the Steve Slater, his supervisor, and was told they had to follow the '85 standards. Among other things, the '85 standards say that if content validity is used as the major form of validity for a test, then the names, backgrounds, expertise, of the test-constructors had to be described in the technical manual, this along with the method used to reach consensus. Oregon -- accorind to Choi -- has no technical manual for the CIM/CAM and apparently there is no list of who drew up the questions for the CIM, the methods used to reach consensus, etc. I wrote to Ron Houser, apparently the head of the DOE's assessment division) about a month ago asking for this information (I'd rather have a paper trail) and have not heard back from him. Among other things, the '85 standards say that before one can claim that a test and educational program will result in a given desirable outcome (and clearly the Certificate of Initial Mastery tests along with the educational programs that go with it are claiming those very things), there must be empirical support for such a claim. The problem is, however, there is no empirical support for it. And the testing people I have contacted at the University of Oregon on this matter say one is right to be concerned about a test without external data to support it. Does anyone else have such concerns? Have the CIM/CAM been validated against any real world measures? Are educational people in other states violating their own state statutues by claiming there own "tests" are valid without actual proof of this? (I've got a rather long list of specific complaints from a methodological point of view against the Oregon version of the CIM/CAM and would be happy to send it to anyone who wishes it.) Yours, Caleb Burns, Portland, Oregon For ideas in general to help people: http://www.teleport.com/~calebb/ EDUCATION CONSUMERS CLEARINGHOUSE Well, if you've been following my rantings (check out http://www.leconsulting.com/arthurhu/index/washtest.htm) you'll find that for example, Washington has benchmarks that say that conditional probability is at 10 grade, proportionality is at 7th grade, yet they have a test with both of these items in their 4th grade test, while everyone at the Dept of Education says that is absolutely impossible to have any grade inappropriate content on their test because of the way it was constructed. The Seattle Times and other media refuse to air any concerns that the test is not only hard, it's way too hard. The New Standards is being sold as based on international standards, but there is absolutlely no evidence that the very difficult questions they have are in any way based on similar questions from other other nations, and are not based on norm-referenced comparisons with other nations. In other words, it's complete, total BS, and the tests are far worse than the tests they intend to replace. > Date: Sat, 28 Mar 1998 14:20:01 -0800 (PST) > From: Caleb Burns > Reply-to: Caleb Burns > To: education-consumers@tricon.net > Subject: Attacking CIM/CAM on test construction basis > The CIM/CAM tests -- at least in this state -- were not constructed in > keeping with the basic test construction standards found in the '85 > "Standards of Educational and Psychological Test Construction." The only > kind of validity claimed by the test-makers is "content validity" and when > Does anyone else have such concerns? > > Have the CIM/CAM been validated against any real world measures? > > Are educational people in other states violating their own state statutues > by claiming there own "tests" are valid without actual proof of this? > > (I've got a rather long list of specific complaints from a > methodological point of view against the Oregon version of the CIM/CAM > and would be happy to send it to anyone who wishes it.) > > Yours, > > Caleb Burns, Portland, Oregon > > For ideas in general to help people: > http://www.teleport.com/~calebb/ > > > > > EDUCATION CONSUMERS CLEARINGHOUSE > Date sent: Sun, 29 Mar 1998 19:18:34 -0500 From: "Richard G. Innes" <70224.434@compuserve.com> Subject: Attacking CIM/CAM on test construction basis To: ed-consumers main listing Ref: Caleb Burns Post: <<...the '85 standards say that if content validity is used as the major form of validity for a test, then the names, backgrounds, expertise, of the test-constructors had to be described in the technical manual, this along with the method used to reach consensus.>> We have never been able to get the Kentucky Department of Education to shake loose the identities of the people writing the Kentucky assessment, either. Actually, if I wrote some of those questions, I'd most definitely want my identity hidden! <> Interesting. Kentucky's KIRIS test would also flunk on this note. <<...the testing people I have contacted at the University of Oregon on this matter say one is right to be concerned about a test without external data to support it.>> To date, KIRIS has scant external validity data, and what is available is mostly negative. None, except from potentially very biased sources, is positive. I suspect some more "interesting" external evidence is going to be released shortly. I'll update ECC/Loop when that happens, as a lot of you are having to contend with glowing, but actually unfounded, comments about our reform as people try to push the same ideas in your states. <> Give me a week or two. Richard Innes EDUCATION CONSUMERS CLEARINGHOUSE