z68\doc\web\2003\07\timss.txt] Harlow, A., & Jones, A. Why students answer TIMSS science test items the way they do. Abstract: The purpose of this study was to explore how year 8 students answered Third International Mathematics and Science Study (TIMSS) questions and whether the test questions represented the scientific understanding of these students. "only 13% of the written test items actually elicited the knowledge held by the students in the middle school interview sample. For 58% of the items in the test, students had more knowledge than they wrote in their written responses, and for 29% of the items, students who had the 'correct' written response did not have a complete understanding of the concept being assessed." (p. 10) FRIENDS: A FORMER DOCTORAL STUDENT OF MINE SENT ME THIS. IT WILL BE ON GREAT INTEREST TO YOU. DON Don- I don't know if you are on the eddra listserv, but here is a recent post that I thought you might find interesting. If you have already seen it, I apologize. Bob V Begin forwarded message: From: "Ken Appleton" Date: Mon Jul 14, 2003 11:21:41 PM US/Pacific To: Subject: [eddra] Test results Greetings all Frank Vincent asked why test results are used to judge qulaity of learning. Good question! Below is the abstract (and a further brief summary) of a paper presented last week at the Australasian Science Education Research Association conference in Melbourne. Harlow, A., & Jones, A. Why students answer TIMSS science test items the way they do. Abstract: The purpose of this study was to explore how year 8 students answered Third International Mathematics and Science Study (TIMSS) questions and whether the test questions represented the scientific understanding of these students. One hundred and seventy-seven students were tested using written test questions taken from the science test used in the Third International Mathematics and Science Study. The degree to which a sample of 38 children represented their understanding of the topics in a written test compared to the level of understanding that could be elicited by an interview is presented in this paper. In exploring student responses in the interview situation this study hoped to gain some insight into the science knowledge that students held and whether or not the test items had been able to elicit this knowledge successfully. We question the usefulness and quality of data from large-scale summative assessments on their own to represent student scientific understanding and conclude that large scale written test items, such as TIMSS, on their own are not a valid way of exploring students' understanding of scientific concepts. Considerable caution is therefore needed in exploiting the outcomes of international achievement testing when considering educational policy changes or using TIMSS data on its own to represent student understanding. (URL: http://www.edfac.unimelb.edu.au/DSME/asera03/AbsractAK.html) Some specifics: Of the TIMSS items sampled, and the students followed up post-test, "only 13% of the written test items actually elicited the knowledge held by the students in the middle school interview sample. For 58% of the items in the test, students had more knowledge than they wrote in their written responses, and for 29% of the items, students who had the 'correct' written response did not have a complete understanding of the concept being assessed." (p. 10) And politicians & bureaucrats go around chest thumping or in sack-cloth on the basis of the TIMSS results? Ken Ken Appleton, Ph D, M Ed, B Sc, T Cert Associate Professor Faculty of Education & Creative Arts Central Queensland University Rockhampton Qld Australia Email: k.appleton@cqu.edu.au WWW: http://www.edca.cqu.edu.au/appletok Voice: (07) 49309520 Fax: (07) 49309604 (International callers replace leading zero with 61) ******************************************************* This correspondence is for the named person's use only. It may contain confidential and/or legally privileged information. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence in error, please immediately delete it from your system and notify the sender. You must not disclose, copy, retransmit or rely on any part of this correspondence if you are not the intended recipient. Any opinions expressed in this message are those of the individual sender, except where the sender expressly, and with authority, states them to be the opinions of Central Queensland University. Bob V url: Articles of current interest as well as links to a wide range of topics related to the goal of improving learning for all. Donald C. Orlich SMEEC Pullman, WA 99164-4237