+OK 20184 octets Received: from smtp06.nwnexus.com (smtp06.nwnexus.com [206.63.63.45]) by mail1.halcyon.com (8.8.8/8.8.8) with ESMTP id UAA27456 for ; Sun, 14 Mar 1999 20:34:05 -0800 (PST) Received: from ripple.dundee.net (ripple.dundee.net [206.249.104.12]) by smtp06.nwnexus.com (8.8.8/8.8.8) with SMTP id UAA08784 for ; Sun, 14 Mar 1999 20:33:53 -0800 (PST) Message-Id: Date: Sun, 14 Mar 1999 22:50:01 -0500 To: "ClearingHouse" From: Education Consumers ClearingHouse Subject: [education-consumers] A BRIEF DESCRIPTION OF VALUE-ADDED GAIN ASSESSMENT Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Reply-To: Education Consumers ClearingHouse Precedence: bulk Status: ===================================================================== Value-added analysis of student achievement gains promises to revolutionize education by making it possible to track the productivity of individual teachers, schools, and school systems. Now we as parents, taxpayers, and education decision-makers can see for ourselves whether schools are working. It isn't simple, but like financial markets, computers, and lots of other complicated things in our lives, it is worth knowing about. I hope you find the following helpful. John J. E. Stone, Ed.D. Education Consumers ClearingHouse P.O. Box 4411 Johnson City, TN 37602 phone & fax 423-282-6832 e-mail professor@education-consumers.com http://education-consumers.com ******************************** Draft of 3/14/99 A BRIEF DISCUSSION OF VALUE-ADDED ASSESSMENT by J. E. Stone Value-added assessment is a system of statistical analysis that summarizes annual gains in student achievement. It was developed by Dr. William Sanders and pioneered in Tennessee. By tracking achievement test scores on a student by student basis, it yields estimates of the impact had by teachers, schools, and schools systems on average student progress. Value-added assessment permits the consumers of education--everyone from parents to taxpayers to policymakers--to see for themselves how much individual teachers, schools, and school systems are helping students to learn. Not incidentally, it also permits education's decision-makers to assess the effectiveness of everything from the latest curricular innovations, to the preparedness of novice teachers, to the quality of teacher training programs. The statistical analysis employed in value-added assessment is an advanced form of "analysis of variance" called Henderson's "mixed model." It is described in "The Tennessee Value Added Assessment System" by Sanders, Saxton, and Horn (1997) and it produces a "best linear unbiased estimate" of the influence on annual student achievement gains contributed by teachers, schools, and school systems. Of technical significance, value-added estimates of teacher influence are derived from a multi-year "layered" computational model and corrected by a "shrinkage estimate." These two features substantially reduce the possibility of false negative or false positive estimates and ensure that the resulting indicators of achievement gain are as exact as fairness will permit. Compared to other methodologies for computing student achievement gains, value-added analysis is more precise and less vulnerable to manipulations that can distort results. For example, a methodology such as hierarchical linear regression analysis (the type of value-added analysis used in Dallas, TX) will exaggerate the differences attributable to funding and underestimate the differences attributable to teaching effectiveness if a variable such as per-pupil spending variable is prematurely entered into the statistical adjustment of student gains. Preconditions Value-added assessment is statistically robust but the validity of its results depends on certain preconditions. At a minimum, it requires annual testing of students in all grades with a reliable and valid achievement test. Portfolio assessment and other forms of assessment lacking reliability and objectivity will not suffice. Neither will standardized achievement tests that have been revised to enhance their marketability to educators at the expense of diminished academic content. No amount of analysis can transform the substance and meaning of fundamentally flawed data. Perhaps this limitation is best expressed in the statistician's time-honored adage: "Garbage in, garbage out." By way of clarification, however, nothing in the use of value-added assessment precludes teachers from using portfolios or any other form of assessment they deem necessary and desirable. Most educators believe that schooling should serve aims beyond those that can be measured by achievement tests and so they favor a variety of assessments. Parents and the public are not necessarily opposed to these broader aims, but they disagree strongly about education's priorities: Whereas educators think of measured academic achievement as only one outcome among many, parents, taxpayers, and policymakers view it as the indisputable essential. In other words, the parties who fund the schools and who enroll their children are not and will not be satisfied with schooling that fails to produce visible gains in achievement no matter what other benefits such schooling is said to produce. Like an annual audit conducted by an external auditor, value-added assessment is a means whereby the consuming public can see whether the schools are respecting the "owners" priorities and it does so in objective and documented terms. In addition to requiring the use of a valid and reliable achievement test, value-added assessment requires that the items used in each annual testing be fresh, non-redundant, and tied to an underlying linear scale. The forms used at each grade level must include a sufficiently wide range of items such that "ceiling" and "floor" effects are highly improbable. Also the scores produced by the test must be reported on a common scale that spans the range of grades for which the test is appropriate. The purpose of these requirements is to insure that the effectiveness of teachers, schools, and system is tracked yearly, measured in understandable terms, and not artificially limited by the sensitivities of the assessment process. In particular, the use of fresh test items insures that the gains calculated from value-added assessment represent student progress along the full spectrum of curricular objectives and not just improvements in the material sampled by the test, i.e., it discourages teaching to the test. In order to insure a fair assessment, students whose scores are included in an annual value-added assessment must have attended school for at least 150 days and not be eligible for special education services. [Special education students are assessed through "individual education plans."] For a student score to be attributed to a given teacher, the student must have been taught by that teacher for at least 75 days per semester. Advantages Value-added assessment offers several important advantages as compared to other forms of educational accountability. 1. It expresses teacher, school, and school system effectiveness in terms of increases in achievement over previous performance. In other words, each student is compared to his or her own baseline of achievement over a period of several years. By contrast, most present-day educational accountability systems assess effectiveness by comparing current student achievement to a desired level of achievement--one referenced to an established norm or a stipulated criterion. The failure of educational accountability systems to consider gain relative to previous achievement can result in misleadingly negative evaluations for educators who are making substantial but insufficient gains with disadvantaged students or misleadingly positive evaluations of educators who are making mediocre gains with talented and advantaged students. 2. It excludes from the estimates of teacher, school, and school system effectiveness the influence of all pre-existing differences among students. These include but are not limited to race, socioeconomic status, previous learning, intelligence, and all other factors known and unknown that have influenced previous achievement. In contrast to "regression analysis" approaches to student gain assessment, the "mixed model" approach employs statistical "blocking" to remove the contribution of suspected biasing influences. Blocking has the advantage of removing differences without the necessity of measuring and computing the contribution of each of the various factors to be excluded. As counterintuitive as the notion removing differences may seem, value-added assessment has successfully demonstrated that differences can be removed and the playing field leveled. Repeated empirical studies of Tennessee's value-added scores have shown no relationship between annual gains and previous levels of achievement, race, eligibility for free or reduced lunch, or any other of a variety of other student characteristics (University of Tennessee Value-Added Research and Assessment Center, 1995). Although value-added assessment makes it possible to judge educators regardless of the students they have been assigned, it must be noted that pre-existing differences among students are not the only extra-classroom influences that can influence outcome achievement gains. Neither mixed-model analysis nor any other means of educational gain assessment automatically removes the effects that might result from "exogenous" influences arising during the course of the school year. For example, student illness or a natural disaster during the course of a school year might adversely affect achievement gains or, conversely, improvements in family income or the introduction of better community health care might contribute positively to achievement. The influence of exogenous variables can and must be considered--especially as they impact a given school year--and mixed model methodology is able to incorporate such considerations. Happily, however, value-added analysis--properly interpreted--minimizes the need to do so. First, data is averaged over a period of years (permitting positive and negative influences to counterbalance each other) and, second, the gains of teachers, schools, and systems can be compared to the gains of other teachers, schools, and systems that have been exposed to the same or similar influences. 3. Mixed model value-added assessment is able to isolate the achievement effects attributable to an individual teacher so long as students have been taught by that teacher for at least 75 days per semester. As a result, it is possible to assess teaching effectiveness regardless of whether teaching has been undertaken on a departmental basis, a team basis, or a traditional self-contained classroom basis. 4. The influence of a given teacher on student gains is expressed in the form of a "shrunken" or "regressed" estimate, i.e., an estimate that guards against an unfair assessment. In other words, the value-added system takes a very conservative approach to assessing teacher impact and thus insures that those who are identified as effective or ineffective are deserving of their classification. 5. Value-added assessment using mixed-model methodology makes use of all student scores despite the fact that some students will have missed tests and have incomplete sets of data. By contrast, methodologies such as regression analysis exclude students for whom complete data is lacking and thus they typically remove substantial numbers of students when analyses span of 4 or 5 years. Because poorer performing students are usually the ones to miss tests, the exclusion of such students can substantially inflate achievement gain estimates. 6. Value-added assessment permits comparisons of teachers, schools, and systems to national average student gains thus simplifying their interpretation. For example, the gains attributable to a given teacher, school, or system can be reported as 70% or 110% of the annual gain in achievement observed in a national sample. Although percentage of national average gain is a useful indicator of effectiveness, a caveat must be noted. Gain scores depict how well students are progressing beyond their previous skills and knowledge, however, they do not show how students stand with respect to an external benchmark of attainment, i.e., a national norm or criterion referenced standard. For this reason, gain standards alone are not sufficient basis for judging educational outcomes. A complete assessment requires consideration of both value-added performance and performance referenced to an external standard. Tennessee, for example, includes both indicators in its annual school report cards. It is possible to consider both indicators simultaneously. Although not currently used in any state, the annual learning gains produced by a given teacher, school, or system can be compared to the annual learning gains necessary to bring students to an externally referenced benchmark. For example, a school system with a substantial number of disadvantaged students might need to produce learning gains equal to 110% of the national average gains in order to reach national grade level standards by the 8th grade. References Value-added assessment is simple in principle but founded on some rather complex statistics. Several discussions are listed below. The good news is that it has been successfully used in Tennessee for nearly 10 years. Many schools have learned how to interpret it and how to adjust their practices to improve results. Although no one has yet written an account of mixed model methodology suitable for a general audience, its role in value-added assessment has been thoroughly examined and critically reviewed by a number of scholars and policy experts. The following is only a partial list of references on the topic: For a general description see: McLean, R. A., & Sanders, W. L. (1984). Objective component of teacher evaluation: A feasibility study (Working paper No. 199). Knoxville: University of Tennessee, College of Business Administration. Sanders, W. L., & Horn, S. P. (1994). The Tennessee Value-Added Assessment System (TVAAS): Mixed model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8(1), 299-311. Sanders, W. L. & Horn, S. P. (1995). Educational assessment reassessed: The usefulness of standardized and alternative measures of student achievement as indicators for the assessment of educational outcomes. Education Policy Analysis Archives [online serial], 3(6). Available: http://epaa.asu.edu/epaa/v3n6.html Sanders, W. L., Saxton, A. M., & Horn, S. P. (1997). The Tennessee value-added assessment system, a quantitative, outcomes-based approach to educational measurement. In Jason Millman (Ed.). Grading teachers, grading schools, Is student achievement a valid evaluation measure? (pp. 137-162). Thousand Oaks, CA: Corwin Press. Sanders, W. L., Saxton, A. M., Schneider, J. F., Dearden, B. L., Wright, S. P., & Horn, S. P. (1994). Effects of building change on indicators of student academic growth. Evaluation Perspectives, 4(1), 3, 7. For empirical findings with respect to value added model performance see: University of Tennessee Value-Added Research and Assessment Center. (1995). Graphical summary of educational findings from The Tennessee Value-Added Assessment System (TVAAS), (Available from UTVARC , 225 Morgan Hall, P.O. Box 1071, Knoxville, TN 37901-1071) For an evaluation and policy analysis see: Gormley, W. T. Jr., & Weimer, D. L. (1999). Organizational Report Cards. Cambridge, MA: Harvard University Press. For a comprehensive technical review see: Bock, R. D. & Wolfe, R. (1996, January 23). Audit and review of the Tennessee Value-Added Assessment System (TVAAS): Preliminary report. (Available from the Tennessee Office of Education Accountability, Comptroller of the Treasury, State Capitol, Nashville, TN 37243-0260) For technical discussion of "mixed model" statistical analysis see: Harville, D. A. (1976). Extension of the Gauss-Markov Theorem to include the estimation of random effects. Annals of Statistics, 4(2), 384-395. Harville, D. A. (1977). Maximum likelihood approaches to variance component estimation and to related problems. Journal of the American Statistical Association, 72, 320-338. Henderson, C. R. (1982). Analysis of variance in the mixed model: Higher level, nonhomogeneous, and random regressions. Biometrics, 38, 623-640. McLean, R. A., & Sanders, W. L. (1984). Objective component of teacher evaluation: A feasibility study (Working paper No. 199). Knoxville: University of Tennessee, College of Business Administration. McLean, R. A., Sanders, W. L., & Stroup, W. W. (1991). A unified approach to mixed linear models. American Statistician, 45(1), 54-64. Patterson, H. D., & Thompson, R. (1971). Recovery of interblock information when block sizes are unequal. Biometrika, 58, 545-554. Raudenbush, S. W., & Bryk, A. S. (1988). Methodological advances in analyzing the effects of schools and classrooms on student learning. Review of Research in Education, 15, 423-479. Sanders, W. L. (1989). A multivariate mixed model. In Applications of mixed models in agriculture and related disciplines (Southern Cooperative Series Bulletin No. 343, pp. 138-144). Baton Rouge: Louisiana Agricultural Experiment Station. For a review of several current approaches to assessing educational effectiveness: Millman, J. (Ed.). (1997). Grading teachers, grading schools. Is student achievement a valid evaluation measure? Thousand Oaks, CA: Corwin Press, Inc. For sample value-added reports: Tennessee Department of Education 6th Floor, Andrew Johnson Tower 710 James Robertson Parkway Nashville TN 37243-0375 (615)741-2731 http://www.state.tn.us/education/rptcrd98/rcstudent.htm For addition information regarding value-added analysis and its implementation: Dr. William Sanders Director University of Tennessee Value-Added Research and Assessment Center 225 Morgan Hall P. O. Box 1071 Knoxville, TN 37901 Phone (423) 974-7336 Fax (423) 974-7448 wsander2@utk.edu ***************************************************************** ===================================================================== EDUCATION CONSUMERS CLEARINGHOUSE networking and information for parents and taxpayers on the internet Subscriptions & Archives: http://education-consumers.com or You are currently subscribed to education-consumers as: arthurhu@halcyon.com TO UNSUBSCRIBE: Send a blank email to leave-education-consumers-989462S@lists.dundee.net ===================================================================== For less mail, click on the following link and choose 1) a daily digest, 2) a daily list of subjects, or 3) no mail (read postings on Web) http://lists.dundee.net/scripts/lyris.pl?enter=education-consumers For more help & info: http://www.lyris.com/help or .