- Abstract
- Introduction
- Content Classification Schemes
- Content Classification Schemes and the TAAS
- Dimensions of the Essential Elements and Mathematics Objectives
- Essential Elements Across Grade Levels
- Grade Level Distributions of the TAAS Based on Essential Elements
- Content Weakness Examples in the TAAS Exit Exam
- Content "Slippage" due to TAAS Exit Exam Presentation Format
- Examples of Low-Level Items in the TAAS Exit Exam
- Sample Content Drawn from Japan
- Judging the TAAS Exit Exam Items
- Percent Correct by Item Grade Level Rating
- Grade Level Ratings of TAAS Exit Exam Items by Objective
- Technical Issues: Equating Tests for Difficulty
- Technical Issues: Texas Learning Index Scores
- Technical Issues: Curriculum Content Requirements
- Minimum Expectations for "Passing" TAAS Exams
- Distributions of Raw Scores on the TAAS Exit Exam
- Algebra End-of-Course Exam Ratings
- Content Weaknesses in the Algebra End-of-Course Exams
- Potential Discontinuity in Texas Exam Objectives
- Summary and Conclusions

This report evaluates the mathematics assessments in use in Texas -- the Texas Assessment of
Academic Skills (TAAS) mathematics exams and the end-of-course algebra 1 exams -- with respect
to their mathematics content and grade level expectations as a way of understanding their impact on
mathematics achievement in Texas. Several means of evaluating these assessment tools suggest that
only low levels of achievement are being measured. Indeed, the high school exit exam seems more
appropriate to sixth-grade achievement. Thus, the ability of the assessments to measure high levels
of achievement is suspect. This suggests that Texas may not be effectively utilizing the power of a
statewide assessment system to drive up achievement. Setting low expectations may also fail to keep
students on track for later success in algebra and beyond. These effects seem to be a consequence
of an assessment system that is too tightly focused on minimal achievement levels.

The goal of the assessment program in Texas is to measure student progress toward achieving academic excellence. The primary purpose of the state student assessment program is to provide an accurate measure of student achievement . . .

Texas Student Assessment Program Technical Digest

for the Academic Year 1996-1997

The statement of goal and purpose above does not make it explicit that the assessment program is intended to promote greater academic achievement statewide in Texas, or the desire to bring the achievement of Texas students up to "world-class standards" or similar lofty ambitions. Surely, promoting greater achievement is a central objective behind the implementation of the assessment system in Texas.

This report evaluates the mathematics assessments in use in Texas -- the Texas Assessment of Academic Skills (TAAS) mathematics exams and the end-of-course algebra 1 exams. Particular attention is paid to the mathematics content and grade level expectations of these exams as they relate to stimulating greater achievement in mathematics.

Statewide examinations in mathematics carry enormous potential as a tool to improve achievement at a relatively low cost. They can be useful to students to gage their own progress and their relative strengths and weaknesses, and it is clear that the exams can provide a motivational influence. They are also useful in evaluating mathematics education. For example, they are helpful in identifying instructional methods and curriculum materials that are highly successful. Thus, these assessment devices are also motivational at the system level. Finally, they provide a degree of objective validity to the high school diploma and other indicators of academic progress - something that is needed after years of grade-inflation and social promotion have deteriorated the utility of these traditional indicators of academic success.

The introduction of assessment programs, particularly high-stakes assessments, can provide a strong source of motivation. This has become increasingly important as more and more teachers experience difficulty in motivating their students for academic success. The impact is widely evident in Texas where students spend extra time in test-preparation, textbooks address the assessments directly, and teachers strive to prepare their students for success on the examinations. Private industry even enters the picture by providing study materials and instruction specifically for the exams.

The potential benefits of a powerful statewide assessment system in mathematics are not guaranteed.
Great care must be taken in designing these assessments, and the associated reporting mechanisms,
so that the greatest possible benefits to mathematics achievement will result. With this in mind, this
report evaluates the mathematics content and difficulty of these assessments. The goal is to clarify
and interpret the nature of these assessments as a way of understanding their impacts on mathematics
achievement in Texas.

Various organizational schemes have been employed over the years to classify the content of mathematics education. Traditionally in the U.S. we have thought about mathematics education in terms of the subject areas of mathematics - things like arithmetic, algebra, geometry, and trigonometry. In more recent times, we have seen a variety of other sets of taxonomic categories proposed. These may be called strands, objectives, domains, or elements, but the underlying purpose is the same - to provide a structure for studying and thinking about the content of the mathematics curriculum.

In general, these systems provide a high-level structure of categories that runs across all grade levels. This has the obvious advantage of providing a unified vision of the curriculum across grades, and may help to clarify how particular content areas build over the years. Yet, the use of any such system must be approached with caution. It is clear, for example, that the meaning or interpretation of any one category changes considerably over the grade levels in terms of the kinds of material that it addresses. Also, none of the classification schemes is very successful in that particular mathematical ideas or procedures or problems are often difficult to assign to one and only one category.

In reality, all of these classification schemes are artificial and inexact. They should be thought of as nothing more than crude organizing methods that can serve as a helpful technique for the study of the curriculum. However, more often than not the system of classification takes on a life of its own and begins to influence both curriculum and assessment. There can be positive benefits that result. For example, discontinuities in the curriculum may be discovered and corrected. On the other hand, there can be negative consequences as well. One possibility is that there may be pressure to weight each content area equally across the grade levels. This seems certain to misrepresent what is important year to year, and gives far too much authority to a classification scheme that has arbitrary and inexact characteristics. For example, there is every reason to expect that instruction in the operations of arithmetic will be predominant over the use of algebraic manipulations in the early grades and that this will tend to reverse as students progress through the grade levels.

The dimensionality of the classification scheme may be used in attempts to change the face of the
curriculum rather than to describe it. For example, as new categories are added, there will be
a tendency to take emphasis away from "old" categories like arithmetic and algebra. While changes
to the curriculum are not necessarily negative, each must be evaluated carefully on its own merit.
There is no merit in curriculum changes introduced simply to satisfy the needs of the classification
system itself.

The overall achievement goals in Texas have been divided into two, roughly parallel classification systems that are related to the development of the TAAS. These are illustrated below as a hierarchy of specifications that runs from overall achievement goals down to the actual test items.

Broad categories or strands are identified in the seven Essential Elements and in the thirteen TAAS Mathematics Objectives. The Objectives were generated subsequent to the Essential Elements, and are intended to encompass the materiel in the Essential Elements.

Grade level specifics are provided by the details of the Essential Elements and the Instructional Targets specified for the TAAS.

A cross-reference between the grade-level
details of the Essential Elements and the
Instructional Targets for the TAAS has been
provided by the Texas Education Agency in
the __TAAS Mathematics Objectives and
Measurement Specifications__. This cross-reference provides a means to equate the
two classification systems.

Test items are generated to match the TAAS Instructional Targets.

The number of test items in various content areas is based on weights for each Objective. Thus, relative emphases in the TAAS are determined by Objective.

The two classification schemes should be studied together to understand the content emphases.

Both the Essential Elements and the Mathematics Objectives have strands that run across all grade levels. The dimensions of the two classification schemes can be roughly identified by the names in the table below:

## Essential Elements |
## TAAS Mathematics Objectives |

1 Problem solving | 1 Number concepts |

2 Patterns, relations, and functions | 2 Relations, functions, and algebraic concepts |

3 Number and numeration concepts | 3 Geometric properties and relationships |

4 Operations and computation | 4 Measurement concepts |

5 Measurement | 5 Probability and statistics |

6 Geometry | 6 Addition |

7 Probability, statistics, and graphing | 7 Subtraction |

8 Multiplication | |

9 Division | |

10 Estimation | |

11 Solution strategies | |

12 Mathematical representation | |

13 Evaluation of reasonableness |

By counting the cross-references between the details of the Essential Elements and the Instructional Targets for each Mathematics Objective and summing these across grade levels, a rough map between the Essential Elements and the Mathematics Objectives can be generated. The results of this mapping are represented graphically below.

By inspecting the figure up and down for each Mathematics Objective, the Essential Elements addressed by the Objective can be identified. There are no great surprises in this mapping.

By inspecting each row of the figure, the Mathematics Objectives that load on each Essential Element can be identified. An admittedly oversimplified summary is presented below.

## Essential Elements |
## TAAS Mathematics Objectives |

1 Problem solving | 11 Solution strategies
12 Mathematical representation 13 Evaluation of reasonableness |

2 Patterns, relations, and functions | 2 Relations, functions, and algebraic concepts |

3 Number and numeration concepts | 1 Number concepts |

4 Operations and computation | 6 Addition
7 Subtraction 8 Multiplication 9 Division 10 Estimation |

5 Measurement | 4 Measurement concepts |

6 Geometry | 3 Geometric properties and relationships |

7 Probability, statistics, and graphing | 5 Probability and statistics |

There are a few obvious omissions from the table above. For example, Objective 2 - Relations, functions, and algebraic concepts - also taps Essential Element 4. Also, Essential Element 7 - Probability, statistics, and graphing - relates to the probability and statistics Objective, but shares some relationship to other Objectives.

Of primary importance in this mapping is the fact that the problem solving and the operations Essential Elements have been expanded to a more predominant position in the Objectives. This is important because the distribution of test items is set by the Objective, not by the Essential Element.

This means that substantially more than 1/7th of the items will address Essential Element 4 - Operations and computation. This is altogether reasonable for the early grades where an emphasis on the operations of arithmetic is indicated. Whether it continues to be appropriate at higher grade levels is open to question and depends in part on the way the Objectives are elaborated. In particular, the transition to algebraic subject matter should appear in later grades.

There will also be an increased emphasis on Essential Element 1 - Problem solving. This is further
augmented by the fact that it is related to Objectives 11 and 12 since these Objectives are given more
items in the TAAS. This is reflective of the desire in Texas to provide more emphasis on "problem
solving and complex thinking skills." On the other hand, since these objectives are more difficult to
define, they will also be related to the Essential Elements for operations and computation and
probability, statistics, and graphing.

The mapping discussed above was collapsed across grade levels. Indeed, there is good reason to believe that relative emphases would vary across grade levels.

To address this issue, the references to Essential Elements were expressed as proportions within each Objective at each grade level. These were then weighted by the item counts for each Objective at each grade level so that weights for each Essential Element could be estimated. These were then expressed as a percentage of emphasis in the corresponding TAAS exam. The relative weights are illustrated in the figure below.

As anticipated, Essential Element 4 - Operations and computation - receives the largest emphasis at each grade level. Also as anticipated, Essential Element 1 - Problem solving - has the next largest weighting each year. The remaining Essential Elements have roughly equal weights. Moreover, this pattern is relatively consistent from grade 3 through the high school exit exam.

It is evident at this point that the system of specifications - Essential Elements, Objectives, Instructional Targets, and item counts per objective - has exerted an influence on the distribution of emphases in the TAAS. In part, this influence is entirely appropriate, such as the emphasis on operation and computation especially in the early test grades. On the other hand, the relatively consistent weights for the other Essential Elements are cause for concern. In particular, the lack of a demonstrated growth in the emphasis on algebraic content over the grades is worrisome. Similarly, one would expect a transition in the relative weights for measurement and geometry across the grades. The seriousness of these issues depends in part on the changing definitions of the strands across the grade levels in terms of the Instructional Targets and the actual test items that are generated. In any case, this information provides cause for concern that content emphases may be unduly influenced by the classification systems used to generate test specifications and test items.

Following the procedure of tracking references between Essential Elements and Instructional Targets, it is possible to relate item specifications for the TAAS to mean grade levels given in the Essential Elements. The results of this approach are illustrated below.

It is evident in the figure that mean item specifications in the TAAS lag a year behind the expected grade level in the Essential Elements, and that the expectations for the exit exam are nearly identical to the grade 8 exam. Both of these phenomena reflect processes intentional in the test design. Although legally mandated as a criterion-referenced assessment, item specifications generally address Essential Elements distributed across a range of grade levels - at grade level, one year behind, and two years behind. Thus, the average lag of about one grade level. In addition, special consideration was made in designing the exit exam since high school students are not required to take higher level mathematics courses in order to graduate. Consequently, the exit exam appears in this analysis to be undifferentiated from the eighth grade exam.

Two points need to be clarified regarding the figure above. First, the minimum passing rates on these exams are approximately 70% correct. Thus, students will need to get at least some of the grade-level appropriate items correct to pass. This would have the effect of bringing the exam levels closer to the "on-target" line represented in the figure. However, these are also multiple-choice items with chance correct rates ranging from 20% to 25%. With chance taken into account, the grade-level estimates relative to the Essential Elements presented in the figure should be approximately correct.

Thus, relative to the Essential Elements, the TAAS exams appear to be behind grade level and the exit exam appears to be similar to the grade 8 exam.

However, these conclusions are based upon information provided by Texas Education Agency
documents regarding item specifications and the Essential Elements. These conclusions do not speak
to how the Essential Elements relate to improving achievement overall, and they do not address the
relationship between the specifications and actual item content and difficulty levels. Rather than
address each step in the hierarchy of achievement specifications that ultimately results in test items,
content issues relating to the test items themselves are presented below.

An inspection of TAAS exit exam items reveals several achievement areas that are either missing or weakly represented in TAAS items. The following elements are provided to illustrate this finding, and are based upon the inspection of 240 TAAS items -- 60 per year for 4 years of exams.

* Addition and subtraction of fractions with unlike denominators*: Three addition items and
three subtraction items were found. Their denominators are simple small integers in each
case.

* Multiplication and division of fractions or mixed numbers*: There were no instances of the
multiplication of two fractions. There was one instance of the division of a mixed number by
a fraction.

* Terminating and repeating decimals*: There were no items related to this distinction.

* Factors of numbers*: There were no items found that directly addressed the factors of
numbers, prime and composite numbers, greatest common factor or least common multiple.

* Powers, roots, and exponents*: There were two items found that called for the squares of
integers (15 and 40). There was one item found that called for finding the two integers that
bound the root of a number.

* Properties of real numbers*: There were two items found that dealt directly with the
distributive property asking for the equivalence of two expressions.

* Absolute value and negative numbers*: There were no items found that dealt with absolute
value. There was one item found that required sorting signed integers, one that asked about
the distance between two altitudes one of which was below sea level, and one that required
evaluating an expression containing a sum where one replacement value was negative.

* Area and volume*: There was one item found asking for the lateral surface area of a cylinder
(although the formula is supplied). There was one item found asking for the volume of a
rectangular prism.

* Median and mode*: There was one item found that asked for a median.

* Solving equations*: There were two items found that asked for the solutions of equations:

1.5*x* - 6 = 4.5 and c = $15 + $7.50p when c is $45

The above content areas provide a flavor for elements of mathematics that are not well represented
in the exit level TAAS.

Multiple choice items are often difficult to write because students have the opportunity to work backwards from the available response choices. In other cases, item solutions can often be found by methods other than those intended to tap target objectives. The result is that what casual inspection may suggest to be the content addressed by an item is often different from the actual solution methods available to students. Three examples taken from the TAAS exit exam follow:

Students are asked for the ordered pair that represents the intersection of two lines given by linear equations. However, the lines are clearly graphed. This problem thus only requires being able to identify a point in the coordinate grid.

Students are told that two ladders are leaning against a building at the same angle. They are given the length of both ladders and the distance from the ladder base to the wall for the longer ladder (They are also informed in a figure that the ground forms a right-angle with the wall). Students evidently are to use reasoning about similar triangles and proportions to determine the distance from the base of the shorter ladder to the wall. Unfortunately, only one response choice is reasonable given the illustration that accompanies the problem. In fact, all incorrect response choices greatly exceed the entire length of the shorter ladder.

Three items appear to require the use of the Pythagorean theorem to solve for unknown lengths of right triangle sides, or at least the recognition and application of Pythagorean triples. However, the figures are drawn reasonably close to scale and only one response alternative for each item is reasonably possible given the figure.

Thus, some of the most difficult content areas addressed in the TAAS exit exam have simpler
alternative solution strategies available that students are likely to employ.

Looking at actual TAAS exit exam items helps to clarify the nature of these tests. Some of the items discussed above are among the most advanced TAAS items in terms of the mathematics content. To illustrate the lower end of the spectrum, some of the items with the lowest level of mathematics content are listed below.

The total attendance recorded at the 1984 Summer Olympic Games in Los Angeles, California, was 5,797,923. What is this number rounded to the nearest thousand?

Mrs. Ramos has a plastic cube on her desk that holds photographs. There is a picture on every face of the cube except the bottom. How many pictures are displayed on the cube?

What is the approximate length of a new pencil before it is sharpened? (Response choices are 1.9 millimeters, 19 millimeters, 19 centimeters, and 1.9 meters)

Devon's house is on a rectangular block that is 330 yards long and 120 yards wide. What is the distance around his block?

Kenyon is 5 feet 6 inches tall. His sister Tenika is 7 inches taller than he is. How tall is Tenika?

At a restaurant Steve ordered food totaling $6.85. If he paid with a $20 bill, how much change should he receive?

Certainly every high school graduate should be competent at solving problems of this nature.
However, these items do not reflect the kinds of skills and knowledge that are grade level appropriate
for high school students. There can be little question that these items are more appropriate to
examinations used in much earlier grades.

In Japan, 12-year-olds are given a mathematics examination that consists of 225 story problems. These items show a depth of content for Japanese 12-year-olds that is striking in contrast to the TAAS exit examination items. Some of these items have been translated into English (Pacific Software Publishing) and are reproduced here by permission.

How many 'C' balls does it take to balance one 'A' ball?

Jenny wanted to purchase 2 dozen pencils and a pen. Those items cost $8.45 and she did not have enough money. So she decided to purchase 8 fewer pencils and paid $6.05. How much was a pen?

Hose A takes 45 minutes to fill the bucket with water. Hose B can do the same in 30 minutes. If you use both hoses, how long will it take to fill the bucket?

A job takes 30 days to complete by 8 people. How long will the job take when it is done by 20 people?

Bob, Jim and Cathy each have some money. The sum of Bob's and Jim's money is $18.00. The sum of Jim's and Cathy's money is $21.00. The sum of Bob's and Cathy's money is $23.00. How much money does each person have?

Tom's mother is 30 years old. The three children are 5, 3, and 0 years old. 12 years later, the total age of Tom's mother and father is twice as much as the total ages of all three children. How old is Tom's father?

Ellen baked cookies of the neighborhood children. She gave each child 6 cookies and she had 7 cookies remaining. So, she gave one more cookie to each child, but, was one cookie short. How many cookies did she bake in total?

It is 6 miles between Joe's house and Larry's house. Joe and Larry started to walk to each other's houses at noon, meeting at 12:30. Joe walked 2 miles per hour faster than Larry. How fast did Larry walk?

To assess the target grade level of the TAAS exams against external criteria, individual exit exam items were evaluated as to grade level based on the newly established California Mathematics Standards. These standards provided a desirable benchmark for several reasons:

- They were designed carefully to be on-track with the best international competition, including Japan and Singapore.
- They are perhaps the most highly detailed of all sets of state mathematics standards, greatly facilitating item evaluation.
- They have been judged as the best available mathematics standards among all sets of state standards, even exceeding those from Japan. (R. Raimi and L. Braden, State Mathematics Standards)

Two of the authors (P.C. and D.K.) independently judged the grade level of every TAAS Mathematics exit exam item for the prior four years using the California standards as a guide. Although some degree of subjectivity was involved, these ratings proved to have a reasonably high level of rater reliability (r=.813). When the two ratings did not concur, they were averaged. The average distribution of item grade levels on a TAAS exit exam is illustrated below.

The ratings against the California Mathematics Standards yielded a mean grade level of 5.3 for the TAAS exit exam. The most advanced TAAS exit items were judged as equivalent to the California grade 7 standards.

Admittedly, the California Standards are set at a high level, being roughly equivalent to progress in Singapore and Japan. Nonetheless, the low estimated grade level is striking. Moreover, the California Standards are designed to complete the content of pre-algebra by grade 7 so that students will be ready to study algebra and geometry in grades 8 and above.

This finding raises the possibility that students could pass the TAAS exit level examination and still
not be ready for the study of algebra. This possibility is consistent with the fact that Texas students
enjoy greater success on the TAAS exams than on the algebra 1 end of course exam.

If the grade level ratings of TAAS exit exam items based on the California standards are a valid indicator, then TAAS exam scores would presumably be elevated by the inclusion of items below grade level. Likewise, the percentage of students with correct answers on individual items ought to vary as a function of item grade level ratings. Analysis of data for individual items from field tests and actual tests in Texas shows this is indeed the case as indicated below.

These results suggest that passing rates on the TAAS exit exam would be considerably lower if all
items fell at the 7^{th} grade (or higher) level based on the California standards.

To study grade level expectations across content areas on the TAAS exit exam, the mean grade level rating was computed for each of the Mathematics Objectives indicated on the TAAS. This is possible because each TAAS item is tied to a Mathematics Objective. The grade level means are illustrated below.

Mean grade level ratings were significantly lower for the Measurement, Addition, and Subtraction
Objectives than for the other Objectives (p<.05 in each case). This may not be a surprising result as
the topic areas of measurement, addition, and subtraction might be expected to enter the curriculum
earlier than some of the other Objectives. Although more complex material could enter these content
domains, this does not appear to be the case from the grade level ratings. Thus, it appears that the
emphasis given by the weights in these three Objectives is grade level inappropriate in the exit exam.
Likewise, Objectives 2 and 3, which should reflect algebraic and geometric material, do not appear
to reflect more advanced content as might be expected.

The proportion of students meeting the minimum expectations on the TAAS exams has been rising. This is a positive indication, but the conclusion relies heavily on equating derived scores for test difficulty differences year to year. If the tests cannot be shown to be of equivalent difficulty (in derived scores), than the finding of improvement may be suspect.

TAAS exams are roughly equated for difficulty in the process of item selection. Furthermore, derived
scores are further equated through statistical procedures that are mathematically sound. Since tests
contain only a finite number of items, at least small fluctuations in the difficulty associated with a
passing (minimum expectations) score are certain. Some further inaccuracy in the process of equating
difficulty must exist. However, the degree of accuracy in the equating process is not documented in
the __Technical Digest__. Furthermore, the tests are equated year to year, so that errors in this process
may be compounding over time.

To address the equivalence of TAAS exit exams year to year, a comparison among the mean item
grade-level ratings based on the California standards was computed for four years of exams. While
this does not address the statistical equating process, it does provide an indication of relative item
difficulties across years. The results of this test were not statistically significant (p = .46), with grade
level means hovering around 5.3 for each of the test years. Thus, this method supported the notion
that the tests were of roughly equal difficulty levels across years.

Another design concern relates to the use of the Texas Learning Index (TLI). The design of the TLI
is extremely useful in that it provides scores that students can use to gage their relative achievement
progress over the grades. Although a valuable resource, the TLI should be recognized as an
essentially norm-based score for grades 3 to 8. This means that any unevenness in achievement
across the grade levels in the norming year will be perpetuated in the TLI score distributions in
succeeding years. For example, if the gain in mathematics achievement in the sixth grade was low
as measured in the spring of 1994 due to some curriculum defects at the time, then a "passing" TLI
score of 70 for grade 6 will continue to reflect this low level of achievement in future years. It is even
possible that these future scores would be seen as endorsing achievement rates that would not be
found if a criterion-referenced system was in place.

During the screening of potential TAAS items, a review is conducted to insure that the item content
is reasonably well addressed by the existing curriculum in Texas. Students are not to be tested on
material they have not had the opportunity to study. However, this deviates from criterion-referenced
assessment by adding the stipulation that criteria must be covered by the curriculum. The stipulation
inhibits the ability of the assessments to drive advancements in the curriculum. There may be a risk
of "curriculum stagnation" in which deficiencies in the curriculum tend to be perpetuated.

There are several reporting methods summarizing TAAS performance data. These include average scale scores, TLI scores, Texas Percentile Rank scores, Normal Curve Equivalents, and the percentage meeting minimum expectations. However, a great deal of emphasis is placed on the figures for the percentage of students meeting minimum expectations. The inherent risk in setting a "floor" or minimum requirement is that it will effectively become a "ceiling," meaning that there may be a tendency to set a cap on achievement levels.

The basis of this thinking is that an emphasis on bringing up minimum achievement levels will tend to focus both the curriculum and instruction at levels that are too low to provide the greatest benefit to student achievement overall. Simply put, there is a risk that designating these targets will leave many students with lower achievement prospects than they are capable of.

This concern is amplified by the level of TAAS Instructional Targets relative to the Texas Essential Elements, the low achievement expectations evident in the items themselves and in comparison to the test for Japanese 12-year-olds, and the grade level ratings of TAAS items relative to the California Mathematics Standards. If the TAAS assessment levels are lower than optimal and there is a focus on minimum achievement relative to those assessments, there is an inherent risk that curriculum and instruction will be swayed toward sub-optimal levels as a result of the assessment process. Thus, it is possible that the TAAS exam system is not nearly as effective as it might be in promoting greater mathematics achievement statewide in Texas.

The presence of the TAAS examinations, and the minimum expectations used for graduation
requirements, make this a high-stakes assessment system. As a consequence, a great deal of attention
is given to preparation for TAAS exams. This includes material in school textbooks, classroom
instruction time, and a sector of private industry that supplies materials and instruction. This
illustrates the power of a high-stakes examination system. The findings given above suggest that
Texas may not be making the greatest possible use of this power.

The distributions of raw scores on the TAAS exit exams are give below for three test years. These show H3 negative skews. The presence of negative skew is not surprising given that the initial target of 70% correct is surpassed by a majority of students. However, the degree of skew is sufficient to suggest that the TAAS cannot function effectively in the identification of high achievement levels, and ceiling effects in the distribution are obvious. Since the exams do not differentiate well at higher achievement levels, we cannot tell whether or not the implementation of the assessment system is leading to similar ceiling effects in actual achievement. However, the lack of sensitivity to high achievement levels would suggest that the TAAS will not be effective at motivating achievement for a good proportion of students.

In addition to the TAAS exams, end-of-course examinations are also available in several content areas. In mathematics, students enrolled in introductory algebra take the algebra 1 end-of-course examination. They may use a passing score on this examination as part of their high school graduation requirement instead of the TAAS exit exam in mathematics.

In order to comment on both the content coverage and difficulty level of the Algebra 1 end-of-course exams, all of the exam items for four successive years were evaluated by one author (W.B.). To accomplish this, a list of common algebra concepts and skills was prepared by looking through several standard pre-algebra and algebra 1 textbooks and the examinations themselves. These were chosen so that an objective analysis of the content of each item could be completed.

Each item also received a 1-5 rating with the following identifiers:

1 Prior to Pre-Algebra

2 Pre-Algebra

3 Low Difficulty Algebra

4 Moderate Difficulty Algebra

5 High Difficulty Algebra

A rating of "3" represents the level of standard but easy algebra, the level of universal mastery of the content of algebra 1. A "2" represents standard pre-algebra, say at the level of Saxon Algebra 1/2 or Japanese Grade 7 Math (two of the sources used). A "1" is below that, roughly fourth or fifth grade math competence without even algebra readiness implied.

Going in the other direction, a "4" represents problems that require a more sophisticated level of algebra competence for solution. For example, clearing fractions and then solving a linear equation, or finding an obvious least common multiple of the denominators of two rational functions and adding them. A "5" level item is beyond that, though still appropriate for a broad screen, end-of-course algebra test. Examples would be something like clearing the fractions and then solving a resulting quadratic equation or solving an equation involving radicals that requires squaring both sides twice or that requires rejection of one or more of the "solutions" that were introduced in that process, or simplifying complex rational functions that involve quadratic (though easily factorable) expressions.

Generally speaking, if it appeared that an algebra-ready student who had not yet studied the subject should be able to solve the problem by inspection or by testing the given answer choices rather than using actual techniques of algebra, the item was rated at the pre-algebra level. That was especially true if that seemed to be the most likely approach to the problem, for example without clearing fractions, squaring both sides, or factoring as would be algebraically indicated. Some of the items would receive higher ratings if response grids rather than a multiple choice format were used.

The distribution of item ratings for the Algebra 1 end-of-course exams is indicated below.

The mean rating across all items was 2.46 on the 5-point rating scale. As is evident in the figure, this means that the exams are primarily a combination of pre-algebra material and algebra at a low difficulty level. Mean ratings did not change significantly across test years (p=.24).

Approximately 14% of the items would have received higher ratings had alternative, less-advanced
solution strategies not been available for the item in context (mostly if response grids could replace
the multiple choice format). Thus, if these alternative solution strategies were not available, the mean
rating would increase to 2.62 on the 5-point rating scale.

It is disturbing to see almost no factoring necessary on the tests. In fact, there is almost no algebraic simplification or algebraic arithmetic operation competency tested. Factoring appears to be intended in several items on each test, but it is always possible to get around it, sometimes trivially as with solution checking or with "factoring blocks" misused to give the actual factorization in this artificial form. Factoring simple polynomial expressions when possible is a very helpful tool in subsequent courses successive to this one so the idea needs to get more emphasis. The omission of factoring in this test sends a message to the state's algebraic teaching community that symbolic manipulation skill does not matter.

Another area of concern is the need for better and more traditional word problems. The word problems that are presented mask a lack of algebraic depth. Direct and inverse variation (find the constant, for example, without a given model to follow) or, "How much water should be added to 20 quarts of a 30% solution to obtain a 14% solution," have H3 practical application in science. Less practical, perhaps, but excellent mathematical reading and algebra training are integer number problems such as, "The quotient of the successor of an integer number and one-third of the number is 4. What is the number?"

Finally, graphing linear equations in two variables is far too poorly done for the space that it consumes. Standard and useful items such as finding the slope-intercept form of a line that contains two given points, or a point and perpendicular to a given line, are almost nonexistent. Lots of pictures of graphs take up lots of pages but are not nearly as informative as a picture of one line with a couple of specified points for definiteness and then a response-grid question for the slope, the y-intercept, or the x-intercept. A good level "3" question here would be two lines given in standard form and graphed that appear to intersect at an integer pair, say (3,2). The question would then be something like, " These lines intersect at a point close to (3,2) but we cannot tell from the graph if that is precisely correct. Which of these is the correct value for the x-coordinate of the point of intersection?" A level "5" question along this line would be to not give equations of the lines but to give the coordinates of two points on each line so the first step is to find equations of the lines and the second is to use them to find the actual point of intersection. Questions like this can be multiple choice and still be testing the underlying algebra concepts. If they are not a bit more complicated, such as these suggestions are, the ability to just check the answers defeats confirmation of the intended algebra concepts.

The following points with respect to content should be noted.

1. There are too many formulas given. Most, such as the area of a rectangle or the slope of a line, should be assumed to be known.

2. There are no rational function reduction or arithmetic items.

3. There is almost no confirmation of algebraic arithmetic skills at all: products of binomials, reduction of fractions involving exponential monomials, etc.

4. There are no problems that require factoring polynomials, even "taking out" a common factor.

5. Standard word problems that lead to algebraic solution are inadequate in number and in depth.

6. Although there are enough items, linear graphing is inadequate; e.g., there are no items that involve the slopes of perpendicular lines.

7. Many items can be done by checking given answer choices or by inspection. Even just a "none of the above" would be helpful.

8. The distance formula is never used and all Pythagorean Theorem items can avoid it.

9. Radical equation exercises are trivial or don't exist.

10. Scattergram questions are trivial and should not be included.

11. "Which BEST describes . . ." language is used even when a perfect fit is among the choices.

12. The response-grid format is insufficiently used. It is down to one item in 1998.

In summary, this test is more of an algebra readiness test than it should be. There are no items that
require more than the most trivial symbolic manipulation, a standard part of algebra, and many of the
items only appear to require an algebraic solution. They can be done, and will be done, by inspection
or by testing the given answer choices. The statistics and probability questions are not at algebra
level under the most generous interpretation.

By combining together much of the information given above, a preliminary look at the progression through algebra 1 in the Texas assessments is possible. This method is admittedly tentative, but may be useful for understanding the progression to algebra in Texas.

To compile this data:

- California grade-level estimates for TAAS grades 3 to 8 were assumed to be discrepant from
the Essential Elements reference mapping in a manner that was proportional to the difference
seen in the exit exam.
- Algebra exam items rated at 3 and above were assumed to reference California grade 8. Items rated at 2 were assumed to map to California grade 7. Items rated at 1 were assumed to map to California grade 5.

The results of the computations based on these assumptions are indicated below.

This preliminary model suggests that there is a discontinuity between the grade 8 (or even the exit level) TAAS exam and the algebra end-of-course exam. This is true even though the ratings placed the algebra end-of-course exam at a low level vis-a-vis California Standards (grade=7.34).

The implication of this finding is that Texas students who are minimally competent through grade 8 may have a difficult time mastering the content of algebra since this constitutes roughly a two-year jump in the model above. This is consistent with the fact that a smaller proportion of Texas students meet minimal expectations for the end-of-course algebra exam than do so for the TAAS mathematics exams.

While this finding should be considered preliminary based on the required assumptions, the
implications are quite serious and demand further study. The possibility exists that the assessment
system is not targeting the skills and understandings that are sufficient for success in algebra even at
the modest levels addressed by the end-of-course exam. If this is true, then the assessment system
is not contributing to the improvement in achievement that might otherwise be obtained.

An Anecdote: It was a warm evening in Arizona, and some of the vacationers at the hotel were lounging by the Jacuzzi. A 14-year-old from Texas chanced to strike up a conversation with a math instructor from California. The instructor asked the teenager about the TAAS exam in Texas. "Oh, that . . ." replied the youth, "well, the math is very easy and that's all they teach us, so it gets pretty boring."

Texas has maintained a relatively stable statewide mathematics assessment system, and there is evidence to suggest that some improvements have been made in achievement. The percentage of students meeting minimum expectations on the TAAS has been going up, and the National Assessment of Educational Progress (NAEP) scores for Texas look promising relative to the rest of the country. But, judging by international comparisons like the Third International Mathematics and Science Study (TIMSS), doing better than other parts of the U.S. may not be saying much.

The evidence reviewed above is consistent in indicating that Texas assesses mathematics achievement at a low level. Indeed, the content of the high school exit exam is more appropriate as a target for the sixth grade. Students could pass this exam and yet have difficulty with the Japanese exam given to 12-year-olds.

Whether or not establishing low targets is a good way to stimulate achievement in mathematics statewide is a critical strategic question. The consequences for far too many students may be similar to the experience of the teenager in the anecdote above. Low-level objectives are not consistent with the high expectations for mathematics achievement that are being called for from one end of the country to the other. Low-level objectives are unlikely to bring student achievement up to the level of our international competition.

The "teaching to the test" phenomenon is often used in an undifferentiated attack on large scale assessment in general. But, this position only belittles the teaching and learning efforts that are motivated by assessment. The assessment system is an effective stimulus for learning to the extent that it promotes greater achievement. Students who acquire the knowledge and skills they need to do well on a challenging test have indeed been learning. Testing itself is not evil, but bad tests can be evil in terms of their consequences. They are a misuse of the power of assessments.

The review of the examinations used in Texas is suggestive of a system wherein the power of statewide assessments has focused on raising achievement only to a minimal level. The low expectations evidenced by the exam items themselves, and the fact that instruction is geared toward these exams, is cause for concern. This concern is amplified by the indications that the system design may be insufficient to promote greater success in algebra and higher level mathematics courses.

Without successful achievement in algebra and beyond, students are failing to reap the rewards of
mathematics education. Algebra is often referred to as a gateway course since it opens the door to
the opportunities and benefits that are associated with achievement in mathematics. __Mathematics
Equals Opportunity__, a recent report from the U.S. Dept. of Education, cited the need to:

*Provide all students the opportunity to take algebra I or a similarly demanding course that includes fundamental algebraic concepts in the 8th grade and more advanced math and science courses in all four years of high school.**Build the groundwork for success in algebra by providing a rigorous curriculum in grades K-7 that moves beyond arithmetic and prepares students for the transition to algebra.**Ensure that all students, parents, teachers, and counselors understand the importance of students' early study of algebra as well as continued study of rigorous mathematics and science in high school.*

The findings reported above raise concern that the mathematics assessment system in Texas is not designed to meet these objectives.

There will always be advantaged students that find other routes to high achievement, but this is not true opportunity for all students. There is an incessant relationship between socio-economic status and achievement in mathematics that bodes poorly for the disadvantaged. However, it is a statistical given that the strength of this relationship will appear to decrease if the top of the achievement distribution is truncated by using a test that is too easy. The inherent risk is that data generated by the statewide assessment system would be misleading with respect to equity issues in education.

The power of a statewide assessment system can be exerted in many ways. In Texas, a sharp focus has been drawn by the minimum requirements for high school graduation. Even the score reports for the early grades are keyed to these minimum graduation requirements as the long-term objective. As a result, the public evaluations of schools and districts tend to focus on achievement rates for these minimal objectives as well. But such requirements necessarily target low achievement objectives since no system that fails most students would be socially acceptable.

Alternative systems of rewards and consequences should be investigated that can target higher achievement levels. For example, the public reports for the performance of schools and districts, even without mandated consequences, are powerful motivators at a system level. Greater emphasis in these reports should be given to achievement relative to higher objectives. Reward systems for increases in high-level achievement rates should also be considered.

Any design that emphasizes high achievement levels needs to be carefully monitored to insure that the motivational factors do not lead toward artificial claims of success. This is exemplified by an algebra test that is, in reality, a combination of pre-algebra and low-difficulty algebra topics. High achievement goals must represent honest levels of high achievement, not just high achievement in name only.

In Texas, as in many other states, there has been a call for an emphasis on "problem solving and complex thinking skills." But, also as in many other states, there is difficulty in operationalizing these goals in a way that is objective, can be measured, and shows progressive development across the grade levels. For this reason, these content areas require special scrutiny. There will be a tendency to think of objectives as high-level if they contain language about problem solving and complex thinking. However, this sort of material may not represent advanced achievement in practice. The development in these areas must be tied to more explicit objectives, such as "differentiating between relevant and irrelevant information in a problem situation requiring solution by multiplication or division." In this way, the objectives in problem solving and complex thinking need to be tied to the mathematics content objectives at the appropriate level.

Much of the above discussion has focused on the need to employ the power of a statewide assessment system to promote high achievement levels. At more modest levels, there is evidence to suggest that the system of assessments in Texas has been effective. By comparison to California's new standards, most Texas high school graduates can achieve at least at a fifth- or sixth-grade level. But, the possibility exists that even this benefit may have run its course and that any further gains will reflect diminishing returns.

In summary, the system of mathematics achievement assessment in Texas emerges as a powerful
model but one that is too highly focused on minimal achievement. The incentives for improvement
that accompany the statewide assessment system do not emphasize high achievement sufficiently.
In fact, the design of the assessment devices themselves doesn't even permit the measurement of high
achievement levels with any degree of accuracy. Without a substantial adjustment to the objectives
that are evidenced by the exam items themselves, it seems unlikely that the assessment system will
effectively promote the kind of achievement necessary for students to realize the full benefit of a
rigorous mathematics education.