Cross-posted at Education Week.
The logic seems airtight. You are looking for a good measure of whether students are college ready. Why not find a gold-plated measure, one that has long been used by higher education institutions for that very purpose? Why not use the SAT?
My aim in this blog is persuade you that, reasonable as this sounds, it is not a good idea.
The SAT is not supposed to measure mastery of the Common Core or any particular mathematics and English literacy curriculum. So what, then, is it? When the College Board created the SAT, they called it the Scholastic Aptitude Test. The idea was that it measured an aptitude for academic work. But it was really an intelligence test combined with a measure of general knowledge, the kind that people with money and social standing acquire just by growing up in families with access to high culture. Intelligence was thought to be immutable, in your genes, so the College Board said you couldn’t study for it. In more recent years, the College Board changed the name of the test, because the idea that intelligence was immutable had been discredited. It became the Scholastic Achievement Test. But that called into question what exactly it was that had been achieved. Then it became the SAT. Perhaps it did not matter what it was measuring as long as the scores on the SAT were correlated with academic success in college.
Carol Dweck, a leading American researcher, has just won a prestigious new international prize for dealing a death blow to the idea that intelligence tests measure some immutable property of individual capacity for intelligent behavior. Her work, and much that went before, makes it clear that, if there is something called intelligence, it is very mutable, a measure in part of exposure to experiences available to some and not others as a function of their social and economic status and also a function of perceptions of self-efficacy and the expectations that the adults in their lives hold for them. That includes, especially, their teachers.
By now, you are very impatient with me. All that, you say, may be true, but it does not explain why using the SAT as a gateway to college is a bad idea. If performance on the SAT is well-correlated with performance in college, and if you are looking for a good way to measure college readiness, why not use it for that purpose, whether or not it measures some immutable characteristic of human beings? If they are ready, they are ready. What’s the big deal?
Not so fast. I said a moment ago that the core argument in favor of the use of the SAT was its ability to predict a student’s chances of success in college. But the universities have discovered that it is not such a good predictor. Dweck showed that success in college depends as much on what we are now calling grit and resilience as on the kind of general knowledge that is measured by the SAT. Kids who are really determined and work hard can do as well as others who might have an edge in whatever the SAT measures but are not as prepared to do what it takes to succeed. But the SAT does not measure that determination. Which is why it does not do a great job of predicting success in college, especially for students from disadvantaged families. A growing number of colleges no longer require applicants to submit an SAT score to the admissions office.
But even if it did a better job than it does of predicting success in college, it would still be a bad idea for schools to use the SAT as a measure of college readiness.
Whether the SAT accurately predicts student performance in college is, I submit, the wrong question. Student performance on a wide range of tests can be correlated with student performance in college. If your aim is simply to predict college performance, just pick the cheapest test you can find that meets your standard for reliability.
But, wait. Please don’t do that! The question you should be asking is not what is the cheapest way to sort students. The question you should be asking is how you can use tests to greatly improve the probability that all students, and particularly disadvantaged students, will be ready to succeed in college when they leave high school. That is a very different goal. It turns out that many nations, but not the United States, have figured out that tests can be used not just to measure student performance, but to improve student performance, on a wide scale.
To do that, they start by being clear about the standard of performance they expect their students to reach. Because their whole system is built on the assumption that the dynamics of the global economy will greatly reduce the number of jobs available to high school graduates who have only what we call the basic skills, their minimum standard is much higher than the standard our high school students usually meet. They usually do this not just for literacy, but for writing, speaking and listening, mathematics, science, technology, history, art, music and physical education.
Having fixed the standard they expect students to meet—usually by the time students have finished the 10th grade, not the end of 12th grade—they then create a detailed curriculum framework. That framework spells out which topics in each subject will be covered in which grades or grade spans and the order in which they will be studied. These frameworks reflect what modern psychology tells us about how students learn these subjects. They also reflect what the student needs to know and be able to do to go on to the next stage of their education.
The top-performing countries use this framework to create course syllabi for each subject in the required curriculum. These syllabi leave it up to the teachers to create lesson plans in collaboration with their colleagues, that will work for them and for their students, but, because all the schools use the same frameworks, they make it possible for students in every district and school to transfer to another district or school and not miss a beat.
When the syllabi are created, these countries or provinces then develop end-of-course exams that are designed to capture what they think is really important for the students to learn. They often spend two to three times what we spend on a test, so that they can capture much more of what is important than our cheaper tests can. They usually do this with essay-based and open-ended questions, not multiple-choice tests. But they do a lot less high-stakes testing than we do—with fewer tests and better tests at key points in a student’s school experience.
The tests are centrally scored, usually by regular classroom teachers who have been well-trained to do this. Here is the part I really want you to pay attention to. These countries then collect real examples of student work that the scorers judge to be very good. Those examples are widely published and kept in an online library, so students and teachers can see these examples, collected over many years. The examples are usually published with a commentary explaining why the student work received good scores.
As I have explained to the readers of this blog before, this is crucially important, because it short circuits a system like ours in which the expectations for students vary widely, with the lowest expectations reserved for students who are from poor families or from ethnic or racial minorities. We have very different standards for different students. If we want a system that will produce high performance among disadvantaged students, it is essential that we find a way to create the same high expectations for all students. Having uniform and consistent curriculum frameworks and course syllabi goes a long way toward creating the same expectations for all students, but having, in addition, actual examples of student work that meet high standards and explanations of what it was about those examples that earned a high grade turn out to be crucial features of a system that creates truly common expectations for all students.
With a system like this, everyone understands what it takes to earn high grades and everyone understands that any student who works hard enough at it can succeed. Nothing could be more different from the original idea behind the SAT, that some kids have more natural aptitude for academic work than others, and the job of the tests is to identify those kids and open the gates for them.
The SAT does none of this. Not only is it poorly correlated with student success in college, but it does little to help students do what they have to do to be ready to actually succeed in college. If you want your state to have a test that will do that, here is what I think you need to do:
- Find out what it really takes to succeed in your state’s open admissions colleges. Don’t ask them that question to find out. They will tell you what they would like students to know and be able to do coming in, not what they really need to know and be able to do. And don’t just correlate student performance on some other test with student performance in college. You will have all the problems you have with the SAT. You will have to analyze the actual grade levels of reading challenge posed by the actual texts that the college instructors assign. You will probably find out that the texts are written at a 12th-grade level and the texts your high school students are assigned are written at the 7th-or 8th-grade level. You should analyze the topics in the courses they call “College Math” and “College Algebra,” to see what level of math your high school graduates will have to be ready to do. You will probably find out that most of these topics are actually the topics in Algebra I, a course your high school grads should have mastered in middle school but cannot pass as they head to college. And you will find that most of your high school grads cannot write an acceptable 2-page essay, so the colleges don’t assign much writing. If you set your target at a 12th-grade-reading level, mastery of Algebra I and the ability to write a literate 2-page essay, you will be asking for more than the average American high school graduate can do now.
- If you make that your college and career ready target for the end of 10th grade, then those who don’t reach it will have time to reach it before they graduate. If they do reach it by the end of grade 10, they will be able to take and succeed in a tough high school upper division program like the IB or a program of AP courses, or a full Associate’s Degree program resulting in a credential when they graduate or a demanding career and technical education program leading to a real career that pays well.
- If you want the vast majority of your students to succeed against this new standard, decide on the subjects they have to take, set the 10th-grade standard for them, then create a year-by-year or grade-span-by-grade-span curriculum framework for each subject, spelling out which topics are to be studied in which order. Have your state’s best teachers develop syllabi for each course for use across all schools.
- Create first-rate, end-of-course exams for each subject to be offered by the 10th grade. Collect student work that earns top marks. Publish the questions asked, the answers given, and a commentary explaining why they earned top grades.
- Develop high-quality guidance for your teachers for the formative evaluation of student progress against the syllabi, so they can catch students falling behind quickly and do something about it.
- Collect data on student progress through the grades so that the state can identify schools in which students are not making progress at the necessary rate and organize visits from teams of experts who can figure out what is wrong and what is needed to set it right.
- Require your schools of education to teach beginning teachers how to teach the state-defined courses to students from many different backgrounds, how to work with other teachers to create first-rate lessons that will deeply engage their students, how to identify the problems their students will face as they go through the curriculum framework and how to access and evaluate a range of things that have been shown to work when addressing those problems.
Do this and you will be amazed at the results. You need to think about your college-ready test as part of a tightly knit state instructional system, not just as a measuring tool untethered to anything else. Don’t use the SAT to do this.
