Consequential Validity
Is a test still valid if it sends learners down the wrong path?
Consequential validity is the idea, from Samuel Messick’s unified view of validity, that the effects of using a test, intended and unintended, belong in judging whether its scores and their uses are justified.
A secondary school gives every newly arrived student a written grammar test. Anyone who scores under 40% spends a year in a separate beginners’ class. One girl chats easily in English with her classmates but freezes on gap-fills about the past perfect. She spends the year copying basic word lists, and nobody asks whether the test got her wrong.
Consequential validity is the idea that what happens when a test is used counts when we judge its validity, not just what the scores mean. It comes from Samuel Messick, who in 1989 made the consequences of test use one part of a single, unified validity question. But Messick drew a careful line: a harmful effect counts against validity when a flaw in the test causes it. Such flaws include measuring only part of the ability or letting something irrelevant drive the scores. A grammar-only test that keeps fluent speakers out of ordinary classes fails on exactly this point.
Not everyone accepts the idea. James Popham argued that consequences matter greatly but should be judged separately, keeping validity about whether score-based conclusions are accurate. Today the professional Standards for testing list consequences among the sources of validity evidence, and Michael Kane’s validity argument asks whether the decisions scores support are justified. Either way, the lesson for teachers holds. Placement decisions and washback are part of what a test does, so judging a test means watching what happens after the scores come out.
In actionTwo harms, two diagnoses
Case A: A placement test has only written grammar items. Learners who speak well but struggle with gap-fills are put in a beginners’ class for a year. Case B: A well-designed speaking test shows that an applicant is not yet ready for a course taught in English. She is disappointed and has to wait a year.
In Case A the harm comes from what the test leaves out, so it counts against validity. In Case B the harm is real but follows from an accurate score, so for Messick it is about fairness and policy, not validity.
In the classroom
- 1
Before using a test for a big decision, list who the scores will affect and how, and ask which effects could come from a flaw in the test rather than real differences in ability.
- 2
Review placement results after four weeks: compare each learner’s score with how they actually cope in class, move anyone the test misjudged, and note what the test missed.
- 3
Ask learners what they did differently because of a test: what they studied, skipped or gave up on. Their answers are evidence about its consequences, and weighing them is part of assessment literacy.
Myth and reality
Consequential validity means that any test with harmful effects is an invalid test.
In Messick’s view, harmful effects count against validity when a flaw in the test, such as missing skills or irrelevant difficulty, causes them. Harms that follow from accurate scores still matter, but as questions of fairness and policy.
Reflect
Which decision you made from a test score this term would be hardest to defend if the learner asked what the score really showed?
Worth a minute before your next lesson, or a conversation in the staffroom.
SourcesChecked against Crossref and library records
Messick proposed a consequential basis of validity in 1980 and set it out most fully in his 1989 Educational Measurement chapter; this 1995 paper names consequences as one of six aspects. He wrote of a consequential aspect or basis; “consequential validity” became the common shorthand.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241–256.
Messick applies his ideas to language tests, explaining when washback is, and is not, evidence about a test’s validity.
Popham, W. J. (1997). Consequential validity: Right concern—wrong concept. Educational Measurement: Issues and Practice, 16(2), 9–13.
The best-known critique: take consequences seriously, but keep them out of the definition of validity.
Shepard, L. A. (1993). Evaluating test validity. Review of Research in Education, 19(1), 405–450.
A clear review arguing that a test’s intended uses and their effects should shape how it is validated.
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
Shows how score uses and their consequences fit into a step-by-step validity argument.
ConnectionsSee it in the graph
Know a teacher who’d like this?
Send them the card. It’s made for Facebook, Zalo, LinkedIn and Instagram, and links back here.