The Validity Argument
Not “Is the test valid?” but “Valid for what?”
A validity argument justifies the intended interpretations and uses of test scores by laying out the chain of inferences they depend on and backing each link with evidence.
A university in Hanoi uses an English test to decide who may skip the first-year language course. Is the test valid? Mainstream validity theory says that is the wrong question: what needs justifying is how its scores are interpreted and used.
Samuel Messick’s unified view made construct validity the umbrella concept. For him, validity is an overall judgement of how well evidence and theory support the inferences and actions based on scores, including their social consequences. Lee Cronbach (1988) had framed validation as building an argument, and Michael Kane (1992) turned this into a practical method, the argument-based approach. First, state the intended interpretation and use as a chain of inferences: from scoring a performance, to generalising across tasks and raters, to real-world ability and a decision. Then build the validity argument by testing each link, especially the weakest, against evidence; as Kane (2013) stresses, bolder claims need more support.
Carol Chapelle, Mary Enright and Joan Jamieson (2008) used this framework to build the validity argument for the internet-based TOEFL, spelling out inferences from domain definition to utilisation. Lyle Bachman’s (2005) assessment use argument extends it to the decisions and consequences of test use. In Bachman and Palmer’s (2010) version, designers start from the intended consequences and decisions and work back to the scores.
Not everyone accepts this framing. Borsboom, Mellenbergh and van Heerden (2004) argued for a much narrower concept, in which a test is valid if the attribute it targets exists and causes variation in the scores. For teachers, the payoff is a habit of mind: before trusting a score, ask what claim it supports and what backs each step, including its effects through washback and other consequences.
In actionOne link in the chain
Claim: a band 6 on the placement test means a student can follow first-year lectures in English. Warrant: the listening tasks reflect real lecture demands. Backing: an analysis of lecture recordings. Rebuttal to check: strong scorers who still struggle in lectures.
An invented illustration of Toulmin-style reasoning: each inference needs backing, and a validity argument is only as strong as its weakest link.
In the classroom
- 1
Before using any test score, write one sentence stating the decision it will inform, then list the evidence you would need to trust it for that decision.
- 2
Check that your end-of-unit test samples the tasks the unit actually taught; if it does not, the domain definition link is weak.
- 3
Compare test scores with what you see learners do in class, and investigate cases where the two disagree rather than ignoring them.
Myth and reality
A test is either valid or invalid, and a validity coefficient or a publisher’s label settles the matter.
In the dominant argument-based view, validity is a matter of degree. It belongs to particular interpretations and uses of scores, so the same test can be well supported for one purpose and poorly supported for another.
Reflect
Which test score in your school is used for a decision it was never designed to support, and who would need to see the evidence to change that?
Worth a minute before your next lesson, or a conversation in the staffroom.
SourcesChecked against Crossref and library records
The phrase goes back to Cronbach’s 1988 chapter, Five perspectives on validity argument, alongside Messick’s unified view (1989). Kane’s 1992 paper set out the argument-based approach, later refined as the interpretation/use argument. Chapelle, Enright and Jamieson applied it to the TOEFL (2008), and Bachman and Palmer (2010) extended it as the assessment use argument.
Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
A compact statement of the unified view, including the consequential aspect of validity, in nine pages.
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
Kane’s mature account of the interpretation/use argument, organised around eight general points about validation.
Chapelle, C. A., Enright, M. K., & Jamieson, J. (2010). Does an argument-based approach to validity make a difference?. Educational Measurement: Issues and Practice, 29(1), 3–13.
Lessons from building the TOEFL validity argument: what the argument-based approach changes in practice.
Bachman, L. F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1–34.
Introduces the assessment use argument, extending validation to the decisions and consequences of test use.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071.
The best-known dissent: a short, provocative case for a narrower, causal definition of validity.
ConnectionsSee it in the graph
- Consequential ValidityThe contested place of test consequences within validity.
- WashbackEffects on teaching are among the consequences a test-use argument must consider.
- Dynamic AssessmentMediated scores demand their own arguments about what they mean.
- Communicative CompetenceLanguage test constructs are often defined in terms of communicative competence.
Know a teacher who’d like this?
Send them the card. It’s made for Facebook, Zalo, LinkedIn and Instagram, and links back here.