No.033

The Validity Argument

Not “Is the test valid?” but “Valid for what?”

AssessmentAdvanced2 min read
In one line

A validity argument justifies the intended interpretations and uses of test scores by laying out the chain of inferences they depend on and backing each link with evidence.

A university in Hanoi uses an English test to decide who may skip the first-year language course. Is the test valid? Mainstream validity theory says that is the wrong question: what needs justifying is how its scores are interpreted and used.

Samuel Messick’s unified view made construct validity the umbrella concept. For him, validity is an overall judgement of how well evidence and theory support the inferences and actions based on scores, including their social consequences. Lee Cronbach (1988) had framed validation as building an argument, and Michael Kane (1992) turned this into a practical method, the argument-based approach. First, state the intended interpretation and use as a chain of inferences: from scoring a performance, to generalising across tasks and raters, to real-world ability and a decision. Then build the validity argument by testing each link, especially the weakest, against evidence; as Kane (2013) stresses, bolder claims need more support.

Carol Chapelle, Mary Enright and Joan Jamieson (2008) used this framework to build the validity argument for the internet-based TOEFL, spelling out inferences from domain definition to utilisation. Lyle Bachman’s (2005) assessment use argument extends it to the decisions and consequences of test use. In Bachman and Palmer’s (2010) version, designers start from the intended consequences and decisions and work back to the scores.

Not everyone accepts this framing. Borsboom, Mellenbergh and van Heerden (2004) argued for a much narrower concept, in which a test is valid if the attribute it targets exists and causes variation in the scores. For teachers, the payoff is a habit of mind: before trusting a score, ask what claim it supports and what backs each step, including its effects through washback and other consequences.

In actionOne link in the chain

Claim: a band 6 on the placement test means a student can follow first-year lectures in English. Warrant: the listening tasks reflect real lecture demands. Backing: an analysis of lecture recordings. Rebuttal to check: strong scorers who still struggle in lectures.

An invented illustration of Toulmin-style reasoning: each inference needs backing, and a validity argument is only as strong as its weakest link.

In the classroom

  1. 1

    Before using any test score, write one sentence stating the decision it will inform, then list the evidence you would need to trust it for that decision.

  2. 2

    Check that your end-of-unit test samples the tasks the unit actually taught; if it does not, the domain definition link is weak.

  3. 3

    Compare test scores with what you see learners do in class, and investigate cases where the two disagree rather than ignoring them.

Myth and reality

The myth

A test is either valid or invalid, and a validity coefficient or a publisher’s label settles the matter.

What the evidence says

In the dominant argument-based view, validity is a matter of degree. It belongs to particular interpretations and uses of scores, so the same test can be well supported for one purpose and poorly supported for another.

Reflect

Which test score in your school is used for a decision it was never designed to support, and who would need to see the evidence to change that?

Worth a minute before your next lesson, or a conversation in the staffroom.

SourcesChecked against Crossref and library records

Where it comes from

The phrase goes back to Cronbach’s 1988 chapter, Five perspectives on validity argument, alongside Messick’s unified view (1989). Kane’s 1992 paper set out the argument-based approach, later refined as the interpretation/use argument. Chapelle, Enright and Jamieson applied it to the TOEFL (2008), and Bachman and Palmer (2010) extended it as the assessment use argument.

Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535.

Key readings
  1. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.

    A compact statement of the unified view, including the consequential aspect of validity, in nine pages.

  2. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.

    Kane’s mature account of the interpretation/use argument, organised around eight general points about validation.

  3. Chapelle, C. A., Enright, M. K., & Jamieson, J. (2010). Does an argument-based approach to validity make a difference?. Educational Measurement: Issues and Practice, 29(1), 3–13.

    Lessons from building the TOEFL validity argument: what the argument-based approach changes in practice.

  4. Bachman, L. F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1–34.

    Introduces the assessment use argument, extending validation to the decisions and consequences of test use.

  5. Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071.

    The best-known dissent: a short, provocative case for a narrower, causal definition of validity.

ConnectionsSee it in the graph

Know a teacher who’d like this?

Send them the card. It’s made for Facebook, Zalo, LinkedIn and Instagram, and links back here.

Concepts
Criterion-Referenced and Norm-Referenced TestingNo. 060 · Assessment
Lexical DensityNo. 059 · SFL
Presentation–Practice–ProductionNo. 058 · Pedagogy
The Academic Word ListNo. 057 · Corpus
Postmethod PedagogyNo. 056 · Teacher Dev.
Code-SwitchingNo. 055 · Socioling.
Negotiation of MeaningNo. 054 · SLA
Dialogic TeachingNo. 053 · Discourse
Linguistic RelativityNo. 052 · Cognitive
Consequential ValidityNo. 051 · Assessment
Appraisal TheoryNo. 050 · SFL
The Lexical ApproachNo. 049 · Pedagogy
Lexical BundlesNo. 048 · Corpus
Action ResearchNo. 047 · Teacher Dev.
Linguistic ImperialismNo. 046 · Socioling.
The Interaction HypothesisNo. 045 · SLA
Display and Referential QuestionsNo. 044 · Discourse
Image SchemasNo. 043 · Cognitive
Assessment LiteracyNo. 042 · Assessment
Genre PedagogyNo. 041 · SFL
Communicative Language TeachingNo. 040 · Pedagogy
ConcordanceNo. 039 · Corpus
English as a Lingua FrancaNo. 038 · Socioling.
Comprehensible InputNo. 037 · SLA
Wait TimeNo. 036 · Discourse
Exploratory PracticeNo. 035 · Teacher Dev.
Thinking for SpeakingNo. 034 · Cognitive
The Validity ArgumentNo. 033 · Assessment
Grammatical MetaphorNo. 032 · SFL
The Involvement Load HypothesisNo. 031 · Pedagogy
Semantic ProsodyNo. 030 · Corpus
Native-SpeakerismNo. 029 · Socioling.
Processability TheoryNo. 028 · SLA
Epistemic Status and Epistemic StanceNo. 027 · Discourse
Teacher CognitionNo. 026 · Teacher Dev.
Construction GrammarNo. 025 · Cognitive
Data-Driven LearningNo. 024 · Corpus
The Teaching–Learning CycleNo. 023 · SFL
InvestmentNo. 022 · Socioling.
Dynamic AssessmentNo. 021 · Assessment
Focus on FormNo. 020 · Pedagogy
The Output HypothesisNo. 019 · SLA
The Apprenticeship of ObservationNo. 018 · Teacher Dev.
Prototype TheoryNo. 017 · Cognitive
The Idiom PrincipleNo. 016 · Corpus
Theme and RhemeNo. 015 · SFL
TranslanguagingNo. 014 · Socioling.
RecastsNo. 013 · Discourse
Formative AssessmentNo. 012 · Assessment
Task-Based Language TeachingNo. 011 · Pedagogy
The Noticing HypothesisNo. 010 · SLA
Reflective PracticeNo. 009 · Teacher Dev.
Conceptual MetaphorNo. 008 · Cognitive
CollocationNo. 007 · Corpus
Register: Field, Tenor and ModeNo. 006 · SFL
Communicative CompetenceNo. 005 · Socioling.
IRF / IRE SequenceNo. 004 · Discourse
WashbackNo. 003 · Assessment
ScaffoldingNo. 002 · Pedagogy
InterlanguageNo. 001 · SLA