How 24 questions can map a mind.

Give the same 60-question exam to a novice and an expert, and you waste most of it on both. The novice grinds through dozens of items they never had a realistic chance of answering; the expert coasts through dozens that confirm nothing not already obvious after the first few. In psychometric terms, a question that a particular person is almost certain to get right — or almost certain to get wrong — carries close to zero information about that person. Fixed tests are long because they must serve every examinee they might meet, and for each individual, most of that coverage lands nowhere near the target.

The alternative is nearly as old as testing itself. Alfred Binet’s original intelligence scale was adaptive: a human examiner started near a child’s expected level and moved up or down in difficulty depending on the answers — a lineage the adaptive-testing literature has claimed since its earliest reviews (Weiss, 1982). What Binet did by trained intuition, item response theory turned into an algorithm. And the algorithm turns out to need remarkably few questions.

From counting answers to locating ability

Classical test theory — the framework behind most quizzes and exams — scores people by counting correct answers. The count is easy to compute and surprisingly hard to interpret: 70% on a hard form and 70% on an easy form are different abilities wearing the same number, and every property of the test, from item difficulty to reliability, is entangled with the particular sample of people who happened to take it (Embretson & Reise, 2000).

Item response theory (IRT) starts from a different primitive: the probability that a specific person answers a specific item correctly. The simplest version, introduced by the Danish mathematician Georg Rasch, models that probability as a function of one ability parameter for the person and one difficulty parameter for the item (Rasch, 1960). Richer models add a discrimination parameter — how sharply an item separates people just below its difficulty from people just above it — and a guessing floor for multiple-choice formats (Lord, 1980).

The consequential property is invariance. Once items have been calibrated onto a common scale, a person’s ability estimate no longer depends on which particular items they answered. Two examinees can face entirely different question sets and still land on directly comparable scores (Lord, 1980). That is the license under which adaptive testing operates — and it was Frederic Lord’s 1980 monograph that assembled the machinery of information functions, tailored testing, and equating into a working engineering discipline rather than a mathematical curiosity (Lord, 1980).

The adaptive loop

A computerized adaptive test (CAT) is a short loop, run until a stopping rule fires (van der Linden & Glas, 2010):

1. Estimate. Begin with a provisional ability estimate — a population prior, or one informed by whatever is already known about the examinee.
2. Select. From a calibrated item bank, choose the question that is most informative at the current estimate — in the standard approach, the one maximizing Fisher information, which for typical models peaks when item difficulty sits close to the examinee’s level.
3. Update. Re-estimate ability given the new response, by maximum likelihood or Bayesian updating.
4. Stop. End when the standard error of the estimate drops below a target — or when a maximum length is reached.

The loop behaves like a noisy binary search. Early answers move the estimate in large steps; later ones refine it. Because responses are probabilistic rather than deterministic — a correct answer might be a lucky guess, a wrong one a lapse — convergence is slower than true bisection. But each well-chosen item still buys more precision than almost any item on a fixed form, where the bulk of the questions sit far from any given examinee’s level. Precision accumulates item by item: the standard error shrinks with the square root of the total information collected, and an adaptive test collects information at close to the maximum possible rate for every examinee, not just those near the test’s average difficulty (van der Linden & Glas, 2010).

What the efficiency evidence shows

The claim that adaptivity roughly halves test length is not a vendor talking point; it is the settled center of a five-decade empirical literature. Weiss’s review of the first decade of computerized adaptive testing concluded that adaptive tests could match the precision of conventional tests using substantially fewer items — on the order of half — and, just as important, that their precision was far more uniform across the ability range, where fixed tests measure well near the middle of the distribution and poorly at the extremes (Weiss, 1982). Weiss and Kingsbury extended the case from aptitude to educational measurement, showing the same machinery worked for achievement testing and mastery decisions (Weiss & Kingsbury, 1984).

The largest natural experiment is military. The United States moved its Armed Services Vocational Aptitude Battery from paper to adaptive administration through the 1980s and 1990s — a research program documented in book-length detail — and the adaptive version delivered comparable measurement in substantially less testing time, becoming one of the largest operational CAT programs in the world (Sands, Waters & McBride, 1997). Licensure, admissions, and certification testing followed, and the accumulated design knowledge — item banking, exposure control, content balancing, stopping rules — is codified in the field’s standard references (Wainer, 2000); (van der Linden & Glas, 2010).

The most striking modern efficiency results come from health measurement, where respondent time is scarce and item banks are deep. The NIH-funded PROMIS initiative calibrated large IRT item banks for self-reported outcomes such as pain, fatigue, and emotional distress, and found that short adaptive assessments matched or exceeded the precision of the longer fixed questionnaires they replaced (Cella et al., 2010). Gibbons and colleagues built an adaptive depression measure on a bank of several hundred calibrated items; the adaptive version asked an average of about 12 questions per person, and its scores correlated around .95 with scores computed from the full bank (Gibbons et al., 2012). That is the general shape of the finding across domains: a well-calibrated adaptive test can discard most of its own length without discarding measurement.

Why matched difficulty measures more

The engine underneath all of this is the item information function. In IRT, every item has a curve describing how much statistical information it contributes at each ability level, and that curve peaks near the item’s difficulty — for multiple-choice items with a guessing floor, slightly above it (Lord, 1980). An item far from a person’s level produces a nearly deterministic response and an almost flat likelihood: you learn essentially nothing new. An item near their level behaves like a coin flip weighted by ability, and each such response cuts the plausible range down sharply.

Greedy information-maximization has a known failure mode, though: the same small set of highly discriminating items gets selected for nearly everyone, wearing a groove in the bank and creating a security problem. Operational CATs therefore temper selection with exposure control and content constraints — for instance, stratifying the bank so that less discriminating items are used early, and the sharpest items are saved for when the estimate is already close (Chang & Ying, 1999). Some raw efficiency is deliberately traded away for bank health and content coverage; the halved test lengths reported in the literature are generally measured with constraints of this kind in place.

What the evidence doesn’t show

Four honest limits on this literature:

• Adaptivity buys precision, not validity. A CAT measures the same construct as the fixed test it replaces — just faster. If the underlying items measure the wrong thing, the adaptive version arrives at the wrong answer in half the time. The efficiency literature is almost entirely about reliability and precision per item, not about whether the trait itself predicts anything.
• The gains assume the model’s assumptions roughly hold. Standard CAT machinery assumes one dominant dimension and locally independent items. Real skill domains are often multidimensional, and as content constraints multiply to cover a broad blueprint, the efficiency advantage shrinks (van der Linden & Glas, 2010).
• The item bank is the binding constraint. Calibrating items requires large pretest samples; programs without them get unstable parameters and lose much of the promised gain. The celebrated efficiency results all come from programs with deep, carefully calibrated banks — the military battery, PROMIS, large clinical banks — not from small bespoke question pools.
• The motivational claims are softer than the measurement claims. It is often asserted that matched difficulty makes tests more engaging or less stressful; the field’s own primers treat examinee-experience benefits as plausible rather than established (Wainer, 2000). And no particular number is magic: whether a given adaptive test needs 12, 24, or 40 items depends on bank quality, construct breadth, and the precision target — not on any law of nature.

None of these caveats blunt the central result. They mark its edges: adaptive selection is a precision multiplier on top of a calibrated bank and a defensible construct — and it multiplies whatever it is given, good or bad.