Chapter 3: The Liar’s Bell

Ground floor

A village keeps a boy on the tower to ring the bell when wolves come. Some nights wolves come and he rings: good. Some nights wolves come and he dozes: the flock pays. Some nights the wind moves the grass and he rings anyway: the village runs out in its nightshirts for nothing. And most nights nothing happens and he rings nothing, and nobody ever praises him for it.

Four outcomes, always four: hit, miss, false alarm, correct rejection. Every detector that ever existed lives in this two-by-two square: a smoke alarm, a cancer screen, a spam filter, a border guard, a code reviewer, a person deciding whether two flashes on a screen were the same or different. There is no fifth square to hide in.

And here is the thing the village never understands. There are two completely different ways for the boy to improve his numbers. He can get better eyes, spotting real wolves earlier in dimmer light. Or he can get more trigger-happy, ringing at every rustle, which buys more hits at the price of more false alarms, with his eyes exactly as bad as before. A village that counts only hits will promote the trigger-happy boy every time. Most villages count only hits.

The stairs

Signal detection theory, born in the 1950s out of radar screens and perked ears, splits the two knobs formally, and the split is one of the quiet great ideas of the twentieth century. Sensitivity, written d’ (say “d-prime”), measures the quality of the eyes: how well separated the world’s two states are in the observer’s experience. Criterion, written c, measures the trigger: how much evidence the observer demands before ringing. The founding insight is that hit rate alone confounds them hopelessly, and only the pair, hit rate together with false alarm rate, can pull them apart.

The working formula, worth carrying for life: d’ = z(H) − z(F), where H is the hit rate, F the false alarm rate, and z the function that converts a proportion into standard-deviation units. Feel it on numbers. Boy A rings on ninety percent of wolf nights but also on sixty percent of windy ones: z(0.9) − z(0.6) gives d’ of about 1.0, mediocre eyes, hair trigger. Boy B rings on only seventy percent of wolf nights but on five percent of windy ones: z(0.7) − z(0.05) gives d’ of about 2.2. Boy B separates wolf from wind dramatically better, and the village, counting hits, fired him years ago. Plot every possible criterion for fixed eyes and you draw the ROC curve, the detector’s true portrait, the one that no amount of attitude can game.

Now carry it out of the village, because this chapter is load-bearing for everything that follows. Any system that measures minds with yes-or-no tasks, and most of them do, faces the same fork every time a user’s accuracy rises: did d’ move, or did c? People shift criterion for free, from mood, caffeine, instructions, or simply having learned what the task rewards; sensitivity is the expensive part. A measurement platform that scores raw accuracy is a village promoting trigger-happy boys, and it will manufacture improvement out of nothing, sincerely, at scale, forever. This is the first of the artifacts that built the corpse in the opening scene. Separating the two knobs costs one extra column of arithmetic, and most published cognitive research, let alone most products, has historically not paid it, which is not a compliment to how easy the fix is.

The deeper lesson generalizes past perception, and you should install it as a reflex. Any judge, human, statistical, or artificial, has a sensitivity and a criterion, and evaluating the judge without separating them is meaningless. A reviewer who approves everything has a superb hit rate on good work. A medical screen tuned never to miss will drown you in false alarms, and whether that is correct depends on the price of each square of the two-by-two, which is a decision about values, not about eyes. And an artificial assistant that confidently answers “there is nothing here” has simply set a criterion: its zero is a ruling, not an observation, and the difference between those two words is the difference between an instrument and an oracle. You have met that assistant. Everyone has, by now.

The locked door

Behind the door: unequal-variance signal detection, where the tilt of the ROC curve betrays the structure of memory itself; confidence-rating methods; and the frontier, drift diffusion models, which stop treating a decision as a snapshot and model it as evidence accumulating over milliseconds toward a boundary, explaining speed, accuracy, and their tradeoff with the same few parameters. That door gets its own chapter in this book, two rooms ahead. Names for the classical core: Green and Swets, Macmillan and Creelman.