Why can the same signal accuracy produce very different flag precision?

Define the condition before counting signals

Imagine an invented financial-document screening exercise with a binary condition specified in advance, such as whether an identified statement contains a particular dated disclosure. Assume complete, correct reference labels for every case. A flag is the screening system’s output; a genuine case satisfies the reference condition. Neither label denotes a profitable investment.

Sensitivity means the fraction of genuine cases flagged. Specificity means the fraction of non-cases correctly left unflagged. Flag precision means the genuine cases among all flags. MIT’s probability teaching material explains why reversing a conditional probability requires accounting for the base rate. The two matrices below use wholly original invented counts.

Complete the low-prevalence matrix

Stipulate 10,000 fictional cases, of which 100 satisfy the condition and 9,900 do not. Set sensitivity to 80% and specificity to 90%. The four cells are: 80 true positives, 20 false negatives, 990 false positives and 8,910 true negatives. The four cells sum to 10,000.

There are 80 + 990 = 1,070 flags. Precision is 80 / 1,070 ≈ 7.476636%, although sensitivity is 80%. The denominators explain the difference: sensitivity divides by the 100 genuine cases; precision divides by all 1,070 flagged cases. There are many opportunities for false flags when the non-case group is large.

Complete a higher-prevalence control

Now stipulate another 10,000 cases with 2,000 genuine cases and 8,000 non-cases. Hold sensitivity at 80% and specificity at 90% as a mathematical assumption. The cells become 1,600 true positives, 400 false negatives, 800 false positives and 7,200 true negatives.

There are 2,400 flags, with precision 1,600 / 2,400 = 66.666667%. The change does not establish that a real screening model transports unchanged between populations. It isolates what the arithmetic would do if its two conditional rates stayed fixed while prevalence rose from 1% to 20%.

Inspect the deployment population and error costs

A useful evaluation record contains the condition definition, sampling frame, observation dates, reference-label process and all four counts. A single accuracy percentage hides which errors occurred and which population generated it. If reference labels are incomplete or the model helped create them, the apparent precision needs additional scrutiny.

The acceptable tradeoff also depends on the task. A false flag may waste review time; a missed condition may leave an important question unresolved. Those consequences need explicit assessment, not a universal threshold borrowed from this toy example. A flag is a reason to inspect evidence, not a calibrated probability of future returns or permission for an external action.

Method, sources, and limits

The two original finite matrices were calculated on 27 October 2026. No model, company or user was tested, and no real prevalence or trading performance was estimated. AI assisted the draft; independent AI verification is required before publication and does not establish human financial expertise.