When does a holdout period stop being an independent check?

Write the selection rule before opening the evaluation

Begin with a protocol card: candidate rules, development period, evaluation period, selection metric, risk constraints and permitted revisions. For an explicitly synthetic plan, label observations D01–D20 development and E01–E10 evaluation. Select a rule using D only, write its version and selection reason, then freeze them before opening E. These labels are invented observations, not historical market sessions.

A chronological split organizes an experiment, but it does not answer every question about dependence or representativeness. A brief period may omit conditions important to the intended use. An evaluation remains bounded by its data and methodology even when nobody tuned against it.

Keep a fictional three-trial log

In a fictional trial log, A was selected using D01–D20. Trial 1 evaluates A on E01–E10 and produces a stipulated ending value of 101 units from 100. After seeing it, the researcher changes the rule: Trial 2 evaluates B on the same E and gives 103. Another revision, Trial 3, evaluates C there and gives 102. No actual market data or product run underlies these invented numbers. The log columns are trial, version, data inspected before the revision, evaluation range and result.

If B is then chosen because 103 is largest, E helped choose B. Reporting B as though it were the preselected untouched test omits Trials 1 and 3 and the revisions between them. Preserve all three rows and the decision to choose B. Label E as reused for selection; do not erase its history by renaming it a holdout. The invented values alone cannot quantify an overfitting probability.

Recognize less obvious forms of reuse

A researcher can reuse evaluation information without changing a numeric threshold. Changing an agent prompt after reading its answers, dropping an awkward company, selecting a favorable subperiod, or changing the metric can all make the evaluation influence the final story. Record those decisions rather than treating only code edits as revisions.

Label a reused evaluation as part of development or model selection. A later check can use newly designated data under a fresh protocol, but repeatedly relabeling the same observations does not make them unseen. In a limited dataset, the honest conclusion may be that no untouched evaluation remains.

Preserve a report another person can interpret

Attach the trial log to the final version: all tried variants, which outputs were inspected, any exclusions, the chosen rule, and the remaining evaluation boundary. Keep execution costs, distribution treatment and information cutoffs fixed when the question is about a rule change. If they change too, disclose that the comparison combines several changes.

NIST’s AITE program illustrates why evaluators protect test data from development: it uses a sequestered environment to reduce contamination. That is an evaluation-design reference, not a claim that NIST tested an investing strategy or endorses this worksheet. The SEC also distinguishes hypothetical backtests from actual performance. Neither an untouched test nor a detailed log guarantees future returns.

Method, sources, and limits

This is an original experiment-planning exercise, checked against primary references on 26 October 2026. No strategy, participant or product was tested. The example explains how a selection decision changes interpretation; it does not estimate statistical significance, supply an investment recommendation or establish a universal split ratio.