Split conformal prediction: intervals with marginal coverage

Definition

Split conformal prediction adds prediction intervals to an already fitted regression model. A separate calibration sample determines how large the intervals should be. Under exchangeability of calibration and future examples, the procedure provides finite-sample marginal coverage without requiring a correct parametric model of the outcome distribution. [1][2]

How it works

Fit a predictor f on training data and keep it fixed. For each of n calibration examples, compute the absolute residual |y − f(x)| and sort these scores. For an error target alpha, use rank k = ceiling((n + 1) × (1 − alpha)). Let q be the k-th smallest score; return [f(x) − q, f(x) + q].

If k exceeds n, the basic construction needs an infinite bound, not a silently substituted lower quantile. More calibration data can improve resolution. [1]

Practical workflow

1. Fix the target, prediction horizon, model and score before using calibration outcomes.
2. Reserve calibration observations that were not used to fit or tune the predictor.
3. Choose the coverage target, compute the corrected quantile and save the calibration dataset version.
4. Evaluate coverage and interval width on a further untouched sample; report results for relevant slices as well as overall.

Changing the model after calibration requires reconsidering its calibration scores. Do not repeatedly tune the score using the same calibration sample while claiming the basic guarantee.

Illustrative example

Illustrative calculation, not an observed result: use n = 99 calibration deliveries and target 90% coverage. The required rank is ceiling(100 × 0.90) = 90. Suppose the 90th smallest absolute error is four minutes. A new delivery predicted to take twenty minutes receives the interval [16, 24] minutes.

This calculation explains interval construction. It does not show that a particular delivery, route or workload subgroup has a 90% conditional probability of being covered.

Limitations and evaluation

Marginal coverage averages over the calibration sample and a future observation. It does not guarantee equal coverage for every subgroup or every input. Exchangeability can fail with temporal dependence, feedback or distribution shift; the basic result must not be carried over automatically. [1]

The absolute-residual construction has constant width for a fixed calibration sample and may be inefficient when noise varies with the input. Adaptive scores or other conformal variants require their own specification. Report width alongside coverage: very broad intervals can achieve coverage while providing little practical value. No local model training or coverage experiment is claimed.

Sources

[1] A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification [2] Distribution-Free Predictive Inference For Regression