2025 · Preprint · Preprints.org

The Coverage-Deferral Trade-Off: Fairness Implications of Conformal Prediction in Human-in-the-Loop Decision Systems

A study of how conformal prediction changes both uncertainty coverage and who is referred for human review.

Research question

Whose case gets a second look?

Evidence and scope

A prediction set can contain both possible answers. In this study, those cases go to a human reviewer. I compared a single calibration threshold with thresholds fitted separately for each group. The choice of threshold changes how often the right answer is in the set and how often a person has to wait for review.

On the Adult income dataset, groupwise calibration narrowed the gap in coverage from 5.0 to 1.2 percentage points. The gap in referral to a human grew from 5.1 to 12.9 points. Table 4 reports these results over 100 random seeds, with 90% target coverage and simulated review.

Primary source

Contribution
Davut Emre Taşar · Sole author
Source attribution
Davut Emre Taşar
Publication status
Preprint · Preprints.org
Publication or platform
Preprints.org
Identifier
10.20944/preprints202512.2631.v1

Sources

Research areas · Fairness · Uncertainty

Related work