Confident Learning: auditing label quality with targeted review

Definition

Confident Learning is a framework for identifying and learning with noisy classification labels. It estimates relationships between the observed labels and unknown underlying labels, using a class-conditional noise formulation. In an auditing workflow, its output prioritizes examples for inspection; a flagged example is not automatically a verified annotation error. [1]

How it works

A detector combines the recorded label of each example with model-predicted probabilities for every class. Confident Learning uses class-dependent confidence thresholds to estimate likely label confusion and identify suspicious examples. Its theoretical guarantees depend on stated assumptions.

Cleanlab recommends out-of-sample probabilities, commonly generated by cross-validation, and can return issues ranked by label-quality scores. The probability columns must follow the same class order as the encoded labels. In-sample predictions can hide errors that a model has memorized. [2]

Practical workflow

1. Version the dataset, label definitions and class-to-column mapping.
2. Generate probabilities without training on the example being scored; respect entity groups and time constraints when splitting.
3. Rank candidate issues and allocate a review budget, including some unflagged examples for comparison.
4. Have reviewers apply the annotation rules and record the original label, decision, corrected label if any and reason.
5. Retrain only after documenting accepted changes; evaluate on a separately governed test set.

Preserve ambiguous and difficult examples until their status is resolved. Removing every model disagreement can erase valid cases.

Illustrative example

Illustrative example, not a measured audit: a support ticket is labelled billing. A model assigns probabilities 0.08 to billing, 0.86 to account access and 0.06 to software defects. The message says the customer cannot sign in to view an invoice.

This is a useful review candidate, but the correct label depends on whether the taxonomy follows the immediate obstacle or the customer’s ultimate goal. A reviewer may correct the label, retain it or request clearer guidance. Reviewing the fifty highest-ranked tickets is a workload choice, not a claim that the detector found fifty true errors.

Limitations and evaluation

Model weakness, class overlap and rare valid cases can cause disagreement. Class-conditional noise assumptions need not hold when mistakes depend strongly on the input within each class. [1]

Report how many candidates were reviewed, confirmed, rejected or left ambiguous. A reviewed sample drawn only from the highest-ranked issues cannot establish the error rate of the entire dataset. Keep label corrections traceable, and do not repeatedly clean a benchmark to favour the model under evaluation. No local label audit or training experiment is claimed.

Sources

[1] Confident Learning: Estimating Uncertainty in Dataset Labels [2] Cleanlab: Label-issue detection and filtering