Pool-based active learning: uncertainty sampling without biased evaluation

What pool-based active learning does

Pool-based active learning chooses which existing unlabeled examples should receive labels next. It starts with a labeled seed set and a larger unlabeled pool, trains a model, selects queries, obtains labels and repeats. The aim is to use annotation effort effectively. It does not guarantee fewer labels for every task, and it differs from generating synthetic queries or automatically assigning pseudo-labels. [1]

How uncertainty sampling works

An uncertainty strategy prioritizes examples whose predicted label is uncertain. For binary probabilistic classification, probabilities near 0.5 are typical candidates. Multiclass alternatives include a small gap between the two largest class probabilities or high predictive entropy. These criteria need not rank multiclass examples identically. [1]

The score reflects the current model, not an objective certificate of usefulness. A confidently wrong model can overlook informative regions, while ambiguous or corrupted examples may remain uncertain even after annotation. Batch selection should also consider redundancy and coverage.

A practical annotation loop

Before acquisition, set aside representative validation and final test data that the selection policy cannot use. Create a seed set covering plausible classes and operating conditions. Train, score the eligible pool, select a batch and have qualified annotators label it using documented rules. Remove acquired examples from the pool, add their labels to training and repeat. Track label disagreements and unavailable labels rather than silently converting them into negative examples. [1]

Keep an experiment log of selected identifiers, model versions, query scores, random seeds, label revisions and annotation effort. Use validation to choose the strategy and stopping budget.

Illustrative allocation

Suppose 200 seed labels are available for classifying support requests, and the next round permits 50 annotations. An experimental policy could allocate 40 to uncertain, diverse requests and 10 to randomly sampled requests. This mixture is an illustrative design choice, not a proven optimal ratio. Compare it with a policy acquiring 50 random requests, using the same initial seed set and an independent evaluation set. No improvement is assumed.

Selection bias and evaluation

Actively selected labels generally do not follow the population distribution. Performance on those selected examples therefore cannot serve as representative deployment performance. Research also shows that selection bias need not always harm model training; correcting it is not automatically beneficial for every learner. [2]

Compare learning curves against random acquisition at matched label budgets, and report actual annotation time when difficult examples cost more. Repeat with different seed sets where feasible. Reserve the final test for the chosen procedure and report minority-class behavior as well as the aggregate metric. This is a source-based workflow, not a local experiment.

Sources

[1] Burr Settles: Active Learning Literature Survey

[2] Farquhar, Gal and Rainforth (2021): On Statistical Bias in Active Learning