Post-training quantization versus quantization-aware training: choosing and checking a smaller model

What it is

Quantization represents model values with fewer bits or fewer available numerical levels. Post-training quantization (PTQ) applies conversion after training. Quantization-aware training (QAT) emulates inference-time quantization during further training so parameters can adapt to its errors; a later conversion produces the deployable quantized model. QAT’s training model is not automatically an integer-only executable. [1][2]

How it works

In simple affine integer quantization, q = clip(round(x/s) + z, qmin, qmax) and x_hat = s*(q-z), where s is a positive scale and z an integer zero point. Rounding introduces approximation error; clipping saturates values outside the supported range. With fixed bit width, widening the represented range reduces clipping but increases spacing between adjacent values.

Static activation quantization estimates ranges using representative calibration inputs. Dynamic quantization can determine activation ranges during inference, while weight-only or FP16 conversion has different requirements. These approaches should not all be described as fully integer execution. [1][3]

Practical workflow

First export and evaluate a floating-point model to separate conversion problems from quantization problems. Then try a PTQ scheme supported by the intended runtime and hardware. For static calibration, select a representative training-data subset with production preprocessing, input shapes and relevant operating conditions. Ordinary activation calibration uses inputs and does not require their labels. Reserve independent evaluation data for acceptance checks. [3]

If PTQ fails the required quality target, investigate calibration and sensitive operations before considering QAT. QAT requires a working training pipeline and additional computation. After QAT, convert again and evaluate the exported artifact; the training simulation alone does not establish deployment behavior. [2]

Illustrative example

On an illustrative grid with s = 0.1 and z = 0, x = 0.26 becomes integer 3 and reconstructs as 0.3, giving an absolute error of 0.04 when clipping is inactive. Repeated layer computations can amplify or reduce the effect of such errors on outputs. These numbers demonstrate rounding only; they do not predict a model’s accuracy loss.

Limits and verification

Measure task quality, relevant subgroup behavior, latency, peak memory and artifact size on the target device. Check operator support, input/output types and floating-point fallbacks: successful conversion does not prove integer-only execution or a speed improvement. [3] Outlier-heavy activations and unrepresentative calibration can cause substantial error. QAT can improve robustness to quantization but does not guarantee recovery of the original quality. [2]

Document bit widths, per-tensor or per-channel scaling, calibration data, converter and runtime versions, and acceptance thresholds. No local quantization experiment or hardware benchmark was performed for this article.

Sources

[1] TensorFlow: Post-training quantization [2] TensorFlow: Quantization aware training [3] Google AI Edge: Post-training quantization