SHAP attributions: reference values, output units and causal limits

What SHAP explains

SHAP assigns feature contributions to a particular model prediction using a framework related to Shapley values. In an additive explanation, the reference value plus the feature contributions reconstructs the explained output, subject to the explainer's assumptions and numerical approximation. A contribution describes the allocation of a prediction relative to a reference; it is not a feature's universal importance or an automatically causal effect. [1]

Choose the explanation target

Declare the model output being explained. For a classifier, a raw margin, a probability and a loss are different quantities. Their attributions have different units. In supported TreeExplainer configurations, probability explanations sum to the predicted probability, while raw outputs depend on the model and may represent log odds. [2]

Also specify the background data and how feature dependence is handled. Correlated inputs require assumptions about what it means to include or exclude a feature. Changing the background or dependence setting can change the reference value and allocation. [2]

A practical workflow

Validate predictive performance before explaining the model. Select a documented reference population and a compatible explainer, then record the model version, feature representation, background sample and explanation settings. Inspect several ordinary cases and important failure cases rather than treating a single attractive chart as validation.

Check that the contributions and reference reconstruct the selected output within the method's numerical tolerance. Examine whether explanations change materially under reasonable alternative background samples. If reporting average absolute SHAP values, state that this summarizes contribution magnitude, not whether a feature generally increases or decreases predictions. [1][2]

Illustrative decomposition

Imagine a delivery-time model explained in days: reference prediction 5, distance contribution +2 and stock-availability contribution −1, giving 5 + 2 - 1 = 6 days. These invented values demonstrate additive accounting; no model was trained. They do not establish that shortening a real delivery route would save exactly two days. Other operational variables and the way the data were collected may account for the observed prediction pattern.

Causal limits and interpretation

A predictive model can exploit proxies and confounding relationships. Explaining it faithfully does not identify what would happen after an intervention. Even an explainer setting named interventional does not turn an ordinary predictive model into an identified causal model. Causal claims need an appropriate study design and explicit assumptions. [2][3]

Use SHAP to inspect model behavior and generate questions for investigation. Report the reference, output units and important sensitivity findings alongside the attribution. This article synthesizes primary sources; no local explanation experiment was performed.

Sources

[1] Lundberg and Lee (2017): A Unified Approach to Interpreting Model Predictions

[2] SHAP API: shap.TreeExplainer

[3] SHAP: Be Careful When Interpreting Predictive Models in Search of Causal Insights