Correlation versus causation in incident and operations data

What it is

The SciPy documentation describes the Pearson coefficient as a measure of the linear relationship between two datasets, ranging from −1 to +1, where −1 or +1 imply an exact linear relationship and 0 no correlation; its p-value roughly indicates the probability that an uncorrelated system produces a correlation at least as extreme, and the test assumes normally distributed samples. The Spearman coefficient is described as a nonparametric measure of the monotonicity of the relationship; because it works on ranks rather than values, a single extreme point or a curved relationship distorts it less than the Pearson coefficient. Neither says which variable moves the other. Greenland and co-authors add the sharper point: a p-value is computed assuming chance was operating alone under all the model's assumptions, and those assumptions include how the data were collected and selected.

Operations data violate those assumptions in recurring ways: a common cause (weekday traffic raises both deploy counts and error counts), reverse causation (latency causes retries, so retries correlate with latency), selection (only incidents that were noticed have a record), aggregation across groups (the mix effect described under Simpson's paradox) and shared trends (two series that both grow over months correlate whatever their relation).

Why it matters

Postmortems and capacity decisions are built on correlations. "Errors rise with pod count" leads to the wrong fix if pod count rises with traffic and traffic causes the errors. Acting on a non-causal correlation does nothing at best and removes something useful at worst.

How to apply

• Plot before computing; a scatter plot shows clusters, curvature and single points that create the coefficient. Prefer Spearman for outlier-prone metrics.
• Draw the causal story as arrows and ask what else points at both variables; then stratify by it (by hour, by traffic band, by tenant).
• Check timing: the cause must precede the effect; compute the correlation at several lags in both directions.
• Detrend or difference time series before correlating; consecutive samples are not independent, so p-values from raw series come out too small.
• Prefer intervention to observation: switch the suspected cause off with a flag or in a canary and watch the effect. A randomised experiment removes the common causes an observer cannot list.
• Count the comparisons: among fifty metrics some pairs correlate by chance; treat a discovered correlation as a hypothesis to test on fresh data.

Pitfalls

Dashboards aligned on one time axis invite eyeballed correlations between any two spiky series. Metrics computed from each other (error rate and success rate) correlate by construction. Data collected only during incidents cannot show what the metric does when nothing is wrong.