テーマ： statistics
-
中央値・パーセンタイル・比率のブートストラップ信頼区間
生の観測値を復元抽出で何度もリサンプリングし、各リサンプルについて統計量を計算し、得られた分布から区間を読み取る。これにより、教科書の公式が通用しない中央値・パーセンタイル・比率・差分についての不確実性を求められる。手法名、リサンプリング回数、サンプルサイズを報告し、小さなサンプルの極端なパーセンタイルについては信用しないこと。
-
Differential privacy in one paragraph, and where it does not fit
Differential privacy bounds how much any one person's record can change the output distribution of a query mechanism, by adding calibrated noise and charging each answer to a budget; it fits repeated aggregate releases over large populations and does not fit record-level data, small groups, exact operations or one-off internal analyses.
-
Measurement uncertainty and significant figures in technical reports
A measured value without an uncertainty is incomplete: repeat the measurement, report the mean with the standard deviation of the mean and the number of runs, round the uncertainty to one or two significant figures and the value to the same place, and say what the interval means; digits beyond the uncertainty are noise.
-
Correlation versus causation in incident and operations data
A correlation coefficient measures how two series move together; it says nothing about which one drives the other, whether a third factor such as traffic drives both, or whether the data were selected by the outcome. Plot first, condition on the obvious common causes, check timing, and confirm with an intervention such as a flag or canary before acting.
-
p-values: what they measure and what they do not
A p-value is the probability, computed under the whole statistical model including the null hypothesis, of a test statistic at least as extreme as the observed one. It is not the probability that the null is true, not the probability that chance produced the result, not a measure of effect size, and 0.05 is a convention rather than a boundary between truth and noise.
-
Log scales, truncated axes and other ways a chart misleads
A log axis turns equal ratios into equal distances and is the right choice for data spanning orders of magnitude, but it hides absolute differences and cannot show zero; a bar chart whose axis does not start at zero lies about proportions. Label the scale, keep the baseline for bars, and use symlog for counts that include zero.
-
How far off were the variance assumptions behind sample-size calculations in small online experiments, and in which direction?
Open question: the NIST/SEMATECH handbook notes that the classic sample-size formula requires the standard deviation to be known, and in practice it is guessed from earlier data; for small product experiments planned this way, how did the assumed variance compare with the variance observed once the data arrived, was the error systematically optimistic, and what did teams do when the experiment turned out to be underpowered?
-
How should a dashboard show the uncertainty of a metric so that operators react to signal rather than noise?
Open question: dashboards draw a percentage from three requests with the same confidence as one from three million, and a p99 from a sparse histogram bucket as a precise line; which ways of showing sample counts, interval bands or estimation error have been shown to reduce false alarms and missed problems for on-call operators?
-
Reservoir sampling: a uniform sample from a stream of unknown length
Keep k items from a stream without knowing its length: fill the reservoir with the first k, then replace a random slot with probability k/i for item i. One pass, O(k) memory, every item ends up in the sample with probability exactly k/n, shown by a short telescoping argument.
-
Choosing classification metrics: precision, recall, F1, thresholds and calibration
Accuracy hides what matters when classes are unequal or errors have different costs; precision and recall describe the two error types, F1 combines them, threshold-free scores describe the ranking, and calibration says whether a predicted probability of 0.8 means 80 percent. Pick the metric from the decision the model supports, before training.
-
Improvements measured after targeting the worst-performing cases are partly regression to the mean
Hypothesis: when an engineering effort selects the worst-scoring endpoints, tests, hosts or tenants by a noisy metric and reports their improvement after intervention, a substantial part of the reported gain would have occurred without the intervention, because extreme measurements tend to be followed by less extreme ones. A proposed test with an untouched control group is described.
-
Confidence intervals in outline: what the interval says and what it does not
A 95% confidence interval comes from a procedure that captures the true value in 95% of repeated samples; one particular interval either contains it or not. Its width shrinks with the square root of the sample size and grows with the spread. Read it as the range of values compatible with the data, report it next to every estimate, and compute the interval of a difference when comparing.
-
Overfitting and regularisation in outline: bias, variance and the penalty knob
A model overfits when it learns noise in the training rows and its validation score falls behind its training score; regularisation trades some fit for stability by penalising large coefficients or limiting model capacity, and learning and validation curves show which side of the trade-off a model is on.
-
Analysing an A/B test: fixed horizons, peeking and multiple comparisons
Two habits quietly turn an A/B test into a random number generator: stopping when the p-value first dips below the threshold, and testing many metrics or segments until one of them 'wins'. Fix the horizon and the primary metric in advance, use a sequential method if the results must be watched, correct secondary comparisons, and report everything that was looked at.
-
Pre-registering a small experiment before looking at the data
Write down the hypothesis, primary outcome, sample and stopping rule, exclusions and analysis plan, then commit or register the document with a timestamp before collecting or seeing the data; report the planned analysis first and label everything else exploratory.
-
Effect size versus statistical significance: which one decides
Significance says whether the data are unusual under the null model at the sample size used; the effect size says how big the difference is in units that matter. Large systems make trivial effects significant and small pilots make large effects non-significant. Define the smallest effect worth acting on before the experiment and compare the interval to it.
-
Variance, standard deviation, MAD and IQR: reporting the spread
A location without a spread is half a number. The standard deviation describes spread well for roughly normal data and is dominated by the tails otherwise; the interquartile range and the median absolute deviation describe the bulk. Name the measure, give the sample size, use the n−1 form for estimates from samples, and never label an error bar with a bare ±.
-
Estimating how many samples a comparison needs before collecting them
Small samples mislead because their means and spreads wander far from the truth, so a difference between two small groups is often noise. Decide the smallest difference worth detecting, estimate the spread from a pilot, choose the error rates, and compute the sample size per group before the comparison; it grows with the square of spread over difference.
-
Mean, median and mode: choosing a summary statistic that does not mislead
The arithmetic mean is pulled by skew and outliers, the median ignores how far the extremes go, and the mode only makes sense for categorical or clearly peaked data. Choose by the shape of the distribution and by the question asked, print the count next to every summary, and never average ratios or percentiles.
-
Simpson's paradox and base-rate neglect in reports
Two arithmetic effects make a correct table support a wrong sentence: an association can reverse when a population is split into groups that were mixed in different proportions, and a signal's accuracy says little about what a positive signal means until the base rate is known. Ask how groups were mixed and keep denominators visible.
機械可読： JSON