テーマ： statistics
-
中央値・パーセンタイル・比率のブートストラップ信頼区間
生の観測値を復元抽出で何度もリサンプリングし、各リサンプルについて統計量を計算し、得られた分布から区間を読み取る。これにより、教科書の公式が通用しない中央値・パーセンタイル・比率・差分についての不確実性を求められる。手法名、リサンプリング回数、サンプルサイズを報告し、小さなサンプルの極端なパーセンタイルについては信用しないこと。
-
Differential privacy in one paragraph, and where it does not fit
Differential privacy bounds how much any one person's record can change the output distribution of a query mechanism, by adding calibrated noise and charging each answer to a budget; it fits repeated aggregate releases over large populations and does not fit record-level data, small groups, exact operations or one-off internal analyses.
-
技術報告における測定の不確かさと有効数字
不確かさを示さない測定値は不完全です。測定を繰り返し、平均と平均の標準偏差、実行回数を報告します。不確かさを有効数字1桁か2桁に丸め、値も同じ桁にそろえ、区間の意味を示します。不確かさが裏付ける範囲を超えた桁はノイズです。
-
Correlation versus causation in incident and operations data
A correlation coefficient measures how two series move together; it says nothing about which one drives the other, whether a third factor such as traffic drives both, or whether the data were selected by the outcome. Plot first, condition on the obvious common causes, check timing, and confirm with an intervention such as a flag or canary before acting.
-
p-values: what they measure and what they do not
A p-value is the probability, computed under the whole statistical model including the null hypothesis, of a test statistic at least as extreme as the observed one. It is not the probability that the null is true, not the probability that chance produced the result, not a measure of effect size, and 0.05 is a convention rather than a boundary between truth and noise.
-
Log scales, truncated axes and other ways a chart misleads
A log axis turns equal ratios into equal distances and is the right choice for data spanning orders of magnitude, but it hides absolute differences and cannot show zero; a bar chart whose axis does not start at zero lies about proportions. Label the scale, keep the baseline for bars, and use symlog for counts that include zero.
-
How far off were the variance assumptions behind sample-size calculations in small online experiments, and in which direction?
Open question: the NIST/SEMATECH handbook notes that the classic sample-size formula requires the standard deviation to be known, and in practice it is guessed from earlier data; for small product experiments planned this way, how did the assumed variance compare with the variance observed once the data arrived, was the error systematically optimistic, and what did teams do when the experiment turned out to be underpowered?
-
How should a dashboard show the uncertainty of a metric so that operators react to signal rather than noise?
Open question: dashboards draw a percentage from three requests with the same confidence as one from three million, and a p99 from a sparse histogram bucket as a precise line; which ways of showing sample counts, interval bands or estimation error have been shown to reduce false alarms and missed problems for on-call operators?
-
リザーバサンプリング：長さが不明なストリームから一様に抽出する
長さを知らずにストリームから k 個を保持します。最初の k 個でリザーバを埋め、その後の項目 i では確率 k/i でランダムなスロットを置き換えます。一回の走査、O(k) のメモリで、各項目が最終標本に入る確率は正確に k/n です。短い積の相殺による議論で示せます。
-
Choosing classification metrics: precision, recall, F1, thresholds and calibration
Accuracy hides what matters when classes are unequal or errors have different costs; precision and recall describe the two error types, F1 combines them, threshold-free scores describe the ranking, and calibration says whether a predicted probability of 0.8 means 80 percent. Pick the metric from the decision the model supports, before training.
-
Improvements measured after targeting the worst-performing cases are partly regression to the mean
Hypothesis: when an engineering effort selects the worst-scoring endpoints, tests, hosts or tenants by a noisy metric and reports their improvement after intervention, a substantial part of the reported gain would have occurred without the intervention, because extreme measurements tend to be followed by less extreme ones. A proposed test with an untouched control group is described.
-
信頼区間の概要：区間が示すこと、示さないこと
95%信頼区間は、標本抽出を繰り返したときに95%の区間が真の値を含む手順から得られます。特定の区間は真の値を含むか、含まないかのどちらかです。幅は標本数の平方根に応じて狭まり、ばらつきとともに広がります。データと整合する値の範囲として読み、各推定値と併記し、比較時には差の区間を計算します。
-
過学習と正則化の概観：バイアス、分散、ペナルティの調整
モデルが訓練行のノイズを学び、検証スコアが訓練スコアを下回ると、過学習が起きる。正則化は大きな係数にペナルティを与えたり、モデルの表現能力を制限したりして、当てはまりの一部と引き換えに安定性を得る。学習曲線と検証曲線は、モデルがこのトレードオフのどちら側にあるかを示す。
-
Analysing an A/B test: fixed horizons, peeking and multiple comparisons
Two habits quietly turn an A/B test into a random number generator: stopping when the p-value first dips below the threshold, and testing many metrics or segments until one of them 'wins'. Fix the horizon and the primary metric in advance, use a sequential method if the results must be watched, correct secondary comparisons, and report everything that was looked at.
-
データを見る前に小規模な実験を事前登録する
仮説、主要評価項目、標本と停止規則、除外条件、分析計画を書き出し、データを収集または閲覧する前に、タイムスタンプ付きで文書をコミットまたは登録する。計画した分析を最初に報告し、それ以外はすべて探索的分析と明示する。
-
Effect size versus statistical significance: which one decides
Significance says whether the data are unusual under the null model at the sample size used; the effect size says how big the difference is in units that matter. Large systems make trivial effects significant and small pilots make large effects non-significant. Define the smallest effect worth acting on before the experiment and compare the interval to it.
-
Variance, standard deviation, MAD and IQR: reporting the spread
A location without a spread is half a number. The standard deviation describes spread well for roughly normal data and is dominated by the tails otherwise; the interquartile range and the median absolute deviation describe the bulk. Name the measure, give the sample size, use the n−1 form for estimates from samples, and never label an error bar with a bare ±.
-
Estimating how many samples a comparison needs before collecting them
Small samples mislead because their means and spreads wander far from the truth, so a difference between two small groups is often noise. Decide the smallest difference worth detecting, estimate the spread from a pilot, choose the error rates, and compute the sample size per group before the comparison; it grows with the square of spread over difference.
-
Mean, median and mode: choosing a summary statistic that does not mislead
The arithmetic mean is pulled by skew and outliers, the median ignores how far the extremes go, and the mode only makes sense for categorical or clearly peaked data. Choose by the shape of the distribution and by the question asked, print the count next to every summary, and never average ratios or percentiles.
-
Simpson's paradox and base-rate neglect in reports
Two arithmetic effects make a correct table support a wrong sentence: an association can reverse when a population is split into groups that were mixed in different proportions, and a signal's accuracy says little about what a positive signal means until the base rate is known. Ask how groups were mixed and keep denominators visible.
機械可読： JSON