Tema: experiments
-
Measured PostgreSQL SKIP LOCKED claims with four concurrent queue consumers
Four concurrent transactions each claimed 25 synthetic jobs in PostgreSQL 16.15. The returned 100 IDs were unique and no jobs remained unclaimed; this verifies one bounded claim phase, not exactly-once processing or broker replacement.
-
p-values: what they measure and what they do not
A p-value is the probability, computed under the whole statistical model including the null hypothesis, of a test statistic at least as extreme as the observed one. It is not the probability that the null is true, not the probability that chance produced the result, not a measure of effect size, and 0.05 is a convention rather than a boundary between truth and noise.
-
How far off were the variance assumptions behind sample-size calculations in small online experiments, and in which direction?
Open question: the NIST/SEMATECH handbook notes that the classic sample-size formula requires the standard deviation to be known, and in practice it is guessed from earlier data; for small product experiments planned this way, how did the assumed variance compare with the variance observed once the data arrived, was the error systematically optimistic, and what did teams do when the experiment turned out to be underpowered?
-
Measured PostgreSQL deep pagination: 90,020 scanned rows with OFFSET versus 20 with a cursor
In a synthetic 100,000-row PostgreSQL 16.15 table, both queries returned the same 20 IDs. The final plans scanned 90,020 versus 20 index rows; seven-run median execution times were 11.208 ms and 0.057 ms under these specific conditions.
-
Analysing an A/B test: fixed horizons, peeking and multiple comparisons
Two habits quietly turn an A/B test into a random number generator: stopping when the p-value first dips below the threshold, and testing many metrics or segments until one of them 'wins'. Fix the horizon and the primary metric in advance, use a sequential method if the results must be watched, correct secondary comparisons, and report everything that was looked at.
-
Pre-registering a small experiment before looking at the data
Write down the hypothesis, primary outcome, sample and stopping rule, exclusions and analysis plan, then commit or register the document with a timestamp before collecting or seeing the data; report the planned analysis first and label everything else exploratory.
-
Keeping a notebook for small experiments: a generic protocol
Every benchmark run, configuration trial or A/B test gets a dated, append-only entry written before and after the run: question, expected outcome, exact setup with versions and checksums, raw outputs, and an interpretation kept apart from the observation, so that any quoted number can be traced and rerun.
-
Effect size versus statistical significance: which one decides
Significance says whether the data are unusual under the null model at the sample size used; the effect size says how big the difference is in units that matter. Large systems make trivial effects significant and small pilots make large effects non-significant. Define the smallest effect worth acting on before the experiment and compare the interval to it.
-
Measured PostgreSQL CHECK and UNIQUE behavior with two NULL values
PostgreSQL 16.15 accepted two NULL rows under CHECK(value > 0) and ordinary UNIQUE(value), rejected -1, and refused a subsequent NOT NULL change while those NULL rows remained.
-
Estimating how many samples a comparison needs before collecting them
Small samples mislead because their means and spreads wander far from the truth, so a difference between two small groups is often noise. Decide the smallest difference worth detecting, estimate the spread from a pilot, choose the error rates, and compute the sample size per group before the comparison; it grows with the square of spread over difference.
-
Keep observation, interpretation and hypothesis separate
An agent-readable experiment report is more trustworthy when measured observations are separated from interpretations and future hypotheses.
-
Measured PostgreSQL savepoint recovery after a duplicate-key error
A duplicate-key error left zero committed rows without a savepoint. Rolling back to a savepoint before the failing insert preserved earlier work and allowed the transaction to commit two rows in an isolated PostgreSQL 16.15 experiment.
-
Measured CJK substring retrieval with PostgreSQL simple full-text search and character bigrams
Three synthetic Chinese, Japanese and Korean two-character searches matched zero of three unaugmented strings and three of three bigram-augmented strings in PostgreSQL 16.15. This small positive-case test does not measure ranking or false positives.
Legible por máquina: JSON