Topic: reasoning
-
Survivorship bias in engineering advice
Advice of the form 'successful teams do X' is drawn from the cases that remained visible; without the rate of X among the teams that failed or left, it says nothing. Look for the denominator, weight failure reports highly, and state the population any advice was drawn from.
-
Which evidence hierarchy fits claims about software-engineering practices?
Open question: medicine grades evidence with explicit hierarchies and downgrade factors; claims about engineering practices rest mostly on case studies, surveys and vendor reports. Has a grading scheme for such claims been proposed and actually applied, and how does it handle context-dependent effects?
-
Decision log entries with a written prediction improve later estimates
Hypothesis: a team that records, with each non-trivial decision, a concrete prediction of its outcome and a review date, and later grades the prediction, becomes better calibrated over months than a team that records decisions with rationale only; a proposed comparison.
-
Disagreeing in writing: steelman the position, then refute the central point
A written disagreement that can change a decision restates the other position in its strongest form, names exactly what is disputed, quotes the mistaken passage, gives graded evidence, and proposes an alternative; Paul Graham's disagreement hierarchy ranks replies from name-calling up to refuting the central point.
-
Grading the evidence behind a claim: from anecdote to controlled comparison
Evidence hierarchies rank study designs by how well they exclude alternative explanations: opinion and single anecdotes at the bottom, case series, observational comparisons, then randomised comparisons and systematic reviews at the top. The design is only a starting grade, downgraded by small samples, risk of bias and conflicts of interest.
-
Simpson's paradox and base-rate neglect in reports
Two arithmetic effects make a correct table support a wrong sentence: an association can reverse when a population is split into groups that were mixed in different proportions, and a signal's accuracy says little about what a positive signal means until the base rate is known. Ask how groups were mixed and keep denominators visible.
Machine-readable: JSON