Тема: testing
-
Бенчмаркинг изменения: прогрев, повторы, разброс и что указывать в отчёте
Сравнение по времени выполнения — это результат только тогда, когда он устойчив к шуму: зафиксируйте нагрузку, отбросьте прогревочные прогоны, чередуйте много повторов каждого варианта, выбирайте статистику до того, как посмотрите на данные, и указывайте разброс и окружение рядом с каждым числом. Разница меньше разброса между прогонами — это не находка.
-
Refactoring in small, verified steps
Refactoring changes structure without changing behaviour; doing it in tiny steps with tests green between steps, and separating refactoring commits from behaviour changes, keeps it safe and reviewable.
-
Einen brauchbaren Fehlerbericht schreiben
Ein Fehlerbericht ist brauchbar, wenn eine fremde Person den Fehler ohne Rückfrage nachstellen kann: eine präzise Überschrift, Umgebung mit Versionen, nummerierte Schritte zum Nachstellen, erwartetes und tatsächliches Ergebnis getrennt, die wörtliche Fehlermeldung und ein möglichst kleines Beispiel. Vermutungen zur Ursache stehen in einem eigenen Abschnitt.
-
Seed data and fixtures for local databases: small, idempotent and versioned with the schema
Separate reference data (needed everywhere), sample data (development and demos) and test data (created by tests); write the seed as idempotent code with upserts on natural keys, keep the sample set small and named, run it after migrations in both the setup script and CI, and never seed developer machines from raw production data.
-
Property-based testing with generated inputs
Instead of hand-picked examples, a property-based test states an invariant and lets a library generate many inputs, shrinking failures to minimal counterexamples; Hypothesis is the reference implementation for Python.
-
Generate, critique, revise: when a self-verification loop pays for itself
A loop in which the model critiques and revises its own output improves results when the critique has an external signal (tests, a validator, a source) and a fixed rubric; without one, published results show it can degrade answers, and each round adds at least two calls whose input grows with the draft.
-
Writing a unit test in JUnit 5 and xUnit.net: annotations, lifecycle and parameterised cases side by side
JUnit Jupiter marks tests with @Test, runs @BeforeEach and @AfterEach around each one on a fresh instance by default, and drives data-driven cases with @ParameterizedTest plus a source annotation; xUnit.net uses [Fact], the constructor and IDisposable for per-test setup on a fresh instance, [Theory] with [InlineData] for cases, and fixtures for shared expensive context. Writing tests with the same shape in both keeps a polyglot team's conventions aligned.
-
Designing a continuous integration pipeline
A CI pipeline should be fast, self-testing and identical for every change: build from a clean checkout, run linters and tests in stages, fail loudly, and keep the total time short enough that people wait for it.
-
Ephemeral databases in containers for integration tests
Start the real database engine in a throwaway container per test run, keep its data on memory, apply migrations once to a template database and copy it per test file; tests then exercise the real planner, constraints and isolation behaviour instead of an in-memory substitute.
-
Use property tests for parsers
Express parser invariants over generated inputs and keep minimized failures as focused regression examples.
-
pass^k over repeated trials predicts production agent incidents better than pass@k
Hypothesis: for agents deployed on repetitive tasks, the all-trials-pass rate (pass^k) on an evaluation set correlates more strongly with the rate of failed or escalated runs in production than the any-trial-pass rate (pass@k), because production gives each task one attempt.
-
Reproducibility of a machine-learning experiment: seeds, environment, data and the limits of determinism
Rerunning an experiment and getting the same number requires fixed random states passed explicitly, pinned library versions, an identified dataset and split, and awareness that GPU kernels and library releases can still change results; the protocol makes runs repeatable where possible and documents where they are not.
-
Red-teaming an agent workflow before it gets real permissions
Attack the agent the way content and users will: indirect prompt injection through every input it reads, tool-argument manipulation, exfiltration through tool calls and budget exhaustion; run scripted probes plus manual attempts, record what the agent did, and fix the boundary, not only the prompt.
-
Test data without production personal data
Give development, CI and staging realistic data by generating it: classify columns, write seeded generators that pass the application's validators, produce volume with generate_series, copy only distributions from production, mark generated records recognisably, and remove the shortcut of dumping production.
-
How much test coverage is enough for a small service?
Open question: for a service of a few thousand lines with a database and an HTTP API, what coverage level and test mix has been observed to keep defect rates acceptable without slowing change?
-
Structured extraction from documents with JSON Schema, validation and bounded retries
Define the target record as a JSON Schema with additionalProperties false, ask the model for exactly that shape, validate every response with a real validator, retry a bounded number of times with the validation error in the prompt, and route what still fails to a person instead of guessing.
-
Reviewing code written by an AI agent
A proposed review protocol for generated changes: compare the diff with the request, confirm every API and dependency exists, verify test claims by running and breaking the tests, read tests before implementation, hunt for swallowed errors, and record what was checked.
-
The test-driven development loop
Write a failing test, make it pass with the simplest change, then refactor with the tests green; the loop keeps design decisions small and gives every line a reason to exist.
-
Stable selectors and auto-waiting in browser end-to-end tests
Two causes dominate flaky browser tests: selectors that break or match the wrong element, and sleeps that assume timing. Locate elements by role, label or test id, make every locator match exactly one element, and synchronise on retrying assertions and actionability checks rather than on time.
-
Running mutation testing without drowning in survivors
Run a mutation tool on one module, classify each surviving mutant as a missing assertion, a missing case or an equivalent mutant, fix the first two, exclude the third, and bound runtime with incremental or diff-scoped runs; use the score as a ratchet per module rather than a global target.
Машиночитаемо: JSON