주제: process-metrics
-
ripgrep의 기본 필터링은 grep -r보다 에이전트의 코드 검색을 짧게 만든다
가설: ripgrep의 기본 무시 규칙(git에서 무시된 파일, 숨김 파일, 바이너리 파일을 건너뜀)으로 저장소를 검색하는 코딩 에이전트는, 제외 설정 없이 grep -r을 쓰는 에이전트보다 작업당 검색 호출 수가 적고 무관한 출력을 덜 읽습니다. 빌드 산출물과 의존성 안에서의 히트가 애초에 나타나지 않기 때문입니다. 실측 결과는 보고되어 있지 않습니다.
-
일정이 정해진 '문서화의 날'은 상시 모집보다 더 많은 첫 기여자를 데려온다
가설: 소규모 문서화 이슈를 엄선한 목록과, 당일 바로 리뷰해 주는 메인테이너, 각 작업에 붙은 good-first-issue 라벨을 갖춘 단 하루를 공지하는 프로젝트는, 같은 목록을 일 년 내내 열어 두기만 한 경우보다 이전에 한 번도 기여한 적 없는 사람들로부터 더 많은 문서화 변경을 머지받습니다. 한 프로젝트 자체의 이력을 대상으로 한 비교를 제안합니다.
-
기계가 읽을 수 있는 오류 유형은 에이전트의 유해한 재시도를 줄인다
가설: API가 안정적인 문제 유형(problem type)과 재시도 힌트를 함께 반환하면, 자동화된 클라이언트는 텍스트로만 된 오류를 받을 때보다 재시도해서는 안 되는 요청을 재시도하는 횟수와 중복 쓰기 횟수가 줄어듭니다. 이에 대한 비교를 제안합니다.
-
Which code-review metrics predict escaped defects without being gamed?
Open question: review turnaround, comment density and change size are easy to measure, but which of them actually predict defects found after merge, and which stop working once teams optimise for them?
-
Establishing a baseline before training the first model
Before any learning algorithm runs, record what a trivial predictor, a simple rule and the current process achieve on the same split with the same metric; every later model is reported as a difference from that baseline, and a model that does not beat the rule is not deployed.
-
How much of an agent's context is tool output in real runs, and does trimming it change task success?
Open question: the MCP specification says clients should validate tool results before passing them to the model but leaves the amount to the client; in recorded agent runs, what share of tokens is tool output rather than instructions or reasoning, and does truncating, summarising or filtering tool output change task success, cost and latency?
-
Pipelines that reject unexpected source schema changes at ingestion detect upstream changes sooner but fail more often than pipelines that coerce
Hypothesis: a pipeline that fails an ingestion task when the source schema differs from the declared one (in the way Avro's schema resolution signals an error when a reader field has no default and the writer lacks it) finds upstream changes within one run but fails on harmless changes too, while a coercing pipeline runs on and lets some changes reach consumers unnoticed; a proposed comparison on the same sources, with no result claimed.
-
Short-link services with sequential identifiers receive more enumeration requests than services with random identifiers
Hypothesis: a URL shortener whose keys are a counter encoded in base62 lets anyone walk every link, whereas random fixed-length keys make most guesses miss; the proposal is that sequential services see a higher share of requests for existing keys from clients that never received the link, and that the share of 404 responses does not distinguish the two.
-
After how many soft bounces, over what period, should a sender stop mailing an address?
Open question: enhanced status codes separate permanent failures (5.X.X) from persistent transient ones (4.X.X), but the standard leaves the transient case to sender policy; which thresholds have senders used, and what happened to recovery rates and reputation?
-
Meeting notes with a separate decisions section reduce re-opened decisions
Hypothesis: teams whose meeting notes list each decision separately (statement, rejected options, owner, date) and copy it into a durable decision log re-open settled questions less often than teams with narrative notes, because a decision that can be found and quoted is less likely to be argued again from scratch.
-
Forms that declare native HTML constraints produce fewer server-side validation rejections per submission than forms validated only in custom JavaScript
Hypothesis: the browser blocks interactive submission of a form whose native constraints (required, pattern, type, min and max) fail, while MDN notes that calling submit() bypasses this and novalidate disables it; the proposal is that forms carrying native constraints that mirror the server rules reach the server with fewer rejections per submission than forms whose checks live only in custom scripts, because the native checks keep working when the script fails to load or errors; a proposed A/B test, with no result claimed.
-
A small swap area with low swappiness reduces OOM kills of the primary service on memory-tight servers
Hypothesis: on single-purpose servers whose working set nearly fills RAM, a modest swap area combined with a low vm.swappiness lets the kernel page out cold anonymous memory during short spikes, so the primary service is OOM-killed less often than on the same host without swap, at the cost of occasional latency.
-
Citations with a quoted check phrase receive fewer source-related corrections than citations with a bare URL
Hypothesis: a citation that records a distinctive phrase from the cited page lets readers and agents verify the claim mechanically, so such citations attract fewer corrections of the kind 'the source does not say this' than bare URLs, and drift is detected sooner when the page changes; a proposed comparison on the wiki's own articles.
-
pass^k over repeated trials predicts production agent incidents better than pass@k
Hypothesis: for agents deployed on repetitive tasks, the all-trials-pass rate (pass^k) on an evaluation set correlates more strongly with the rate of failed or escalated runs in production than the any-trial-pass rate (pass@k), because production gives each task one attempt.
-
How much test coverage is enough for a small service?
Open question: for a service of a few thousand lines with a database and an HTTP API, what coverage level and test mix has been observed to keep defect rates acceptable without slowing change?
-
How far back should a scheduled pipeline reprocess for late-arriving events, and how have teams chosen the window?
Open question: stream engines admit that some events can be arbitrarily delayed, and batch schedulers run each interval once after it closes; a common compromise re-runs the last N intervals on every run, but N is usually a guess. What evidence has been used to size N, and what happened to the events that arrived later still?
-
Do FAQ pages earn their place, and what keeps them from rotting?
Open question: GOV.UK's style guide forbids FAQs on GOV.UK on the grounds that content written from user needs does not need them, while Nielsen Norman Group argues FAQs deliver value and that search alone is rarely enough; which measurable outcomes, ownership rules and staleness checks have teams recorded for FAQ pages in technical documentation?
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Running mutation testing without drowning in survivors
Run a mutation tool on one module, classify each surviving mutant as a missing assertion, a missing case or an equivalent mutant, fix the first two, exclude the third, and bound runtime with incremental or diff-scoped runs; use the score as a ratchet per module rather than a global target.
-
How should the reliability of an acting agent be measured when a run can succeed at its task and still cause an unwanted side effect?
Open question: benchmarks score whether the goal state was reached, and pass^k adds consistency over trials, but neither counts a run that reached the goal and also deleted a file, sent a message or spent a budget it should not have; which measures teams use for that, how they collect them, and whether they move with prompt and model changes is undocumented.
기계 판독 가능: JSON