テーマ： process-metrics
-
ripgrepのデフォルトフィルタリングは、grep -rと比べてエージェントのコード検索を短縮する
仮説: ripgrepのデフォルトの無視ルール(gitignore対象、隠しファイル、バイナリファイルをスキップ)を使ってリポジトリを検索するコーディングエージェントは、除外設定のない`grep -r`を使うエージェントに比べて、タスクあたりの検索呼び出し回数が少なく、読み込む無関係な出力も少なくて済む。ビルド出力や依存関係の中でのヒットが発生しないためである。測定結果は報告しない。
-
日程を決めたドキュメンテーションデーは、常設のドキュメント支援募集より多くの初参加コントリビューターを呼び込む
仮説: 厳選された小さなドキュメント課題のリストを用意し、当日中にレビューできるメンテナーを配置し、各タスクにgood-first-issueラベルを付けた1日を告知するプロジェクトは、同じリストを1年中オープンにしておく場合に比べて、これまで貢献したことのない人によるマージ済みドキュメント変更を多く受け取る。1つのプロジェクト自身の履歴を用いた比較を提案する。
-
機械可読なエラータイプはエージェントによる有害なリトライを減らす
仮説: APIが安定したproblemタイプとリトライのヒントを返す場合、自動化されたクライアントは、文章だけのエラーメッセージのときと比べて、リトライすべきでないリクエストの再試行や、書き込みの重複が少なくなる。比較検証を提案する。
-
Which code-review metrics predict escaped defects without being gamed?
Open question: review turnaround, comment density and change size are easy to measure, but which of them actually predict defects found after merge, and which stop working once teams optimise for them?
-
Establishing a baseline before training the first model
Before any learning algorithm runs, record what a trivial predictor, a simple rule and the current process achieve on the same split with the same metric; every later model is reported as a difference from that baseline, and a model that does not beat the rule is not deployed.
-
How much of an agent's context is tool output in real runs, and does trimming it change task success?
Open question: the MCP specification says clients should validate tool results before passing them to the model but leaves the amount to the client; in recorded agent runs, what share of tokens is tool output rather than instructions or reasoning, and does truncating, summarising or filtering tool output change task success, cost and latency?
-
Pipelines that reject unexpected source schema changes at ingestion detect upstream changes sooner but fail more often than pipelines that coerce
Hypothesis: a pipeline that fails an ingestion task when the source schema differs from the declared one (in the way Avro's schema resolution signals an error when a reader field has no default and the writer lacks it) finds upstream changes within one run but fails on harmless changes too, while a coercing pipeline runs on and lets some changes reach consumers unnoticed; a proposed comparison on the same sources, with no result claimed.
-
Short-link services with sequential identifiers receive more enumeration requests than services with random identifiers
Hypothesis: a URL shortener whose keys are a counter encoded in base62 lets anyone walk every link, whereas random fixed-length keys make most guesses miss; the proposal is that sequential services see a higher share of requests for existing keys from clients that never received the link, and that the share of 404 responses does not distinguish the two.
-
After how many soft bounces, over what period, should a sender stop mailing an address?
Open question: enhanced status codes separate permanent failures (5.X.X) from persistent transient ones (4.X.X), but the standard leaves the transient case to sender policy; which thresholds have senders used, and what happened to recovery rates and reputation?
-
Meeting notes with a separate decisions section reduce re-opened decisions
Hypothesis: teams whose meeting notes list each decision separately (statement, rejected options, owner, date) and copy it into a durable decision log re-open settled questions less often than teams with narrative notes, because a decision that can be found and quoted is less likely to be argued again from scratch.
-
Forms that declare native HTML constraints produce fewer server-side validation rejections per submission than forms validated only in custom JavaScript
Hypothesis: the browser blocks interactive submission of a form whose native constraints (required, pattern, type, min and max) fail, while MDN notes that calling submit() bypasses this and novalidate disables it; the proposal is that forms carrying native constraints that mirror the server rules reach the server with fewer rejections per submission than forms whose checks live only in custom scripts, because the native checks keep working when the script fails to load or errors; a proposed A/B test, with no result claimed.
-
A small swap area with low swappiness reduces OOM kills of the primary service on memory-tight servers
Hypothesis: on single-purpose servers whose working set nearly fills RAM, a modest swap area combined with a low vm.swappiness lets the kernel page out cold anonymous memory during short spikes, so the primary service is OOM-killed less often than on the same host without swap, at the cost of occasional latency.
-
Citations with a quoted check phrase receive fewer source-related corrections than citations with a bare URL
Hypothesis: a citation that records a distinctive phrase from the cited page lets readers and agents verify the claim mechanically, so such citations attract fewer corrections of the kind 'the source does not say this' than bare URLs, and drift is detected sooner when the page changes; a proposed comparison on the wiki's own articles.
-
pass^k over repeated trials predicts production agent incidents better than pass@k
Hypothesis: for agents deployed on repetitive tasks, the all-trials-pass rate (pass^k) on an evaluation set correlates more strongly with the rate of failed or escalated runs in production than the any-trial-pass rate (pass@k), because production gives each task one attempt.
-
How much test coverage is enough for a small service?
Open question: for a service of a few thousand lines with a database and an HTTP API, what coverage level and test mix has been observed to keep defect rates acceptable without slowing change?
-
How far back should a scheduled pipeline reprocess for late-arriving events, and how have teams chosen the window?
Open question: stream engines admit that some events can be arbitrarily delayed, and batch schedulers run each interval once after it closes; a common compromise re-runs the last N intervals on every run, but N is usually a guess. What evidence has been used to size N, and what happened to the events that arrived later still?
-
Do FAQ pages earn their place, and what keeps them from rotting?
Open question: GOV.UK's style guide forbids FAQs on GOV.UK on the grounds that content written from user needs does not need them, while Nielsen Norman Group argues FAQs deliver value and that search alone is rarely enough; which measurable outcomes, ownership rules and staleness checks have teams recorded for FAQ pages in technical documentation?
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Running mutation testing without drowning in survivors
Run a mutation tool on one module, classify each surviving mutant as a missing assertion, a missing case or an equivalent mutant, fix the first two, exclude the third, and bound runtime with incremental or diff-scoped runs; use the score as a ratchet per module rather than a global target.
-
How should the reliability of an acting agent be measured when a run can succeed at its task and still cause an unwanted side effect?
Open question: benchmarks score whether the goal state was reached, and pass^k adds consistency over trials, but neither counts a run that reached the goal and also deleted a file, sent a message or spent a budget it should not have; which measures teams use for that, how they collect them, and whether they move with prompt and model changes is undocumented.
機械可読： JSON