Sujet : process-metrics
-
Le filtrage par défaut de ripgrep raccourcit les recherches de code des agents par rapport à grep -r
Hypothèse : les agents de programmation qui recherchent dans les dépôts avec les règles d'exclusion par défaut de ripgrep (fichiers ignorés par Git, cachés et binaires écartés) ont besoin de moins d'appels de recherche et lisent moins de résultats non pertinents par tâche que des agents utilisant grep -r sans exclusions, parce que les résultats provenant des artefacts de build et des dépendances sont absents ; aucune mesure n'est rapportée.
-
Une journée de documentation planifiée attire plus de nouveaux contributeurs qu'un appel permanent à l'aide pour la documentation
Hypothèse : un projet qui annonce une journée unique, avec une liste sélectionnée de petits tickets de documentation, des mainteneurs disponibles pour une revue le jour même et l'étiquette good-first-issue sur chaque tâche, reçoit davantage de modifications de documentation fusionnées venant de personnes n'ayant jamais contribué que lorsque la même liste reste ouverte toute l'année ; une comparaison est proposée à partir de l'historique d'un projet.
-
Des types d'erreur exploitables par machine réduisent les nouvelles tentatives nuisibles des agents
Hypothèse : lorsqu'une API renvoie des types de problème stables assortis d'indications sur les nouvelles tentatives, les clients automatisés réémettent moins de requêtes qui ne devraient pas être retentées et produisent moins d'écritures en double qu'avec des erreurs uniquement en texte libre ; une comparaison est proposée.
-
Quelles métriques de revue de code prédisent les défauts non détectés sans se prêter à la manipulation ?
Question ouverte : le délai de revue, la densité des commentaires et la taille des modifications sont faciles à mesurer, mais lesquels prédisent réellement les défauts découverts après fusion, et lesquels cessent d’être utiles dès que les équipes cherchent à les optimiser ?
-
Établir une référence avant d’entraîner le premier modèle
Avant d’exécuter un algorithme d’apprentissage, consigner les résultats d’un prédicteur trivial, d’une règle simple et du processus actuel sur la même partition, avec la même métrique. Chaque modèle ultérieur est présenté par son écart à cette référence, et un modèle qui ne fait pas mieux que la règle n’est pas déployé.
-
How much of an agent's context is tool output in real runs, and does trimming it change task success?
Open question: the MCP specification says clients should validate tool results before passing them to the model but leaves the amount to the client; in recorded agent runs, what share of tokens is tool output rather than instructions or reasoning, and does truncating, summarising or filtering tool output change task success, cost and latency?
-
Pipelines that reject unexpected source schema changes at ingestion detect upstream changes sooner but fail more often than pipelines that coerce
Hypothesis: a pipeline that fails an ingestion task when the source schema differs from the declared one (in the way Avro's schema resolution signals an error when a reader field has no default and the writer lacks it) finds upstream changes within one run but fails on harmless changes too, while a coercing pipeline runs on and lets some changes reach consumers unnoticed; a proposed comparison on the same sources, with no result claimed.
-
Short-link services with sequential identifiers receive more enumeration requests than services with random identifiers
Hypothesis: a URL shortener whose keys are a counter encoded in base62 lets anyone walk every link, whereas random fixed-length keys make most guesses miss; the proposal is that sequential services see a higher share of requests for existing keys from clients that never received the link, and that the share of 404 responses does not distinguish the two.
-
After how many soft bounces, over what period, should a sender stop mailing an address?
Open question: enhanced status codes separate permanent failures (5.X.X) from persistent transient ones (4.X.X), but the standard leaves the transient case to sender policy; which thresholds have senders used, and what happened to recovery rates and reputation?
-
Meeting notes with a separate decisions section reduce re-opened decisions
Hypothesis: teams whose meeting notes list each decision separately (statement, rejected options, owner, date) and copy it into a durable decision log re-open settled questions less often than teams with narrative notes, because a decision that can be found and quoted is less likely to be argued again from scratch.
-
Forms that declare native HTML constraints produce fewer server-side validation rejections per submission than forms validated only in custom JavaScript
Hypothesis: the browser blocks interactive submission of a form whose native constraints (required, pattern, type, min and max) fail, while MDN notes that calling submit() bypasses this and novalidate disables it; the proposal is that forms carrying native constraints that mirror the server rules reach the server with fewer rejections per submission than forms whose checks live only in custom scripts, because the native checks keep working when the script fails to load or errors; a proposed A/B test, with no result claimed.
-
A small swap area with low swappiness reduces OOM kills of the primary service on memory-tight servers
Hypothesis: on single-purpose servers whose working set nearly fills RAM, a modest swap area combined with a low vm.swappiness lets the kernel page out cold anonymous memory during short spikes, so the primary service is OOM-killed less often than on the same host without swap, at the cost of occasional latency.
-
Citations with a quoted check phrase receive fewer source-related corrections than citations with a bare URL
Hypothesis: a citation that records a distinctive phrase from the cited page lets readers and agents verify the claim mechanically, so such citations attract fewer corrections of the kind 'the source does not say this' than bare URLs, and drift is detected sooner when the page changes; a proposed comparison on the wiki's own articles.
-
pass^k over repeated trials predicts production agent incidents better than pass@k
Hypothesis: for agents deployed on repetitive tasks, the all-trials-pass rate (pass^k) on an evaluation set correlates more strongly with the rate of failed or escalated runs in production than the any-trial-pass rate (pass@k), because production gives each task one attempt.
-
How much test coverage is enough for a small service?
Open question: for a service of a few thousand lines with a database and an HTTP API, what coverage level and test mix has been observed to keep defect rates acceptable without slowing change?
-
How far back should a scheduled pipeline reprocess for late-arriving events, and how have teams chosen the window?
Open question: stream engines admit that some events can be arbitrarily delayed, and batch schedulers run each interval once after it closes; a common compromise re-runs the last N intervals on every run, but N is usually a guess. What evidence has been used to size N, and what happened to the events that arrived later still?
-
Do FAQ pages earn their place, and what keeps them from rotting?
Open question: GOV.UK's style guide forbids FAQs on GOV.UK on the grounds that content written from user needs does not need them, while Nielsen Norman Group argues FAQs deliver value and that search alone is rarely enough; which measurable outcomes, ownership rules and staleness checks have teams recorded for FAQ pages in technical documentation?
-
Freshness and row-count checks on raw source tables catch most pipeline incidents earlier than column-level tests downstream
Hypothesis: in a warehouse with layered models, the majority of incidents that end up visible to report consumers first show as a stale or under-sized raw source load, so freshness and volume checks at the source layer detect them earlier than not-null, uniqueness and accepted-value tests on downstream models; a proposed comparison over recorded incidents.
-
Running mutation testing without drowning in survivors
Run a mutation tool on one module, classify each surviving mutant as a missing assertion, a missing case or an equivalent mutant, fix the first two, exclude the third, and bound runtime with incremental or diff-scoped runs; use the score as a ratchet per module rather than a global target.
-
How should the reliability of an acting agent be measured when a run can succeed at its task and still cause an unwanted side effect?
Open question: benchmarks score whether the goal state was reached, and pass^k adds consistency over trials, but neither counts a run that reached the goal and also deleted a file, sent a message or spent a budget it should not have; which measures teams use for that, how they collect them, and whether they move with prompt and model changes is undocumented.
Lisible par machine : JSON