How should a product feature backed by a language model be evaluated when there is no single correct output?

Open question

A supervised classifier comes with an evaluation recipe: a held-out split, a metric chosen from the decision, a confusion matrix, an interval. A feature built on a language model (summarise this ticket, draft a reply, extract the parties from a contract, answer a question over documents) often has no single correct output, and the model behind it changes under the team's feet with every prompt edit, provider update or retrieval change. Teams assemble something: a few hundred hand-graded examples, a short rubric, a second model that grades the first, thumbs-up counts from users, and a regression suite of prompts that must not get worse. What is not recorded on the wiki is which of these parts carried weight over time. When a grading model is used, how was its agreement with human graders measured, and did it stay stable after the grader's own model was upgraded? The scikit-learn documentation describes Cohen's kappa as a score for inter-annotator agreement; is a kappa between human graders, or between human and model graders, reported anywhere, and at what level does a team treat the grader as trustworthy? For extraction-like features with a checkable output, does the team fall back to ordinary precision and recall per field, and does that suffice? How are examples chosen so that the set does not silently become a set of the cases the current prompt already handles? And which production signal, if any, moved together with the offline grades when a change was shipped?

What a useful answer contains

The feature type (free text, constrained text, structured output) and the size and origin of the labelled set. The rubric, with each criterion stated as a question a grader can answer. The grading setup: how many human graders, whether a model grader was used, and the agreement measured between them with the statistic named and the number of items. What happened to the evaluation across at least two model or prompt changes: which cases regressed, whether the offline score predicted user-visible change, and what was added to the set afterwards. The cost in grader time per evaluation round, since that decides whether the process survives. Negative reports, such as a model grader that agreed with humans on the first version and diverged after an upgrade, are as useful as positive ones. Single-feature accounts are welcome if the feature type and the set size are stated; comparisons across features in one team are more useful.