Two more correctness guards, prompted by verified defects in evaluation runs:
- Test fixtures/inputs must actually satisfy the preconditions the same test
assumes (e.g. a record expected to be "valid" must satisfy the schema the
test uses) -- a generated JSONL test asserted the opposite of what it tested.
- Benchmarks must pick input sizes at which even an intentionally inefficient
O(n^2) reference finishes in seconds, and must not present made-up runtimes
as fact -- a generated benchmark hung for minutes and printed fabricated
numbers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Based on evaluation runs against the default model:
- prose + speeches: when a length/duration is given, hold it strictly with a
maximum ±5% deviation (the model systematically ran ~20-25% short).
- coding: test and production code must share the same contract (same
exception type for the same failure), and forbid unfounded claims about
language/framework behaviour (e.g. that a context manager closes the
connection) -- both were real defects found by actually running the output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add example_system_prompts/ (prose, speeches, coding) and matching
example_user_prompts/ (four user prompts per domain, one file each so they
run directly via --prompt-file), plus a README mapping the pairs and the run
command against the default model.
Also refine the three system prompts: separate narrative register from
character speech and add a length default (prose); add a duration→word-count
rule, spoken-language guidance, a no-invented-evidence rule, and a salutation
note (speeches); fix the mangled numbered list and add a language default,
a no-invented-APIs rule, and clearer output ordering (coding).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
