Multi-model evaluation runs surfaced recurring coding defects; tighten the
coding system prompt to target them directly:
- Every emitted test must actually pass against the emitted code (mentally
run each test before output) -- several models shipped tests that fail.
- Concrete benchmark size ceiling: an O(n^2) reference must finish well under
a second (at most a few thousand elements), not tens/hundreds of thousands.
- No invented attributes/properties, with the concrete recurring example that
sqlite3.Connection has no `.closed` attribute (crashed two models).
- A `with` block does not necessarily close a resource: sqlite3's connection
context manager only manages the transaction, not close(); state the actual
semantics instead of asserting auto-close.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two more correctness guards, prompted by verified defects in evaluation runs:
- Test fixtures/inputs must actually satisfy the preconditions the same test
assumes (e.g. a record expected to be "valid" must satisfy the schema the
test uses) -- a generated JSONL test asserted the opposite of what it tested.
- Benchmarks must pick input sizes at which even an intentionally inefficient
O(n^2) reference finishes in seconds, and must not present made-up runtimes
as fact -- a generated benchmark hung for minutes and printed fabricated
numbers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Based on evaluation runs against the default model:
- prose + speeches: when a length/duration is given, hold it strictly with a
maximum ±5% deviation (the model systematically ran ~20-25% short).
- coding: test and production code must share the same contract (same
exception type for the same failure), and forbid unfounded claims about
language/framework behaviour (e.g. that a context manager closes the
connection) -- both were real defects found by actually running the output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add example_system_prompts/ (prose, speeches, coding) and matching
example_user_prompts/ (four user prompts per domain, one file each so they
run directly via --prompt-file), plus a README mapping the pairs and the run
command against the default model.
Also refine the three system prompts: separate narrative register from
character speech and add a length default (prose); add a duration→word-count
rule, spoken-language guidance, a no-invented-evidence rule, and a salutation
note (speeches); fix the mangled numbered list and add a language default,
a no-invented-APIs rule, and clearer output ordering (coding).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
