Multi-model evaluation runs surfaced recurring coding defects; tighten the
coding system prompt to target them directly:
- Every emitted test must actually pass against the emitted code (mentally
run each test before output) -- several models shipped tests that fail.
- Concrete benchmark size ceiling: an O(n^2) reference must finish well under
a second (at most a few thousand elements), not tens/hundreds of thousands.
- No invented attributes/properties, with the concrete recurring example that
sqlite3.Connection has no `.closed` attribute (crashed two models).
- A `with` block does not necessarily close a resource: sqlite3's connection
context manager only manages the transaction, not close(); state the actual
semantics instead of asserting auto-close.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dschlueter
e2297f43e4