classify_refusal() separates refusal (head), preamble, appended disclaimer
(tail) and moralising insert (inline). Only the first three are meaningful, and
only where a prompt forbids them: a preamble on a coding answer is normal, so it
no longer counts as evasion.
Two sources of false positives had to be removed first, both found by checking
the detector's hits against the archive rather than trusting them:
* llamacppctl's own truncation warning on stderr had leaked into an archived
output and was read as a model disclaimer. It also inflated that run's word
count, so count_words() strips it too.
* An AI character saying "Bitte beachten Sie:" inside a dystopian story is
plot, not distancing. Quoted speech is removed before markers are matched.
run_prompt_suite.sh gains a CASES filter so a single prompt can be re-run.
Measured result, recorded in KI_TOOLS_PROFILES.md: across seven prose prompts
neither the abliterated model nor the aligned base model evaded once. On
literary prose the abliteration buys nothing measurable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eval_prompt_tests.py measures the objective half of docs/EVAL_RUBRIC.md over
the manual test archive: word count against the target stated in each prompt,
truncation suspicion, and — for the coding domain — it writes the generated
module and tests to a temp dir and actually runs pytest against them.
Deriving the module's filename is the delicate part: a name taken from a test's
`import sqlite3` would shadow the stdlib and fail the run for a reason the model
is not responsible for. Names now come from the last *.py mention before the
block, then from `from X import`, and anything in sys.stdlib_module_names is
rejected. A module that no test imports is reported as such, since that is a
finding about test quality rather than a guess the runner got wrong.
run_prompt_suite.sh drives one prompt domain against a running profile and
stores the outputs under the archive's naming convention. Both scripts join the
ruff gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
