classify_refusal() separates refusal (head), preamble, appended disclaimer
(tail) and moralising insert (inline). Only the first three are meaningful, and
only where a prompt forbids them: a preamble on a coding answer is normal, so it
no longer counts as evasion.
Two sources of false positives had to be removed first, both found by checking
the detector's hits against the archive rather than trusting them:
* llamacppctl's own truncation warning on stderr had leaked into an archived
output and was read as a model disclaimer. It also inflated that run's word
count, so count_words() strips it too.
* An AI character saying "Bitte beachten Sie:" inside a dystopian story is
plot, not distancing. Quoted speech is removed before markers are matched.
run_prompt_suite.sh gains a CASES filter so a single prompt can be re-run.
Measured result, recorded in KI_TOOLS_PROFILES.md: across seven prose prompts
neither the abliterated model nor the aligned base model evaded once. On
literary prose the abliteration buys nothing measurable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eval_prompt_tests.py measures the objective half of docs/EVAL_RUBRIC.md over
the manual test archive: word count against the target stated in each prompt,
truncation suspicion, and — for the coding domain — it writes the generated
module and tests to a temp dir and actually runs pytest against them.
Deriving the module's filename is the delicate part: a name taken from a test's
`import sqlite3` would shadow the stdlib and fail the run for a reason the model
is not responsible for. Names now come from the last *.py mention before the
block, then from `from X import`, and anything in sys.stdlib_module_names is
rejected. A module that no test imports is reported as such, since that is a
finding about test quality rather than a guess the runner got wrong.
run_prompt_suite.sh drives one prompt domain against a running profile and
stores the outputs under the archive's naming convention. Both scripts join the
ruff gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_archive.py had no lint or test coverage, which is how the obsolete
requirements/pyproject mirror check slipped through unnoticed. Close that gap:
- Add tests/test_build_archive.py: a network-free smoke test that runs the
manifest check, dependency parsing, tarball build, and re-verification, and
asserts the moved docs/ files, LICENSE, and package sources are packaged.
- Lint build_archive.py in both the local gate (scripts/check.sh) and the CI
workflow; fix the one issue this surfaced (unused variable py_bin).
- List the new test in the REQUIRED_FILES manifest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forgejo Actions is not enabled on the instance, so run the same checks
locally before pushing:
- scripts/check.sh runs ruff, mypy, and pytest (mirrors the CI workflow).
- .githooks/pre-push invokes it; enable per clone with
`git config core.hooksPath .githooks`. Bypass with `git push --no-verify`.
- Fix the pytest invocation in the CI workflow (and document it): use
`python -m pytest` so the repo root is on sys.path, otherwise the test
modules fail to `import tests.*`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
llama.cpp exempts /health and /v1/models from --api-key; only
/v1/chat/completions is protected. The smoke test probed /v1/models and thus
saw 200 without a key. Probe /v1/chat/completions instead (no key -> 401,
key -> 200). Verified live end-to-end: 9 PASS, 0 FAIL.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
