eval_prompt_tests.py measures the objective half of docs/EVAL_RUBRIC.md over
the manual test archive: word count against the target stated in each prompt,
truncation suspicion, and — for the coding domain — it writes the generated
module and tests to a temp dir and actually runs pytest against them.
Deriving the module's filename is the delicate part: a name taken from a test's
`import sqlite3` would shadow the stdlib and fail the run for a reason the model
is not responsible for. Names now come from the last *.py mention before the
block, then from `from X import`, and anything in sys.stdlib_module_names is
rejected. A module that no test imports is reported as such, since that is a
finding about test quality rather than a guess the runner got wrong.
run_prompt_suite.sh drives one prompt domain against a running profile and
stores the outputs under the archive's naming convention. Both scripts join the
ruff gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_archive.py had no lint or test coverage, which is how the obsolete
requirements/pyproject mirror check slipped through unnoticed. Close that gap:
- Add tests/test_build_archive.py: a network-free smoke test that runs the
manifest check, dependency parsing, tarball build, and re-verification, and
asserts the moved docs/ files, LICENSE, and package sources are packaged.
- Lint build_archive.py in both the local gate (scripts/check.sh) and the CI
workflow; fix the one issue this surfaced (unused variable py_bin).
- List the new test in the REQUIRED_FILES manifest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forgejo Actions is not enabled on the instance, so run the same checks
locally before pushing:
- scripts/check.sh runs ruff, mypy, and pytest (mirrors the CI workflow).
- .githooks/pre-push invokes it; enable per clone with
`git config core.hooksPath .githooks`. Bypass with `git push --no-verify`.
- Fix the pytest invocation in the CI workflow (and document it): use
`python -m pytest` so the repo root is on sys.path, otherwise the test
modules fail to `import tests.*`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
llama.cpp exempts /health and /v1/models from --api-key; only
/v1/chat/completions is protected. The smoke test probed /v1/models and thus
saw 200 without a key. Probe /v1/chat/completions instead (no key -> 401,
key -> 200). Verified live end-to-end: 9 PASS, 0 FAIL.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
