eval_prompt_tests.py measures the objective half of docs/EVAL_RUBRIC.md over
the manual test archive: word count against the target stated in each prompt,
truncation suspicion, and — for the coding domain — it writes the generated
module and tests to a temp dir and actually runs pytest against them.
Deriving the module's filename is the delicate part: a name taken from a test's
`import sqlite3` would shadow the stdlib and fail the run for a reason the model
is not responsible for. Names now come from the last *.py mention before the
block, then from `from X import`, and anything in sys.stdlib_module_names is
rejected. A module that no test imports is reported as such, since that is a
finding about test quality rather than a guess the runner got wrong.
run_prompt_suite.sh drives one prompt domain against a running profile and
stores the outputs under the archive's naming convention. Both scripts join the
ruff gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_archive.py had no lint or test coverage, which is how the obsolete
requirements/pyproject mirror check slipped through unnoticed. Close that gap:
- Add tests/test_build_archive.py: a network-free smoke test that runs the
manifest check, dependency parsing, tarball build, and re-verification, and
asserts the moved docs/ files, LICENSE, and package sources are packaged.
- Lint build_archive.py in both the local gate (scripts/check.sh) and the CI
workflow; fix the one issue this surfaced (unused variable py_bin).
- List the new test in the REQUIRED_FILES manifest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forgejo Actions is not enabled on the instance, so run the same checks
locally before pushing:
- scripts/check.sh runs ruff, mypy, and pytest (mirrors the CI workflow).
- .githooks/pre-push invokes it; enable per clone with
`git config core.hooksPath .githooks`. Bypass with `git push --no-verify`.
- Fix the pytest invocation in the CI workflow (and document it): use
`python -m pytest` so the repo root is on sys.path, otherwise the test
modules fail to `import tests.*`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Back the shipped py.typed promise with an enforced type check and a linter:
- Add ruff + mypy (+ types-requests) to the dev extras and dev requirements,
with [tool.ruff]/[tool.mypy] config in pyproject.toml (mypy checks the
package, not the tests).
- Add a lint job to the Forgejo workflow running ruff check + mypy.
- Fix the issues this surfaced: type FileLock.fd as TextIO | None, add a
targeted type: ignore for the intentional socket.getaddrinfo monkeypatch,
and drop an unused import.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Public-repo hygiene now that the project is hosted on Forgejo:
- LICENSE: add the MIT text that pyproject.toml already declares.
- .forgejo/workflows/ci.yml: run the pytest suite on push/PR against
Python 3.10 and 3.12, so regressions (e.g. the healthcheck port bug)
get caught automatically.
- Untrack llama.cpp.config and gitignore it: it is a machine-specific
runtime config that may later hold an api_key. The tracked template
remains llama.cpp.config.example (copy it to get started).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
