dschlueter
6f2f8aff6c
eval_prompt_tests.py measures the objective half of docs/EVAL_RUBRIC.md over the manual test archive: word count against the target stated in each prompt, truncation suspicion, and — for the coding domain — it writes the generated module and tests to a temp dir and actually runs pytest against them. Deriving the module's filename is the delicate part: a name taken from a test's `import sqlite3` would shadow the stdlib and fail the run for a reason the model is not responsible for. Names now come from the last *.py mention before the block, then from `from X import`, and anything in sys.stdlib_module_names is rejected. A module that no test imports is reported as such, since that is a finding about test quality rather than a guess the runner got wrong. run_prompt_suite.sh drives one prompt domain against a running profile and stores the outputs under the archive's naming convention. Both scripts join the ruff gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
41 lines
1.2 KiB
Bash
Executable file
41 lines
1.2 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
#
|
|
# Local CI gate: lint + type-check + tests. Mirrors .forgejo/workflows/ci.yml
|
|
# so the same checks run before a push even without a Forgejo Actions runner.
|
|
#
|
|
# Run manually: scripts/check.sh
|
|
# Run automatically: git config core.hooksPath .githooks (see .githooks/pre-push)
|
|
# Bypass once (emergency): git push --no-verify
|
|
#
|
|
set -uo pipefail
|
|
|
|
cd "$(git rev-parse --show-toplevel)" || exit 1
|
|
|
|
# Prefer the project venv's tools if present, else fall back to PATH.
|
|
BIN=""
|
|
if [ -x ".venv/bin/ruff" ]; then BIN=".venv/bin/"; fi
|
|
|
|
fail=0
|
|
run() {
|
|
local label=$1; shift
|
|
echo "== ${label} =="
|
|
if "$@"; then
|
|
echo " OK"
|
|
else
|
|
echo " FAILED"
|
|
fail=1
|
|
fi
|
|
}
|
|
|
|
run "ruff (lint)" "${BIN}ruff" check src/ tests/ build_archive.py scripts/eval_prompt_tests.py
|
|
run "mypy (types)" "${BIN}mypy"
|
|
# Use `python -m pytest` (not the pytest console script) so the repo root is on
|
|
# sys.path — the test modules import `from tests.test_docker_ops import ...`.
|
|
run "pytest (tests)" "${BIN}python" -m pytest -q
|
|
|
|
echo
|
|
if [ "$fail" -ne 0 ]; then
|
|
echo "CI gate FAILED — fix the above or push with --no-verify to bypass."
|
|
exit 1
|
|
fi
|
|
echo "CI gate passed."
|