dschlueter
6f2f8aff6c
eval_prompt_tests.py measures the objective half of docs/EVAL_RUBRIC.md over the manual test archive: word count against the target stated in each prompt, truncation suspicion, and — for the coding domain — it writes the generated module and tests to a temp dir and actually runs pytest against them. Deriving the module's filename is the delicate part: a name taken from a test's `import sqlite3` would shadow the stdlib and fail the run for a reason the model is not responsible for. Names now come from the last *.py mention before the block, then from `from X import`, and anything in sys.stdlib_module_names is rejected. A module that no test imports is reported as such, since that is a finding about test quality rather than a guess the runner got wrong. run_prompt_suite.sh drives one prompt domain against a running profile and stores the outputs under the archive's naming convention. Both scripts join the ruff gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
47 lines
1.1 KiB
YAML
47 lines
1.1 KiB
YAML
name: CI
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
|
|
jobs:
|
|
lint:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.12"
|
|
|
|
- name: Install package with dev extras
|
|
run: pip install -e ".[dev]"
|
|
|
|
- name: Ruff (lint)
|
|
run: ruff check src/ tests/ build_archive.py scripts/eval_prompt_tests.py
|
|
|
|
- name: Mypy (type check)
|
|
run: mypy
|
|
|
|
test:
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
# requires-python = ">=3.10"; test the floor and a recent version.
|
|
python-version: ["3.10", "3.12"]
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ matrix.python-version }}
|
|
|
|
- name: Install package with dev extras
|
|
run: pip install -e ".[dev]"
|
|
|
|
- name: Run test suite
|
|
# `python -m pytest` puts the repo root on sys.path so the test modules
|
|
# can `import tests.*` (the pytest console script would not).
|
|
run: python -m pytest -q
|