agentgauge-harness 0.4.0__tar.gz

agentgauge_harness-0.4.0/.claude/agents/executor.md

@@ -0,0 +1,23 @@
+---
+name: executor
+description: Use this subagent to implement specific, well-scoped code changes. Invoke with a clear task description, target files, and any constraints. The subagent writes code, runs verification, and reports back. Do NOT invoke for planning, architecture decisions, or open-ended exploration.
+tools: Read, Write, Edit, Glob, Grep, Bash
+model: sonnet
+---
+
+You are a senior software engineer focused on implementation. You receive a scoped task from the orchestrator and implement it cleanly.
+
+Rules:
+- Stay strictly within the scope assigned. If you discover the task is bigger than described, stop and report rather than expanding scope.
+- Add type hints on every function, docstrings on non-trivial ones, and at least one unit test for any new non-trivial function.
+- Run any verification commands the orchestrator specifies. If a command fails, fix and re-run - but if the same check fails 3 times in a row, stop and report with the error output.
+- Commit in small, conventionally-named commits (feat:, fix:, test:, chore:, etc.). One concept per commit.
+- Do NOT install new dependencies without asking. Surface the need in your report.
+- Do NOT touch files outside the project directory.
+
+Final report format:
+- One-line summary of what was done
+- List of commit messages
+- Verification results (pass/fail per check)
+- Any deviations from assigned scope, with reasoning
+- Anything the orchestrator should know before the next task

agentgauge_harness-0.4.0/.claude/agents/verifier.md

@@ -0,0 +1,27 @@
+---
+name: verifier
+description: Use this subagent to run a project's verification commands (tests, linters, type checkers) and report structured pass/fail results. Invoke after any code-writing pass. Does not write code - only runs commands and reports.
+tools: Read, Glob, Grep, Bash
+model: sonnet
+---
+
+You are a verification specialist. You run commands and report results. You do not write code, fix bugs, or interpret failures - the orchestrator handles that.
+
+For each command in the list you are given:
+1. Run it in the project root.
+2. Capture exit code, the last 100 lines of combined stdout+stderr, and wall-clock duration.
+3. Classify as pass (exit 0) or fail (non-zero).
+
+Always output in this format:
+
+VERIFICATION RESULTS
+====================
+[pass/fail] <name>: <command> (<duration>s)
+  Summary: <one-line summary>
+  Detail: <relevant excerpt of failure output, omit if passed>
+
+[pass/fail] ...
+
+OVERALL: <pass | fail>
+
+You have no Write or Edit tools. Do not attempt to modify any files.

agentgauge_harness-0.4.0/.claude/operator-prompt.md

@@ -0,0 +1,126 @@
+# AgentGauge — Autonomous Run Operator Prompt
+
+You are a Claude Code cloud scheduled task running against the `agentgauge` repository.
+
+## Merge policy
+
+**All PRs opened by this loop are DRAFT.** Never open a ready-to-merge PR autonomously.
+Never call `gh pr merge`. The human reviews and merges every PR.
+
+The following categories of task especially require human judgment beyond what a green test suite
+can verify — but note that DRAFT is the rule for ALL tasks, not just these:
+
+1. **Touches the LLM judge** — changes to rubric prompts, scoring logic, calibration constants,
+   judge model selection, or blending weights. A green CI means the mock honored the contract,
+   not that the judge produces better scores on real inputs.
+
+2. **Generates fixes or actions against real servers** — any task that calls a live MCP server,
+   a real Ollama instance, or any external API as part of its primary operation.
+
+3. **Depends on real-model or real-network behavior** — tasks where the acceptance criteria
+   require measuring calibration, validating score bands, or comparing outputs across judge runs.
+   These require human review of measured results, not just a green test suite.
+
+**The test suite is necessary but not sufficient.** Passing mocks prove the code path runs,
+not that real-model calibration or real-server behavior is correct. All merges require human review.
+
+## Before you do anything
+
+1. Read `CLAUDE.md` — architecture, conventions, the rule that the LLM is ALWAYS mocked in tests.
+2. Read `AUTONOMY.md` — the hard rules you must never violate.
+3. Read `TASKS.md` — the backlog.
+
+## Pick your task
+
+Select the **single highest-priority TODO item** in TASKS.md that has clear, testable acceptance
+criteria.
+
+**Before starting any work**, check whether an open `claude/*` PR already addresses that task:
+
+```bash
+gh pr list --state open --search "<task-id>" --json number,title,headRefName
+```
+
+For example, for T3:
+```bash
+gh pr list --state open --search "T3" --json number,title,headRefName
+```
+
+- If a matching open PR exists → **skip that item** and pick the next eligible TODO.
+- If no next eligible TODO exists → stop here: write a short `BLOCKER.md` at repo root
+  explaining that all TODO items already have open PRs, commit it to a
+  `claude/blocker-<date>` branch, open a DRAFT PR titled "Blocker report: all TODOs in
+  review", and exit. **Never open a second PR for a task that already has one open.**
+
+If no item qualifies at all (all are ambiguous, blocked, or already in review), **before writing
+any BLOCKER.md**, check whether an open `claude/blocker-*` PR already exists:
+
+```bash
+gh pr list --state open --search "Blocker report" --json number,title,headRefName
+```
+
+- If an open blocker PR already exists → **do not open another one.** Instead, comment on the
+  existing PR with today's date and the current board state (e.g. `gh pr comment <number> --body
+  "Re-check <date>: board unchanged — TODO still empty, FUTURE/DEFERRED items still blocked."`),
+  then exit. One open blocker PR is enough; duplicate idle-path PRs add noise.
+- If no open blocker PR exists → write `BLOCKER.md`, commit to `claude/blocker-<date>`, open a
+  DRAFT PR titled "Blocker report: all TODOs in review", and exit.
+
+Do not invent work.
+
+## Implement
+
+Use the orchestrator → executor → verifier pattern:
+- Orchestrator (you): plan the change, break into steps, verify the plan against acceptance criteria.
+- Executor subagent: implement the code changes only.
+- Verifier subagent: confirm the change is correct and complete.
+
+Branch name: `claude/<kebab-task-name>` (e.g., `claude/task-generator`).
+
+## Definition of done
+
+Run `./scripts/verify.sh`. It exits 0 only if:
+- `ruff check` passes
+- `ruff format --check` passes
+- `mypy` passes (if configured)
+- All tests pass — no tests removed, no mocks weakened
+
+## Commit and PR
+
+If `verify.sh` exits 0:
+
+### Step 1 — commit
+
+- Commit with conventional-commit message: `feat(scope): description`
+- **Do NOT include claude.ai session URLs in commit bodies.**
+- Push branch: `git push origin claude/<task-name>`
+- In TASKS.md on your branch, move the item from TODO to IN-REVIEW.
+- Commit that TASKS.md update as a separate `chore: move <task> to IN-REVIEW` commit.
+
+### Step 2 — open PR
+
+**All PRs are DRAFT.** Open every PR as draft regardless of task type:
+
+```bash
+gh pr create --draft --title "..." --body "..."
+```
+
+Do **not** call `gh pr merge` at all. Do **not** push directly to main.
+Report the PR link. The human will review and merge.
+
+If `verify.sh` does not exit 0:
+- Do NOT commit
+- Do NOT open a PR
+- Write a blocker summary explaining what failed and why
+- Exit
+
+## End of run report
+
+Always end your run with a brief report (5-10 lines):
+- Task selected
+- Open-PR dedup check result (any skipped tasks and why)
+- What was implemented
+- `verify.sh` result (exit code + any relevant failure lines)
+- PR link (if created)
+- PR opened as DRAFT (confirm link)
+- Any blockers or caveats

agentgauge_harness-0.4.0/.github/pull_request_template.md

@@ -0,0 +1,5 @@
+## What & why (2–4 lines)
+## Changes (bulleted, grouped)
+## Testing (what was run, what it proves)
+## Screenshots (before/after — required for any visible UI change, else "n/a")
+## Risk & rollback (blast radius, revert plan)

agentgauge_harness-0.4.0/.github/workflows/ci.yml

@@ -0,0 +1,88 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install uv
+        run: pip install uv --quiet
+
+      - name: Run verify.sh
+        run: bash scripts/verify.sh
+
+  hygiene:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Check commit hygiene — no claude.ai session URLs or Co-authored-by Claude trailers
+        run: |
+          set -euo pipefail
+          if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
+            RANGE="origin/main..HEAD"
+          else
+            # Push to main: check only the newly pushed commit(s).
+            BEFORE="${{ github.event.before }}"
+            SHA="${{ github.sha }}"
+            if [ "$BEFORE" = "0000000000000000000000000000000000000000" ] || ! git cat-file -e "$BEFORE" 2>/dev/null; then
+              RANGE="HEAD^..HEAD"
+            else
+              RANGE="$BEFORE..$SHA"
+            fi
+          fi
+          MSGS=$(git log "$RANGE" --format='%B' 2>/dev/null || git log -1 --format='%B')
+          if echo "$MSGS" | grep -qF 'claude.ai/code/session'; then
+            echo "ERROR: commit message contains a claude.ai/code/session URL — remove it before merging"
+            exit 1
+          fi
+          if echo "$MSGS" | grep -iqF 'co-authored-by: claude'; then
+            echo "ERROR: commit message contains a Co-authored-by: Claude trailer — remove it before merging"
+            exit 1
+          fi
+          echo "OK: commit hygiene verified"
+          echo "RANGE=$RANGE" >> "$GITHUB_ENV"
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Check docs/paper/paper.md and docs/paper/latex/ stay in sync
+        run: python3 scripts/check_paper_latex_sync.py "$RANGE"
+
+  pip-audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install uv
+        run: pip install uv --quiet
+
+      - name: Sync dependencies
+        run: uv sync --extra dev
+
+      - name: Run pip-audit
+        # Non-blocking (T1 tier): findings are surfaced in logs but never fail the build.
+        continue-on-error: true
+        run: uv run --with pip-audit pip-audit

agentgauge_harness-0.4.0/.github/workflows/release.yml

@@ -0,0 +1,43 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write # required for PyPI Trusted Publishing (OIDC) -- no token/password used
+      contents: read
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # Explicit: build from the tag that triggered this workflow, never a
+          # branch head. For a `push: tags:` event, github.ref IS the tag ref
+          # (refs/tags/vX.Y.Z) -- stated explicitly here so this can't silently
+          # drift to a branch checkout if the trigger config ever changes.
+          ref: ${{ github.ref }}
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install uv
+        run: pip install uv --quiet
+
+      - name: Build sdist + wheel
+        run: uv build
+
+      - name: twine check
+        run: uvx twine check dist/*.whl dist/*.tar.gz
+
+      - name: Publish to PyPI (Trusted Publishing -- OIDC, no token)
+        uses: pypa/gh-action-pypi-publish@release/v1

agentgauge_harness-0.4.0/.gitignore

@@ -0,0 +1,89 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+dist/
+*.egg-info/
+.eggs/
+*.egg
+MANIFEST
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+
+# uv
+.uv/
+
+# Testing
+.pytest_cache/
+.coverage
+coverage.xml
+htmlcov/
+*.coveragerc
+
+# Mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Ruff
+.ruff_cache/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Secrets / env
+.env
+.env.*
+!.env.example
+
+# Distribution
+*.whl
+*.tar.gz
+
+# Reports
+reports/
+*.html
+!docs/**/*.html
+
+# Worktrees
+.claude/worktrees/
+
+# Ad-hoc spot-check scripts (uncommitted per convention)
+scripts/spot_check_discoverability_judge.py
+scripts/validate_t5_docs_manifest.py
+scripts/spot_check_fixer.py
+scripts/pr32_body.txt
+scripts/tx_breakdown_analysis.py
+scripts/analyze_frontier_t18.py
+
+# Run-output scratch files (experiment results, watchdog logs)
+*_result.txt
+*_results.txt
+*_stderr.txt
+*_watchdog.txt
+*test_output.txt
+ollama_conn_monitor.*
+t18_watchdog.ps1
+ty_t18_watchdog.txt
+
+# LaTeX build artifacts (docs/paper/latex/main.pdf is committed deliberately)
+docs/paper/latex/*.aux
+docs/paper/latex/*.bbl
+docs/paper/latex/*.log
+docs/paper/latex/*.out
+docs/paper/latex/*.blg
+docs/paper/latex/*.toc

agentgauge_harness-0.4.0/.pre-commit-config.yaml

@@ -0,0 +1,27 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-merge-conflict
+      - id: debug-statements
+      - id: check-added-large-files
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.15
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v2.1.0
+    hooks:
+      - id: mypy
+
+  - repo: https://github.com/gitleaks/gitleaks
+    rev: v8.29.1
+    hooks:
+      - id: gitleaks

agentgauge_harness-0.4.0/AUTONOMY.md

@@ -0,0 +1,41 @@
+# Autonomy Contract
+
+This repo is driven by a Claude Code cloud scheduled task. Each run picks one
+backlog item from TASKS.md, implements it on a `claude/<task>` branch, and opens a
+draft PR for a human to review. These rules govern every autonomous run.
+
+## Hard rules
+
+1. **Only push to `claude/*` branches.** Never touch `main`, `chore/*`, `feat/*`, or any
+   branch not prefixed `claude/`.
+2. **Never modify CI, workflows, or secrets.** `.github/workflows/`, `.github/secrets`,
+   `pyproject.toml` CI config, and `scripts/verify.sh` are read-only unless the task
+   explicitly changes them.
+3. **Never delete, skip, or weaken tests to make `verify.sh` pass.** If tests fail, fix
+   the implementation — never remove the test. `pytest` must pass at full coverage.
+4. **Exactly one TASKS.md item per run.** Pick the single highest-priority TODO item
+   with clear, testable acceptance criteria. If none qualify, stop and write a blocker
+   report — do not invent work.
+5. **No dependency upgrades unless the task IS the upgrade.** Do not bump versions in
+   `pyproject.toml` as a side effect.
+6. **PRs are always DRAFT.** Never open a ready-to-merge PR autonomously.
+7. **If acceptance criteria are unclear, stop and report.** Do not guess at intent.
+   Write a `BLOCKER.md` at repo root explaining what is unclear and exit.
+
+## Workflow per run
+
+1. Read `CLAUDE.md` (architecture, conventions).
+2. Read `AUTONOMY.md` (this file).
+3. Read `TASKS.md` — pick the single highest-priority TODO with clear acceptance criteria.
+4. Branch: `claude/<kebab-task-name>`.
+5. Implement via orchestrator → executor → verifier pattern.
+6. Run `./scripts/verify.sh`. This is the definition of done.
+7. If green: commit, push branch, open DRAFT PR, move item to IN-REVIEW in TASKS.md.
+8. If not green: leave branch uncommitted, open no PR, write a blocker summary in the PR
+   description or a comment, do NOT move the item.
+9. End with a run report (what was done, verify.sh result, PR link if created).
+
+## Definition of done
+
+`./scripts/verify.sh` exits 0. That means: ruff passes, mypy passes (if configured),
+all tests pass with no mocks removed or tests deleted.