slmeval-cli 0.1.0__tar.gz

slmeval_cli-0.1.0/.gitattributes

@@ -0,0 +1,15 @@
+# Line endings are a property of this repository, not of each contributor's
+# git config. The harness hashes generated files to decide whether two runs are
+# comparable, so a CRLF committed from Windows must not read as a different
+# dataset in CI on Linux.
+* text=auto eol=lf
+
+# Golden datasets are compared byte-for-byte; never let git touch them.
+*.jsonl text eol=lf
+
+# Rendered goldens, compared byte-for-byte against what the tool prints.
+tests/snapshots/*.txt text eol=lf
+
+# Published teardown artifacts, diffed against a fresh run by
+# `scripts/teardown.py --check`, which CI runs on Windows and Linux.
+teardown/artifacts/* text eol=lf

slmeval_cli-0.1.0/.github/ISSUE_TEMPLATE/benchmark_result.yml

@@ -0,0 +1,38 @@
+name: Benchmark result
+description: Share results from running slmeval on real models.
+labels: [benchmark-result]
+body:
+  - type: markdown
+    attributes:
+      value: Please remove anything private first. Cassettes and traces can contain real data.
+  - type: input
+    id: models
+    attributes:
+      label: Models and how they were served
+      placeholder: gemma3:4b on Ollama, baseline gpt-4o-mini
+    validations:
+      required: true
+  - type: textarea
+    id: report
+    attributes:
+      label: Output of `slmeval report --format md`
+    validations:
+      required: true
+  - type: textarea
+    id: config
+    attributes:
+      label: Your eval.toml
+      render: toml
+    validations:
+      required: true
+  - type: input
+    id: dataset
+    attributes:
+      label: Dataset
+      description: A link, or how many cases and what kind if it can't be shared.
+    validations:
+      required: true
+  - type: textarea
+    id: hardware
+    attributes:
+      label: Hardware, if you measured latency

slmeval_cli-0.1.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,37 @@
+name: Bug report
+description: Something doesn't work as documented.
+labels: [bug]
+body:
+  - type: textarea
+    id: what
+    attributes:
+      label: What happened?
+      description: Include what you expected and the full output, with any private data removed.
+    validations:
+      required: true
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: How can we reproduce it?
+      description: The commands you ran and the relevant parts of eval.toml.
+    validations:
+      required: true
+  - type: textarea
+    id: doctor
+    attributes:
+      label: Output of `slmeval doctor`
+      render: text
+    validations:
+      required: true
+  - type: dropdown
+    id: exit-code
+    attributes:
+      label: Exit code
+      options:
+        - "0"
+        - "1"
+        - "2"
+        - "3"
+        - "4"
+        - Other, or a traceback
+        - Not applicable

slmeval_cli-0.1.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Security vulnerability
+    url: https://github.com/mohitraj3697/slmeval/security/advisories/new
+    about: Please report these privately.

slmeval_cli-0.1.0/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,19 @@
+name: Feature request
+description: Suggest a change or a new capability.
+labels: [enhancement]
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: What are you trying to do?
+      description: Describe the problem before the solution.
+    validations:
+      required: true
+  - type: textarea
+    id: proposal
+    attributes:
+      label: What would you like to see?
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: What have you tried instead?

slmeval_cli-0.1.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,14 @@
+## What and why
+
+<!-- What changes, and what problem it solves. Link any related issue. -->
+
+## Testing
+
+<!-- How you checked it. Put new failure modes under the `fake` provider. -->
+
+## Checklist
+
+- [ ] `ruff`, `mypy` and `pytest` pass
+- [ ] Docs are updated if a command or flag changed
+- [ ] Snapshots, teardown artifacts and README images are regenerated if the output changed
+- [ ] The changelog mentions any change to exit codes or the report `schema`

slmeval_cli-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,163 @@
+#  The offline suite, on both platforms it has to be identical on.
+#
+#  Determinism bugs in a harness like this are nearly all platform bugs: a hash
+#  that iterates a dict, a path separator inside a cassette key, a CRLF
+#  checkout, a console that cannot encode a box-drawing character. None of them
+#  reproduce on the machine they were written on, which is why the matrix is
+#  Windows *and* Linux rather than one of them.
+#
+#  Nothing here needs a GPU, a model server or an API key. If a job in this
+#  file ever needs one, the test that made it necessary belongs behind
+#  `-m live` instead.
+
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  PYTHONUTF8: "1"
+  # The developer-loop target is under 30 seconds; a cold CI runner is slower
+  # than a laptop, so the gate here only has to catch a regression in kind -
+  # someone reintroducing a real sleep, a retry with a real backoff, or a test
+  # that opens a socket.
+  SUITE_BUDGET_S: "120"
+
+jobs:
+  offline:
+    name: offline suite (${{ matrix.os }}, py${{ matrix.python }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        python: ["3.11", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+          cache: pip
+          cache-dependency-path: pyproject.toml
+
+      - name: Install
+        run: python -m pip install --upgrade pip && python -m pip install -e ".[dev]"
+
+      - name: Lint
+        run: python -m ruff check src tests scripts
+
+      - name: Type-check
+        run: python -m mypy src
+
+      - name: Offline suite, within its time budget
+        shell: bash
+        run: |
+          start=$(date +%s)
+          python -m pytest --durations=10
+          elapsed=$(( $(date +%s) - start ))
+          echo "offline suite: ${elapsed}s (budget ${SUITE_BUDGET_S}s)"
+          if [ "$elapsed" -gt "$SUITE_BUDGET_S" ]; then
+            echo "::error::the offline suite took ${elapsed}s, over the ${SUITE_BUDGET_S}s budget."
+            echo "A suite people stop running protects nothing - see the slowest tests above."
+            exit 1
+          fi
+
+  # The one job that installs a model framework. The core must stay installable
+  # in seconds without one, so `langgraph` is an extra and the adapter's tests
+  # skip in every other job here - which means that without this job they would
+  # never run anywhere, and a skipped test reads as a passing one.
+  langgraph:
+    name: langgraph adapter
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+          cache-dependency-path: pyproject.toml
+
+      - name: Install, with the extra
+        run: python -m pip install --upgrade pip && python -m pip install -e ".[dev,langgraph]"
+
+      - name: The adapter, and the example under both adapters
+        run: python -m pytest tests/test_langgraph_adapter.py -v
+
+      - name: Nothing was skipped
+        shell: bash
+        run: |
+          # A green job that skipped the only two tests it exists to run would
+          # be worse than no job: it would read as coverage we do not have.
+          python -m pytest tests/test_langgraph_adapter.py -q -rs 2>&1 | tee out.txt
+          if grep -q "skipped" out.txt; then
+            echo "::error::a langgraph test skipped in the job that installs langgraph."
+            exit 1
+          fi
+
+  # The published teardown, re-run. `tests/test_teardown.py` checks that the
+  # write-up still agrees with the artifacts next to it; only re-running the
+  # benchmark can check that the artifacts still agree with the tool. Without
+  # this, the first change that moved a number would leave a document on the
+  # front page of the repository quoting figures nothing produces any more.
+  #
+  # Both platforms, because the artifacts are committed from one of them and a
+  # path separator or a console codepage leaking into a published file is
+  # exactly the class of bug this repository's CI matrix exists to catch.
+  teardown:
+    name: the published teardown still reproduces (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+          cache-dependency-path: pyproject.toml
+
+      - name: Install
+        run: python -m pip install --upgrade pip && python -m pip install -e .
+
+      - name: Record, run, and diff against what is published
+        run: python scripts/teardown.py --check
+
+  install-smoke:
+    name: install smoke (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      # `pytest` runs against the working tree and so proves nothing about what
+      # `pip install slmeval-cli` ships. This builds the wheel, installs it into a
+      # throwaway venv with nothing else in it, and runs init / validate /
+      # doctor / record / run / report as a new user would.
+      - name: Build and use the wheel
+        run: python scripts/install_smoke.py
+
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: wheel-${{ matrix.os }}
+          path: dist/*.whl
+          if-no-files-found: error

slmeval_cli-0.1.0/.github/workflows/gate-demo.yml

@@ -0,0 +1,107 @@
+# The GitHub Action, proved on the bundled example.
+#
+# This is the "demo repo" check: it exercises `examples/github-action` end to
+# end against `examples/support_agent`, on a stock runner, with **no GPU, no
+# API key and no secret of any kind**. The baseline and both candidates are the
+# scripted `fake` provider and every tool is a local Python function, so the
+# whole thing - record, replay, gate - runs offline.
+#
+# Two jobs, because an action that can only go green is not a gate:
+#
+# `green` gates on a floor the example clears and expects exit 0.
+# `red` gates on a floor nothing can clear and expects exit 1 - a regression,
+# not a crash, not a usage error, and not exit 3. That distinction is the whole
+# contract CI branches on, and it is the one thing worth asserting on every
+# push.
+#
+# The job summary and the pull-request comment this produces are also the
+# artefacts to screenshot for the README: this workflow is where the output in
+# the docs comes from.
+
+name: gate-demo
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pull-requests: write
+
+concurrency:
+  group: gate-demo-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  green:
+    name: the action passes a healthy run
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - id: slmeval
+        uses: ./examples/github-action
+        with:
+          working-directory: examples/support_agent
+          # The checkout itself, not a release: this workflow is what proves
+          # the action works against the code in the pull request.
+          version: .
+          # Safe here, and only here: every tool in the example is a local
+          # function and the provider is `fake`, so recording touches nothing
+          # outside the runner.
+          record: missing
+          # The candidate whose story is "passes, but only with help": it
+          # answers every case correctly and needs a lower rung on every
+          # structured call to do it.
+          run-args: --candidate fenced-json
+          # A floor the example clears, on the estimator that matters, and not
+          # a number tuned to pass: 4/4 cases over 3 runs each has a Wilson
+          # lower bound of 51%, so this is as high as a floor can honestly go
+          # at n=4. Asking for 0.9 here would be asking for a guarantee the
+          # sample size cannot give - which is the whole lesson of the tool.
+          gate-args: --min-pass-k 0.5 --use-wilson-lower
+          artifact-name: slmeval-demo-green
+
+      - name: The outputs say what happened
+        run: |
+          test '${{ steps.slmeval.outputs.verdict }}' = passed
+          test '${{ steps.slmeval.outputs.exit-code }}' = 0
+          test -n '${{ steps.slmeval.outputs.run-id }}'
+          test -n '${{ steps.slmeval.outputs.dataset-hash }}'
+          echo "run ${{ steps.slmeval.outputs.run-id }} passed"
+
+  red:
+    name: the action fails a regression with exit 1
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - id: slmeval
+        # continue-on-error, not a removed threshold: the action *must* fail
+        # the step here. A demo that only ever went green would prove the
+        # pretty half of a gate and none of the useful half.
+        continue-on-error: true
+        uses: ./examples/github-action
+        with:
+          working-directory: examples/support_agent
+          version: .
+          record: missing
+          # Nothing can clear a floor of 100% on a Wilson lower bound, so this
+          # is the regression path.
+          gate-args: --min-pass-k 1.0 --use-wilson-lower
+          comment: "false"
+          artifact-name: slmeval-demo-red
+
+      - name: Exit 1, and labelled as a regression
+        run: |
+          if [ '${{ steps.slmeval.outputs.exit-code }}' != '1' ]; then
+            echo "::error::the gate exited '${{ steps.slmeval.outputs.exit-code }}', not 1."
+            echo "1 means a threshold was missed. 2 is a usage error, 3 is an unreachable"
+            echo "provider and 4 is thin coverage - if an unclearable floor produces any of"
+            echo "those, CI cannot tell a regression from a broken runner."
+            exit 1
+          fi
+          test '${{ steps.slmeval.outputs.verdict }}' = failed
+          echo "a missed threshold exits 1, as the contract says"

slmeval_cli-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,114 @@
+# Publish to PyPI on a tag, with Trusted Publishing.
+#
+# No long-lived token in this repository's secrets. PyPI mints a short-lived
+# credential for this exact workflow in this exact repository, via OIDC, which
+# is why the publish job asks for `id-token: write` and has no `password:`. A
+# token in secrets is a credential that can be exfiltrated by any workflow that
+# can read secrets, and for a measurement tool people are asked to install into
+# their CI, that is the wrong trade.
+#
+# It also refuses to publish a tag that disagrees with the package: a `v0.1.1`
+# tag on a tree whose `__version__` still says `0.1.0` would put a wheel on
+# PyPI that no git tag describes.
+#
+# Setup, once, before the first tag: on PyPI, add a trusted publisher for this
+# repository with workflow `release.yml` and environment `pypi`.
+#
+#     git tag v0.1.0 && git push origin v0.1.0
+
+name: release
+
+on:
+  push:
+    tags: ["v*"]
+  workflow_dispatch:
+    inputs:
+      dry-run:
+        description: Build and check, but do not publish.
+        type: boolean
+        default: true
+
+permissions:
+  contents: read
+
+jobs:
+  # A tag is not a release until the thing it points at actually works. The
+  # offline suite and the install smoke both run again here rather than
+  # trusting that they ran on the commit: a tag can point anywhere.
+  verify:
+    name: verify the tagged tree (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: true
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install
+        run: python -m pip install --upgrade pip && python -m pip install -e ".[dev]"
+
+      - name: The tag and the package agree
+        if: startsWith(github.ref, 'refs/tags/v')
+        shell: bash
+        run: |
+          set -euo pipefail
+          tagged="${GITHUB_REF_NAME#v}"
+          packaged=$(python -c "import slmeval; print(slmeval.__version__)")
+          echo "tag $tagged, package $packaged"
+          if [ "$tagged" != "$packaged" ]; then
+            echo "::error::tag $GITHUB_REF_NAME does not match __version__ $packaged."
+            echo "Publishing would put a wheel on PyPI that no git tag describes."
+            exit 1
+          fi
+
+      - name: Offline suite
+        run: python -m pytest
+
+      - name: The wheel installs and produces a scorecard
+        run: python scripts/install_smoke.py
+
+  build:
+    name: build the distributions
+    needs: verify
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Build
+        run: |
+          python -m pip install --upgrade pip build twine
+          python -m build
+          python -m twine check --strict dist/*
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: distributions
+          path: dist/*
+          if-no-files-found: error
+
+  publish:
+    name: publish to PyPI
+    needs: build
+    if: startsWith(github.ref, 'refs/tags/v') && !inputs.dry-run
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      # The whole point: PyPI mints a short-lived credential for this workflow
+      # over OIDC, so there is no token in this repository to leak.
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: distributions
+          path: dist
+
+      - uses: pypa/gh-action-pypi-publish@release/v1

slmeval_cli-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+.slmeval/
+.venv/
+venv/
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.env

slmeval_cli-0.1.0/CHANGELOG.md

@@ -0,0 +1,59 @@
+# Changelog
+
+Versions follow [semantic versioning](https://semver.org). While the major version is 0, a
+minor release can change the CLI and the stored schema.
+
+The exit codes of `slmeval gate` and the `schema` field of the JSON report are public
+contracts. Any change to them is listed here as breaking.
+
+## [0.1.0] - 2026-11-04
+
+First release, published on PyPI as `slmeval-cli`. The command and the Python import are
+both `slmeval`.
+
+### Harness
+
+- `slmeval.node()` and `slmeval.chat_model()` mark steps and route model calls per candidate.
+  Both do nothing outside a benchmark.
+- `slmeval.instrument_langgraph()` marks every node of a LangGraph builder. Requires the
+  `langgraph` extra.
+- `@sandbox_tool` records a tool call once and replays it for every candidate. A miss gets a
+  verdict and a diff of the arguments against the nearest recording.
+- Recorded responses keep their types, including `Decimal`, `datetime` and enums. A value that
+  can't be stored faithfully raises an error at record time.
+- Seven kinds of fault injection, scheduled per case and seed. Each run records the faults it met.
+- Providers for OpenAI, Ollama, anything serving `/v1/chat/completions`, and a scripted `fake`.
+  Reports show which structured-output mechanism each call needed.
+
+### Measurement
+
+- Dataset assertions, including forbidden tools as a hard fail.
+- Path distance, tool overlap with the baseline, loop detection and recovery from faults.
+- `pass@k`, `pass^k` and Wilson intervals. The gate reads the lower bound by default.
+- Cost per case. Local inference is reported as modelled from token counts and `--gpu-hourly`.
+- Latency only in `--serial-latency` mode.
+- Bugs in slmeval, unrecorded paths and unreachable providers stay out of pass rates and are
+  reported separately.
+- An LLM judge with a calibration harness, off by default. It isn't connected to `slmeval run` yet.
+
+### CLI
+
+`init`, `doctor`, `validate`, `record`, `run`, `report`, `show run|exec|case|uncovered`, `gate`,
+`baseline save|show`, `judge calibrate`, `dataset validate` and `cache stats|prune`.
+
+Reports render as a table, JSON, Markdown, CSV or JUnit XML.
+
+Exit codes: 0 pass, 1 gate failed, 2 usage, 3 environment, 4 coverage.
+
+### Also included
+
+- A [GitHub Action](examples/github-action) that runs the benchmark, writes the job summary and
+  keeps one pull request comment updated.
+- A [runnable example](examples/support_agent) and a [judge calibration example](examples/judge).
+- [Teardown #1](teardown), a published run with scripted models. CI re-runs it on Linux and Windows.
+
+### Not in this release
+
+- `slmeval compare` and `slmeval cassette`. They are hidden from `--help` and exit 2 if called.
+- A native Anthropic provider. Reach Claude through an OpenAI-compatible gateway.
+- Time-to-first-token measurement.

slmeval_cli-0.1.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,7 @@
+# Code of Conduct
+
+This project follows the
+[Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
+
+To report unacceptable behaviour, contact the maintainer privately through
+[GitHub](https://github.com/mohitraj3697). Reports are kept confidential.