callprobe 0.5.0__tar.gz

callprobe-0.5.0/.github/ISSUE_TEMPLATE/new-model-run.md

@@ -0,0 +1,32 @@
+---
+name: New model run
+about: Report a run against a model not yet on the leaderboard
+title: "[run] "
+labels: run
+---
+
+**Model**
+Name and tag, e.g. `qwen2.5:7b`. Quantization if relevant.
+
+**Endpoint**
+Ollama, LM Studio, llama.cpp server, vLLM, or a hosted provider, and the
+`--endpoint` value used.
+
+**Command**
+The exact `callprobe run` invocation. The recommended flags are:
+
+```bash
+callprobe run --model your-model --pad 0,8,16,24 --repeats 3 \
+  --max-tokens 4096 --out results/your-model.json
+```
+
+If you used different flags, say why (context window too small for
+`--max-tokens 4096`, fewer repeats to keep the run short, etc).
+
+**Results file**
+Attach or link the JSON file written by `--out`. This is what a
+maintainer will run `callprobe leaderboard` on to add the row.
+
+**Anything surprising**
+Truncation, errors, a category that fell apart, anything in the failure
+digest worth flagging before it becomes a leaderboard row.

callprobe-0.5.0/.github/ISSUE_TEMPLATE/new-task.md

@@ -0,0 +1,25 @@
+---
+name: New task
+about: Propose a new task for the core suite
+title: "[task] "
+labels: task
+---
+
+**Category**
+Which of `select`, `abstain`, `args`, `depth`, `sequence`, and why that one.
+
+**Bundle**
+Which existing bundle (`support`, `scheduling`, `files`), or a new one.
+If new, what tools does it need?
+
+**The scenario**
+The conversation (or a sketch of it) and what a model should do with it.
+
+**Why this is worth testing**
+What specifically would this catch that the existing tasks don't? A near
+miss, an argument that's easy to get subtly wrong, a distractor that
+could invalidate an abstain task, etc.
+
+**`callprobe validate` output**
+Paste the output if you already have a draft task written. If not, that's
+fine, just flag that it still needs writing and validating.

callprobe-0.5.0/.github/workflows/ci.yml

@@ -0,0 +1,38 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: pytest -q
+
+  wheel-install:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - run: python -m build --wheel
+      - run: python -m venv /tmp/callprobe-wheel-env
+      - run: /tmp/callprobe-wheel-env/bin/pip install dist/*.whl
+      - name: run from outside the checkout
+        working-directory: /tmp
+        run: |
+          /tmp/callprobe-wheel-env/bin/callprobe --version
+          /tmp/callprobe-wheel-env/bin/callprobe validate

callprobe-0.5.0/.github/workflows/release.yml

@@ -0,0 +1,34 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - run: python -m build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

callprobe-0.5.0/.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.pyc
+.venv/
+*.egg-info/
+dist/
+build/
+overnight.log
+logs/

callprobe-0.5.0/CHANGELOG.md

@@ -0,0 +1,48 @@
+# Changelog
+
+## 0.5.0
+
+- Scoring version 2 requires exactly one call for call expectations and
+  rejects truncated responses, including apparent abstentions. Lenient
+  coercion cannot rescue either failure.
+- Preserve all structured calls, IDs, raw arguments, parse errors, and
+  finish reasons in result files.
+- Reject incompatible or missing resume files; preserve the original start
+  time, retry request errors, and write checkpoints atomically.
+- Add `compare --fail-on-regression`, YAML `--policy`, and JSON comparison
+  output. Gates require complete matched coverage and compatible suite and
+  scoring provenance. Policies cover critical tasks, overall/category
+  minimums, maximum success drops, and request error limits.
+- Fail `run --fail-under` on request errors and incomplete runs. Reject
+  mixed scoring versions in leaderboards by default.
+- Add GitHub Action baseline/policy and run-configuration inputs; install
+  the selected action revision instead of an unrelated PyPI version.
+- Include two fresh 150-request Qwen 2.5 7B validation runs, an example CI
+  policy, and a report of the six false passes prevented per run. Both
+  runs scored 78.7% with no matched-case regressions.
+
+Older results remain readable for informational comparison. Rerun them
+before resuming or using them as CI baselines with this rubric.
+
+## 0.4.0
+
+First public release.
+
+- Package the default suite inside the wheel, so `pip install callprobe`
+  works outside a git checkout. Previously the packaged CLI could not find
+  `suites/core` at all.
+- Add `callprobe validate [--suite PATH]`, which checks task expectations
+  against tool schemas without needing a model.
+- Add CI: pytest on Python 3.10 through 3.13, plus a wheel-install smoke
+  test that would have caught the packaging bug above.
+- Retry request errors (429, 5xx, connection and timeout errors) with
+  exponential backoff and jitter instead of scoring them as model failures.
+  Configurable with `--retries` (default 3). Reports honor `Retry-After`.
+- Reports now exclude errored requests from every success/selection/schema
+  /args rate, and warn when errors exceed 2% of requests. Error counts are
+  still shown.
+- Stop tracking log files and the overnight sweep log in git.
+- Add `__version__` and `callprobe --version`.
+- `--api-key` falls back to `API_KEY`, then `OPENAI_API_KEY`.
+- Add PyPI packaging metadata (urls, keywords, classifiers, authors) and a
+  trusted-publishing release workflow.

callprobe-0.5.0/CONTRIBUTING.md

@@ -0,0 +1,80 @@
+# Contributing
+
+## Writing a task
+
+A task lives in `tasks.yaml` and asserts what a model should do for one
+conversation. Pick the category that matches what would actually break if
+a model got it wrong:
+
+- `select` picked the right tool, or correctly called nothing, when the
+  choice between tools is the interesting part
+- `abstain` no tool applies and the correct behavior is to say so or ask
+  a question, not to guess
+- `args` the right tool was the obvious choice, but the argument values
+  are where a model can still get it wrong (arithmetic, units, enums,
+  omitted optional fields, normalization)
+- `depth` the conversation has multiple turns and the answer depends on
+  something said earlier
+- `sequence` several actions are implied, and the task checks which one
+  is the correct first call, not the whole chain
+
+Every task needs `id`, `category`, `bundle`, `messages`, and `expect`.
+`expect.type` is `call` or `no_call`. For a call, `args` is exact match on
+whichever keys you list (extra keys are judged by the tool's schema
+instead), and `arg_checks` covers everything else you want to assert
+about a value, at a dotted path like `address.postal_code` or
+`attendees.0`.
+
+`call` expects exactly one tool call; extra calls fail even when one is
+correct. A truncated response cannot pass either expectation. Full call
+evidence is retained in result JSON. Changes to these scoring semantics
+must increment `SCORING_VERSION` in `scoring.py` and add regression tests,
+so old and new observations are not silently combined.
+
+| op | checks |
+| --- | --- |
+| `eq` / `neq` | equals / does not equal |
+| `in` | value is one of a list |
+| `contains` | value contains a substring or element |
+| `gte` / `lte` | numeric comparison |
+| `matches` | value matches a regex |
+| `exists` / `absent` | the key is present / is not present |
+
+Add `exclude_distractors: [tool_name]` when padding could hand the model
+a distractor tool that would make your expected answer wrong, most often
+on `abstain` tasks. See `abstain-policy-question` in
+`suites/core/tasks.yaml` for the pattern: the distractor
+`search_knowledge_base` would genuinely answer the question, so it's
+excluded from padding for that task specifically.
+
+Once you've written a task, run:
+
+```bash
+callprobe validate
+```
+
+It checks that every key you asserted in `args` or `arg_checks` actually
+exists in the tool's schema, and that every value you asserted is legal
+for that schema. It runs with no model and exits nonzero on any problem.
+A typo here fails every model silently, so this is not optional.
+
+## Submitting a run
+
+Runs against models not yet in the leaderboard are welcome. Use the same
+flags the existing runs used, so the numbers are comparable:
+
+```bash
+callprobe run --model your-model --pad 0,8,16,24 --repeats 3 \
+  --max-tokens 4096 --out results/your-model.json
+```
+
+Include the results JSON file in your PR. Don't hand-edit it or the
+leaderboard, both are generated:
+
+```bash
+callprobe leaderboard results/your-model.json results/other-model.json \
+  > LEADERBOARD.md
+```
+
+Say what endpoint and quantization you ran against in the PR description
+if `--quant` doesn't already capture it.

callprobe-0.5.0/LEADERBOARD.md

@@ -0,0 +1,22 @@
+suite: core v2
+
+| model | success | 95% CI | type-lenient | selection | schema | args | abstain | success @ +24 tools | tokens per success |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| command-r7b | 22.0% | 12.0-34.0% | 22.0% | 22.0% | 22.0% | 22.0% | 100.0% | 22.0% | 9392 |
+| granite3.3:8b | 56.9% | 46.2-67.9% | 56.9% | 68.1% | 66.1% | 59.1% | 38.6% | 55.2% | 2066 |
+| hermes3:8b | 52.3% | 41.5-63.4% | 52.3% | 71.7% | 67.8% | 62.8% | 56.4% | 48.8% | 2173 |
+| llama3.2:3b | 28.4% | 18.3-39.1% | 40.9% | 61.3% | 36.7% | 40.4% | 4.1% | 25.2% | 3864 |
+| mistral-nemo | 63.8% | 52.7-74.3% | 63.8% | 68.7% | 75.2% | 64.4% | 59.5% | 63.6% | 1699 |
+| phi4-mini | 21.8% | 11.7-33.8% | 21.8% | 23.1% | 21.8% | 21.8% | 98.6% | 22.0% | 2992 |
+| qwen2.5:7b | 77.1% | 67.9-86.1% | 77.1% | 88.9% | 91.0% | 80.3% | 83.2% | 75.6% | 1408 |
+
+## Pending rerun against suite v2
+
+These two ran against suite v1 (34 tasks, no `files` bundle) and are not
+comparable to the table above. `callprobe leaderboard` refuses to mix
+them in automatically; they'll move up once rerun.
+
+| model | success | 95% CI | type-lenient | selection | schema | args | abstain | success @ +24 tools | tokens per success |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| llama3.1:8b | 32.1% | 19.4-46.1% | 51.2% | 66.9% | 33.6% | 36.8% | 9.8% | 34.3% | 3441 |
+| qwen3:8b | 88.7% | 78.9-96.6% | 88.7% | 93.1% | 93.6% | 89.0% | 88.6% | 88.2% | 1568 |

callprobe-0.5.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Elad Hirsh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.