probelock 0.1.0__tar.gz

probelock-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,17 @@
+name: ci
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - name: Tests
+        run: uv run --with pytest pytest -q
+      - name: Lint
+        run: uv run --with ruff ruff check .

probelock-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,28 @@
+name: release
+
+# Publish to PyPI via Trusted Publishing (OIDC) — no API tokens stored.
+# One-time setup on PyPI: add a "pending publisher" for project `probelock`
+# (owner: kelkalot, repository: probelock, workflow: release.yml).
+# Then push a tag like `v0.1.0` (or run this workflow manually) to publish.
+
+on:
+  push:
+    tags: ["v*"]
+  workflow_dispatch:
+
+jobs:
+  pypi:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read   # actions/checkout needs this for a private repo
+                       # (declaring any permission defaults all others to `none`)
+      id-token: write  # required to mint the OIDC token for Trusted Publishing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - name: Build sdist + wheel
+        run: uv build
+      - name: Check metadata
+        run: uvx twine check dist/*
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

probelock-0.1.0/.gitignore

@@ -0,0 +1,17 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+
+# Environments / tooling
+.venv/
+.pytest_cache/
+.ruff_cache/
+uv.lock
+
+# OS / editor
+.DS_Store
+.idea/
+.vscode/

probelock-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or Derivative
+          Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

probelock-0.1.0/PKG-INFO

@@ -0,0 +1,257 @@
+Metadata-Version: 2.4
+Name: probelock
+Version: 0.1.0
+Summary: A capability lockfile for local models. Catch silent regressions when you swap a model, quant, or runtime.
+Project-URL: Homepage, https://github.com/kelkalot/probelock
+Project-URL: Repository, https://github.com/kelkalot/probelock
+Project-URL: Issues, https://github.com/kelkalot/probelock/issues
+Author: Michael A. Riegler
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: ci,evals,llama-cpp,llm,local-models,mlx,ollama,quantization,regression-testing,tool-calling
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: jsonschema>=4.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.12
+Provides-Extra: anyllm
+Requires-Dist: any-llm-sdk>=0.2; (python_version >= '3.11') and extra == 'anyllm'
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Provides-Extra: litellm
+Requires-Dist: litellm>=1.0; extra == 'litellm'
+Description-Content-Type: text/markdown
+
+# probelock
+
+**A capability lockfile for local models.** It records what a model does on a set
+of tool-calling and output checks, and fails CI when a model/quant/runtime swap
+lowers a score.
+
+```
+llama-3.1-8b @ Q8_0 (ollama)  →  llama-3.1-8b @ Q4_K_M (ollama)
+Capability            Baseline   Candidate     Δ   Status
+arg_validity              1.00        0.67  -0.33  REGRESSION
+arity_robustness          1.00        0.67  -0.33  REGRESSION
+format_adherence          1.00        1.00  +0.00  ok
+needle_in_tools           1.00        0.33  -0.67  REGRESSION
+no_hallucinated_tool      1.00        0.67  -0.33  REGRESSION
+required_args             1.00        1.00  +0.00  ok
+structured_output         1.00        0.33  -0.67  REGRESSION
+tool_discrimination       1.00        0.33  -0.67  REGRESSION
+tool_permission           1.00        0.67  -0.33  REGRESSION
+tool_restraint            1.00        0.67  -0.33  REGRESSION
+tool_selection            1.00        0.67  -0.33  REGRESSION
+
+FAIL — capabilities regressed or removed: arg_validity, arity_robustness,
+needle_in_tools, no_hallucinated_tool, structured_output, tool_discrimination,
+tool_permission, tool_restraint, tool_selection
+```
+
+Here the Q4 quant scores 0.33–0.67 on several capabilities where Q8 scored 1.00.
+`probelock gate` exits non-zero when a capability drops past the threshold.
+
+## How it differs from promptfoo
+
+> **promptfoo is a test framework you author. probelock is a lockfile you commit.**
+
+1. **Probes are derived from your tool schemas.** Point it at the OpenAI-style
+   tool definitions your agent already ships, and it generates a fixed,
+   reproducible battery of capability checks. You write no test cases.
+2. **No LLM judge.** Every probe is scored by code: JSON-schema validation, exact
+   match, or a tool-name check. Run it twice on the same model and the numbers
+   match. (promptfoo relies on assertions you write and often on model-graded
+   evals, which vary across runs.)
+3. **It compares a model against its own baseline,** across a model/quant/runtime
+   swap, rather than producing an absolute leaderboard. You only ever compare like
+   with like, on your box, with your tools, so the "benchmarks are
+   gameable/hardware-dependent" objection doesn't apply.
+
+## Install & run (only needs [uv](https://docs.astral.sh/uv/))
+
+Run it without installing, or install it into the current environment:
+
+```bash
+uvx probelock --help          # run the latest release
+pip install probelock         # or install it
+```
+
+To run an unreleased revision straight from git:
+
+```bash
+uvx --from git+https://github.com/kelkalot/probelock probelock --help
+```
+
+The examples below use `uv run` from a checkout of this repo. No model is required
+for the demo — a deterministic `SimulatedClient` stands in for two quant levels of
+the same model:
+
+```bash
+# from the probelock/ project dir
+uv run probelock derive --tools examples/agent_tools.json          # see the probe battery
+uv run probelock probe  --tools examples/agent_tools.json --simulate fixtures/profile_q8.json -o q8.lock
+uv run probelock probe  --tools examples/agent_tools.json --simulate fixtures/profile_q4.json -o q4.lock
+uv run probelock diff   q8.lock q4.lock
+uv run probelock gate   --baseline q8.lock --candidate q4.lock     # exits non-zero
+```
+
+Against a local model, swap `--simulate` for an OpenAI-compatible endpoint:
+
+```bash
+uv run probelock probe --tools examples/agent_tools.json \
+    --endpoint http://localhost:11434/v1 --model llama3.1:8b-instruct-q4_K_M \
+    --quant Q4_K_M --runtime ollama --timeout 120 -o q4.lock
+```
+
+A probe the model rejects (e.g. "model does not support tools") or that times out
+scores **0** for that capability and the run continues, so a model that can't
+tool-call still produces a lockfile. An unreachable server, a 404 (wrong model or
+URL), or a run where every probe fails aborts the run, so a misconfiguration never
+becomes a poisoned all-zeros baseline.
+
+## Providers & frameworks
+
+probelock speaks one protocol — OpenAI `/v1/chat/completions` with OpenAI-style
+tools — so anything that exposes it works with `--endpoint`. For providers that
+don't (Anthropic, Gemini, …), route through a unified SDK with `--via`. Every path
+is deterministic; none of them put an LLM in the loop.
+
+| You have… | Use |
+|-----------|-----|
+| Ollama, **vLLM**, **llama.cpp** server, LM Studio, HF TGI, OpenAI, OpenRouter, Together… | `--endpoint <url>/v1 --model <name>` (vLLM needs `--enable-auto-tool-choice`; llama.cpp needs `--jinja`) |
+| **Anthropic / Gemini / Mistral / Bedrock / …** (any-llm) | `--via anyllm --model anthropic/claude-3-5-sonnet` |
+| Any of 100+ providers (litellm SDK) | `--via litellm --model anthropic/claude-3-5-sonnet` |
+| A running **LiteLLM proxy** | `--endpoint http://litellm:4000/v1 --model <name>` (no extra) |
+| In-process HF `transformers` / MLX (no server) | not yet — add a small `Client` adapter |
+
+```bash
+pip install 'probelock[anyllm]'   # or 'probelock[litellm]'
+probelock probe --tools tools.json --via anyllm --model mistral/mistral-large-latest \
+    --samples 5 --temperature 0.7 -o candidate.lock
+```
+
+`--via` clients reuse the same caching, sampling, and error semantics as
+`--endpoint`; they're thin adapters over each SDK's OpenAI-shaped response. Add a
+new backend by implementing the tiny `Client` protocol — that's the only seam.
+
+### Recorded demo (Ollama)
+
+[`demo/`](demo/) has runs against a local Ollama server: a committed `qwen3.5:9b`
+baseline vs a `gemma3:1b` candidate (which does not support tool-calling). See
+[`demo/DEMO.md`](demo/DEMO.md) for the transcript, or replay it:
+
+```bash
+asciinema play demo/probelock-demo.cast   # or: bash demo/demo.sh
+```
+
+The tool-calling capabilities drop `1.00 → 0.00` and the gate exits non-zero.
+`tool_restraint`, `tool_permission`, and `no_hallucinated_tool` stay `1.00` (a
+model that can't call tools can't misuse one), and `gemma3:1b` scores `1.00` on
+`format_adherence` vs `0.50` for `qwen3.5:9b`. The diff is per-capability.
+
+Also committed: `qwen3.5:9b` vs `lfm2.5-thinking:1.2b`:
+
+```bash
+uv run probelock diff demo/qwen3.5-9b.lock demo/lfm2.5-thinking.lock
+```
+
+The 1.2B model matches `qwen3.5:9b` on tool selection, discrimination,
+`needle_in_tools`, `arg_validity`, `required_args`, and the three safety probes;
+`structured_output` and `arity_robustness` drop `1.00 → 0.33`.
+
+## The capabilities (all scored deterministically)
+
+| Capability            | What it checks                                            | Scorer |
+|-----------------------|-----------------------------------------------------------|--------|
+| `tool_selection`      | Calls the right tool for the task                         | tool-name match |
+| `tool_discrimination` | Calls the right tool **and no other** (picks precisely)   | tool-name set |
+| `needle_in_tools`     | Finds the right tool when many (15+) are offered          | tool-name match |
+| `arg_validity`        | Emitted args validate against the tool's JSON schema      | `jsonschema` |
+| `required_args`       | All required args present and non-empty                   | key presence |
+| `arity_robustness`    | Fills **every** parameter (required + optional) when asked | all-present |
+| `structured_output`   | Emits schema-valid JSON on demand (no tools, no fences)   | parse + `jsonschema` |
+| `tool_restraint`      | Does **not** call a tool for a task that needs none (over-trigger) | no tool call |
+| `tool_permission`     | Does **not** call a tool it was explicitly forbidden to use | forbidden tool absent |
+| `no_hallucinated_tool`| Does **not** fabricate a call to a tool that wasn't offered | called ⊆ offered |
+| `format_adherence`    | Follows an exact output constraint                        | exact match |
+
+Three are **negative** probes (a higher score means the bad behavior didn't
+happen): `tool_restraint` (over-triggering), `tool_permission` (calling a forbidden
+tool), and `no_hallucinated_tool` (fabricating a tool). All probes are derived from
+the tool schemas, not hand-authored.
+
+## Architecture
+
+```
+tool schemas ──▶ derive probes ──▶ Client ──▶ ResponseMessage ──▶ deterministic scorer ──▶ Lockfile
+ (your agent)    (zero authoring)  (model)    (the only model      (no LLM judge)          (commit it)
+                                              -touching part)
+                                                                    Lockfile + Lockfile ──▶ diff / gate
+```
+
+The only nondeterministic part is the `Client`; everything else is pure, so the
+same inputs produce the same lockfile and the same diff. At temperature 0 the
+client caches identical requests, so the probes that share one request (the tool
+checks for a given tool) hit the network once. The `SimulatedClient` crafts correct or
+incorrect responses that the real scorers grade, so the scoring path runs even
+with no model present.
+
+## Sampling & noisy gates
+
+With one sample per probe, a capability backed by 3 tools quantizes to
+`{0, 0.33, 0.67, 1.0}` — a single flip moves it 0.33, far past the default 0.05
+gate. So:
+
+- **`probe --samples N [--temperature T]`** runs each probe N times and records the
+  pass-*rate* (raise the temperature for sampling variance).
+- **`gate --confidence 0.95`** only fails on a drop that is statistically
+  significant for the recorded trial count (a one-sided two-proportion test).
+  Sub-significant drops are shown as **`noisy ↓`** and do **not** fail the gate;
+  raise `--samples` to confirm or clear them.
+
+A total collapse (e.g. `1.00 → 0.00`) is significant even at low N; a single-flip
+`1.00 → 0.67` over 3 trials is `noisy` until you raise `--samples`.
+
+## In CI
+
+`probelock init` scaffolds a `probelock.tools.json` and a
+`.github/workflows/probelock.yml` to start from. Commit a baseline lockfile, then
+gate each candidate:
+
+```yaml
+- run: uvx probelock probe
+       --tools tools.json --endpoint $LLM_URL --model $MODEL --samples 5 --temperature 0.7 -o candidate.lock
+- run: uvx probelock gate
+       --baseline probelock.lock --candidate candidate.lock --max-drop 0.05 --confidence 0.95
+```
+
+A composite GitHub Action ([`action.yml`](action.yml)) wraps those two steps. To
+show the result on a pull request, render the diff as Markdown (or `--format html`
+for a self-contained page):
+
+```bash
+probelock diff probelock.lock candidate.lock --format markdown >> "$GITHUB_STEP_SUMMARY"
+```
+
+## Roadmap
+
+- A `--json-mode` `structured_output` probe (`response_format`) beside the strict prompt path.
+- Trend view: compare a capability across more than two lockfiles (a quant ladder Q8→Q5→Q4→Q3).
+- In-process backends (HF `transformers` / MLX) via a small `Client` adapter, no server required.
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE).
+
+## Acknowledgements
+
+Built with [Claude Code](https://claude.com/claude-code).