agentguard-eval 0.3.0__tar.gz

agentguard_eval-0.3.0/.github/workflows/ci.yml

@@ -0,0 +1,24 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: python -m pip install --upgrade pip
+      - run: pip install -e .
+      - run: pip install pytest ruff mypy
+      - run: ruff check src tests
+      - run: mypy
+      - run: pytest

agentguard_eval-0.3.0/.github/workflows/publish.yml

@@ -0,0 +1,22 @@
+name: Publish to PyPI
+
+# Run by hand (Actions > Publish to PyPI > Run workflow) once the PyPI trusted publisher is set
+# up. Uses PyPI Trusted Publishing (OIDC), so there is no token to store: the project's
+# publisher on PyPI is linked to this repo and workflow.
+on:
+  workflow_dispatch:
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python -m pip install --upgrade build
+      - run: python -m build
+      - uses: pypa/gh-action-pypi-publish@release/v1

agentguard_eval-0.3.0/.gitignore

@@ -0,0 +1,9 @@
+.venv/
+__pycache__/
+*.pyc
+dist/
+build/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+*.egg-info/

agentguard_eval-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Zohaib Qamar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agentguard_eval-0.3.0/PKG-INFO

@@ -0,0 +1,184 @@
+Metadata-Version: 2.5
+Name: agentguard-eval
+Version: 0.3.0
+Summary: Guardrails and evals for AI agents: a policy gate for tool calls, a checker for invented facts, and a benchmark for shell-command gates. Deterministic, no API keys.
+Project-URL: Homepage, https://github.com/zbs-ops/agentguard
+Project-URL: Source, https://github.com/zbs-ops/agentguard
+Project-URL: Issues, https://github.com/zbs-ops/agentguard/issues
+Author: Zohaib Qamar
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,ai-safety,evaluation,guardrails,hallucination,llm,pytest
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Security
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# agentguard
+
+> **Problem:** AI agents with tools invent facts, overspend, and get their shell commands talked around. Most stacks catch none of it.
+>
+> **Approach:** three deterministic guardrails, no API keys and no model calls: a policy gate for tool calls, a checker for invented values, and a benchmark for shell-command gates.
+>
+> **Proof:** 194 tests. The checker and the benchmark are ports of work measured on real agent logs and on three shipping AI coding tools.
+
+```bash
+pip install agentguard-eval
+```
+
+The package installs as `agentguard-eval` (PyPI found `agentguard` too close to an existing name) and imports as `agentguard`.
+
+Python 3.10+. Zero runtime dependencies. Everything here runs offline and gives the same answer every time, which is what makes it testable and what makes it safe to put in a CI job.
+
+---
+
+## What's in it
+
+Three things that go wrong when you give a language model real tools, and a guard for each.
+
+### 1. A policy gate: decide whether a tool call may run
+
+Every tool call is a *proposal*. The gate returns one of three answers: run it, hold it for a person, or refuse it. Reads run unattended; money, customer messages, and deletes wait for a human. It fails closed, checks the budget before anything else, and catches an agent that estimates one cost while its arguments move a bigger one.
+
+```python
+from agentguard.gate import evaluate, Budget
+
+d = evaluate("refund.create", estimated_cost_cents=100, args={"amount_cents": 9000},
+             budget=Budget(spent_cents=0, limit_cents=50_000))
+print(d.verdict.value, d.decided_by)   # REQUIRE_APPROVAL cost-understated
+print(d.reason)   # "The agent estimated $1.00, but the arguments move $90.00 — escalated for review."
+```
+
+Seven ordered checks, each one asserted by a test: a malformed tool name is refused before it can borrow another rule, an unknown tool fails closed, a negative or non-finite cost is refused before the budget math, the budget is checked before the tier, an understated or unusually large spend is escalated, and a runaway loop of writes is paused.
+
+`evaluate` judges one call. `Governor` runs the whole approval workflow around it: it holds what's waiting for a person, checks who is approving, charges the budget only when something actually runs, and writes every step to a hash-chained ledger.
+
+```python
+from agentguard.gate import Governor
+
+g = Governor(reviewer_keys={"you@company.com": "your-secret"})
+a = g.propose("billing-agent", "refund.create", {"amount_cents": 9000}, estimated_cost_cents=9000)
+print(a.state.value)                     # PENDING_APPROVAL
+
+token = g.mint_token(a.id, "you@company.com")   # only the key holder can make this
+g.decide(a.id, True, "you@company.com", token=token)
+print(a.state.value, g.spent_cents)     # EXECUTED 9000
+print(g.verify().message)               # All 4 entries verify against their hashes.
+```
+
+Each rule here is a bug the TypeScript project found and fixed: the agent that asked can't approve its own request; typing a reviewer's name isn't enough, it takes a token bound to that one action; an approval stops counting 24 hours after the request; and the budget is checked again at the moment an approved action runs, because two refunds that each fit alone can both get a yes. Ledgers are interchangeable with the TypeScript version: the same calls produce the same ledger byte for byte, hashes included (`tests/test_governor.py` checks this against a ledger the TypeScript store wrote).
+
+### 2. A provenance checker: catch invented facts
+
+Checks whether an agent's answer is backed by what its tools actually returned. Four checks: a confident answer after a tool failed, a value no tool returned and nobody stated, a record for the wrong entity passed off as the right one, and a refusal that cites a limit appearing nowhere in the transcript.
+
+```python
+from agentguard.provenance import check_transcript, load_transcript
+
+t = load_transcript({"steps": [
+    {"role": "tool_call", "id": "c1", "tool": "orders.get", "args": {"order_id": "W1"}},
+    {"role": "tool_result", "id": "c1", "tool": "orders.get", "status": "ok", "payload": {"eta": "2026-10-22"}},
+    {"role": "assistant", "content": "It arrives 2026-10-24."},
+]})
+v = check_transcript(t)
+print(v.honest, v.flags[0].kind, v.flags[0].values)   # False unsupported-value ('2026-10-24',)
+```
+
+It reads the user, the system prompt, the tool calls and their results, and normalizes values so the same fact in different forms ("May 19, 2024" and `2024-05-19`, "$1,024" and `1024`) matches. It knows an agent computes (a refund that's the difference of two returned prices is grounded), and it leaves vague language alone, because exact-match is the honest limit of a checker with no model behind it.
+
+For the short, high-yield list, `id_flags(verdict)` keeps only flags about an invented order number or product code, the fabrication that does the most harm.
+
+It reads your agent's real logs directly. `to_transcript` takes an **OpenAI** chat log (`tool_calls` / role `tool`), an **Anthropic** messages log (`tool_use` / `tool_result` blocks), or agentguard's own `{"steps": [...]}`, and works out which one it got:
+
+```python
+import json
+from agentguard.provenance import check_transcript, to_transcript
+
+v = check_transcript(to_transcript(json.load(open("run.json"))))
+```
+
+Tool calls a model also writes into its message text (Claude's `<function_calls>` markup) are stripped first, so tool names aren't read as invented claims. On tool-truth's example logs, agentguard prints exactly the flags and values the TypeScript original prints, in the same order (`tests/test_parity.py`).
+
+### 3. A benchmark: how often can a command gate be talked around?
+
+Give an agent a shell and you put a gate in front of it: a list of commands it may not run. This rewrites a forbidden command in 13 known bypass styles (chaining, quoting, `$IFS`, variable indirection, `awk`, `base64`, equivalent binaries) and counts a bypass **only when the forbidden effect actually happens in a sandbox**.
+
+```python
+from agentguard.shell import run_bench, format_report
+
+print(format_report(run_bench()))
+```
+
+```
+gate                bypass rate   (bypasses / effective)
+substring-denylist          31%   (14/45)
+word-denylist               78%   (35/45)
+tokenized-denylist          71%   (32/45)
+allowlist                    0%   (0/45)
+```
+
+Two findings worth the whole module: a more precise denylist is not a safer one (whole-word matching misses `rm${IFS}file` that a dumb substring search catches), and only the allowlist, which names what may run and refuses everything else, holds at 0%.
+
+---
+
+## A pytest plugin
+
+Installing agentguard adds fixtures so you can assert on agent behaviour in your own suite:
+
+```python
+def test_my_agent_is_grounded(assert_grounded):
+    assert_grounded(my_agent_transcript)      # fails, listing the invented values
+
+def test_my_policy_allows_reads(assert_tool_allowed):
+    assert_tool_allowed("crm.read")           # fails if the gate wouldn't auto-allow it
+
+def test_my_shell_gate_holds(assert_gate_holds):
+    assert_gate_holds(my_gate)                # fails, listing the commands that got through
+```
+
+## A command line
+
+```bash
+agentguard check run.json            # OpenAI, Anthropic or agentguard format; exit 1 if flagged
+agentguard check run.json --ids-only # only high-confidence invented IDs
+agentguard gate refund.create --cost 9000 --amount 9000
+agentguard bench                     # the shell-gate scorecard
+agentguard verify-ledger ledger.json # exit 1 and name the first edited entry
+```
+
+`agentguard check` exits non-zero when a transcript is flagged, so it can gate a CI job over saved agent runs.
+
+---
+
+## Where this comes from
+
+The checker and the benchmark are Python ports of two projects I tested on real data, and the write-ups there carry the honest numbers:
+
+- **[tool-truth](https://github.com/zbs-ops/tool-truth)**: the checker, run on 660+ real agent conversations from τ-bench and τ²-bench. The first version was useless on real logs; after fixes it caught agents inventing an order number a customer then relied on. Held-out results, good and bad, are written up in full.
+- **[gate-bypass-bench](https://github.com/zbs-ops/gate-bypass-bench)**: the benchmark, which I ran against the real command rules of Google's Gemini CLI, OpenAI's Codex CLI, and opencode. Their deny rules were bypassed 78–84% of the time; the same engines as allowlists held at 0%.
+- **[agent-governance](https://github.com/zbs-ops/agent-governance)**: the gate and the approval workflow, with a live demo and a tamper-evident audit log.
+
+Background: Chen & Lin, *denylist fragility in AI agents* ([arXiv:2606.15549](https://arxiv.org/abs/2606.15549)); Sethi et al., *fabrication after tool failure* ([arXiv:2609.14758](https://arxiv.org/abs/2609.14758)); Saltzer & Schroeder (1975) on fail-safe defaults.
+
+## Scope and limits
+
+- **No model, by design.** The checker uses exact-match provenance. It catches invented concrete values; it cannot judge paraphrase or multi-step arithmetic, and the TypeScript write-ups measure that boundary on real data.
+- **The gate's policy is a sensible default, not your policy.** The tiers and the $500 cap are the demo's; the point is the shape.
+- **The Governor keeps state in memory.** Save `g.ledger()` (each entry has `to_dict()`) wherever you keep records; `verify-ledger` checks it later. Reviewer keys are plain secrets you hand it; in production they belong behind your identity provider.
+- **The benchmark runs POSIX `sh`.** The rewrites re-spell dangerous-shaped commands; they only touch files inside a per-case sandbox.
+
+## How I build
+
+I work with an AI coding assistant as a pair programmer. I decide what gets built, review every change, and own what ships.
+
+## License
+
+MIT

agentguard_eval-0.3.0/README.md

@@ -0,0 +1,160 @@
+# agentguard
+
+> **Problem:** AI agents with tools invent facts, overspend, and get their shell commands talked around. Most stacks catch none of it.
+>
+> **Approach:** three deterministic guardrails, no API keys and no model calls: a policy gate for tool calls, a checker for invented values, and a benchmark for shell-command gates.
+>
+> **Proof:** 194 tests. The checker and the benchmark are ports of work measured on real agent logs and on three shipping AI coding tools.
+
+```bash
+pip install agentguard-eval
+```
+
+The package installs as `agentguard-eval` (PyPI found `agentguard` too close to an existing name) and imports as `agentguard`.
+
+Python 3.10+. Zero runtime dependencies. Everything here runs offline and gives the same answer every time, which is what makes it testable and what makes it safe to put in a CI job.
+
+---
+
+## What's in it
+
+Three things that go wrong when you give a language model real tools, and a guard for each.
+
+### 1. A policy gate: decide whether a tool call may run
+
+Every tool call is a *proposal*. The gate returns one of three answers: run it, hold it for a person, or refuse it. Reads run unattended; money, customer messages, and deletes wait for a human. It fails closed, checks the budget before anything else, and catches an agent that estimates one cost while its arguments move a bigger one.
+
+```python
+from agentguard.gate import evaluate, Budget
+
+d = evaluate("refund.create", estimated_cost_cents=100, args={"amount_cents": 9000},
+             budget=Budget(spent_cents=0, limit_cents=50_000))
+print(d.verdict.value, d.decided_by)   # REQUIRE_APPROVAL cost-understated
+print(d.reason)   # "The agent estimated $1.00, but the arguments move $90.00 — escalated for review."
+```
+
+Seven ordered checks, each one asserted by a test: a malformed tool name is refused before it can borrow another rule, an unknown tool fails closed, a negative or non-finite cost is refused before the budget math, the budget is checked before the tier, an understated or unusually large spend is escalated, and a runaway loop of writes is paused.
+
+`evaluate` judges one call. `Governor` runs the whole approval workflow around it: it holds what's waiting for a person, checks who is approving, charges the budget only when something actually runs, and writes every step to a hash-chained ledger.
+
+```python
+from agentguard.gate import Governor
+
+g = Governor(reviewer_keys={"you@company.com": "your-secret"})
+a = g.propose("billing-agent", "refund.create", {"amount_cents": 9000}, estimated_cost_cents=9000)
+print(a.state.value)                     # PENDING_APPROVAL
+
+token = g.mint_token(a.id, "you@company.com")   # only the key holder can make this
+g.decide(a.id, True, "you@company.com", token=token)
+print(a.state.value, g.spent_cents)     # EXECUTED 9000
+print(g.verify().message)               # All 4 entries verify against their hashes.
+```
+
+Each rule here is a bug the TypeScript project found and fixed: the agent that asked can't approve its own request; typing a reviewer's name isn't enough, it takes a token bound to that one action; an approval stops counting 24 hours after the request; and the budget is checked again at the moment an approved action runs, because two refunds that each fit alone can both get a yes. Ledgers are interchangeable with the TypeScript version: the same calls produce the same ledger byte for byte, hashes included (`tests/test_governor.py` checks this against a ledger the TypeScript store wrote).
+
+### 2. A provenance checker: catch invented facts
+
+Checks whether an agent's answer is backed by what its tools actually returned. Four checks: a confident answer after a tool failed, a value no tool returned and nobody stated, a record for the wrong entity passed off as the right one, and a refusal that cites a limit appearing nowhere in the transcript.
+
+```python
+from agentguard.provenance import check_transcript, load_transcript
+
+t = load_transcript({"steps": [
+    {"role": "tool_call", "id": "c1", "tool": "orders.get", "args": {"order_id": "W1"}},
+    {"role": "tool_result", "id": "c1", "tool": "orders.get", "status": "ok", "payload": {"eta": "2026-10-22"}},
+    {"role": "assistant", "content": "It arrives 2026-10-24."},
+]})
+v = check_transcript(t)
+print(v.honest, v.flags[0].kind, v.flags[0].values)   # False unsupported-value ('2026-10-24',)
+```
+
+It reads the user, the system prompt, the tool calls and their results, and normalizes values so the same fact in different forms ("May 19, 2024" and `2024-05-19`, "$1,024" and `1024`) matches. It knows an agent computes (a refund that's the difference of two returned prices is grounded), and it leaves vague language alone, because exact-match is the honest limit of a checker with no model behind it.
+
+For the short, high-yield list, `id_flags(verdict)` keeps only flags about an invented order number or product code, the fabrication that does the most harm.
+
+It reads your agent's real logs directly. `to_transcript` takes an **OpenAI** chat log (`tool_calls` / role `tool`), an **Anthropic** messages log (`tool_use` / `tool_result` blocks), or agentguard's own `{"steps": [...]}`, and works out which one it got:
+
+```python
+import json
+from agentguard.provenance import check_transcript, to_transcript
+
+v = check_transcript(to_transcript(json.load(open("run.json"))))
+```
+
+Tool calls a model also writes into its message text (Claude's `<function_calls>` markup) are stripped first, so tool names aren't read as invented claims. On tool-truth's example logs, agentguard prints exactly the flags and values the TypeScript original prints, in the same order (`tests/test_parity.py`).
+
+### 3. A benchmark: how often can a command gate be talked around?
+
+Give an agent a shell and you put a gate in front of it: a list of commands it may not run. This rewrites a forbidden command in 13 known bypass styles (chaining, quoting, `$IFS`, variable indirection, `awk`, `base64`, equivalent binaries) and counts a bypass **only when the forbidden effect actually happens in a sandbox**.
+
+```python
+from agentguard.shell import run_bench, format_report
+
+print(format_report(run_bench()))
+```
+
+```
+gate                bypass rate   (bypasses / effective)
+substring-denylist          31%   (14/45)
+word-denylist               78%   (35/45)
+tokenized-denylist          71%   (32/45)
+allowlist                    0%   (0/45)
+```
+
+Two findings worth the whole module: a more precise denylist is not a safer one (whole-word matching misses `rm${IFS}file` that a dumb substring search catches), and only the allowlist, which names what may run and refuses everything else, holds at 0%.
+
+---
+
+## A pytest plugin
+
+Installing agentguard adds fixtures so you can assert on agent behaviour in your own suite:
+
+```python
+def test_my_agent_is_grounded(assert_grounded):
+    assert_grounded(my_agent_transcript)      # fails, listing the invented values
+
+def test_my_policy_allows_reads(assert_tool_allowed):
+    assert_tool_allowed("crm.read")           # fails if the gate wouldn't auto-allow it
+
+def test_my_shell_gate_holds(assert_gate_holds):
+    assert_gate_holds(my_gate)                # fails, listing the commands that got through
+```
+
+## A command line
+
+```bash
+agentguard check run.json            # OpenAI, Anthropic or agentguard format; exit 1 if flagged
+agentguard check run.json --ids-only # only high-confidence invented IDs
+agentguard gate refund.create --cost 9000 --amount 9000
+agentguard bench                     # the shell-gate scorecard
+agentguard verify-ledger ledger.json # exit 1 and name the first edited entry
+```
+
+`agentguard check` exits non-zero when a transcript is flagged, so it can gate a CI job over saved agent runs.
+
+---
+
+## Where this comes from
+
+The checker and the benchmark are Python ports of two projects I tested on real data, and the write-ups there carry the honest numbers:
+
+- **[tool-truth](https://github.com/zbs-ops/tool-truth)**: the checker, run on 660+ real agent conversations from τ-bench and τ²-bench. The first version was useless on real logs; after fixes it caught agents inventing an order number a customer then relied on. Held-out results, good and bad, are written up in full.
+- **[gate-bypass-bench](https://github.com/zbs-ops/gate-bypass-bench)**: the benchmark, which I ran against the real command rules of Google's Gemini CLI, OpenAI's Codex CLI, and opencode. Their deny rules were bypassed 78–84% of the time; the same engines as allowlists held at 0%.
+- **[agent-governance](https://github.com/zbs-ops/agent-governance)**: the gate and the approval workflow, with a live demo and a tamper-evident audit log.
+
+Background: Chen & Lin, *denylist fragility in AI agents* ([arXiv:2606.15549](https://arxiv.org/abs/2606.15549)); Sethi et al., *fabrication after tool failure* ([arXiv:2609.14758](https://arxiv.org/abs/2609.14758)); Saltzer & Schroeder (1975) on fail-safe defaults.
+
+## Scope and limits
+
+- **No model, by design.** The checker uses exact-match provenance. It catches invented concrete values; it cannot judge paraphrase or multi-step arithmetic, and the TypeScript write-ups measure that boundary on real data.
+- **The gate's policy is a sensible default, not your policy.** The tiers and the $500 cap are the demo's; the point is the shape.
+- **The Governor keeps state in memory.** Save `g.ledger()` (each entry has `to_dict()`) wherever you keep records; `verify-ledger` checks it later. Reviewer keys are plain secrets you hand it; in production they belong behind your identity provider.
+- **The benchmark runs POSIX `sh`.** The rewrites re-spell dangerous-shaped commands; they only touch files inside a per-case sandbox.
+
+## How I build
+
+I work with an AI coding assistant as a pair programmer. I decide what gets built, review every change, and own what ships.
+
+## License
+
+MIT

agentguard_eval-0.3.0/examples/anthropic-log.json

@@ -0,0 +1,17 @@
+[
+  { "role": "user", "content": "Where is order ORD-48213?" },
+  {
+    "role": "assistant",
+    "content": [
+      { "type": "text", "text": "Let me look that up." },
+      { "type": "tool_use", "id": "toolu_1", "name": "orders.lookup", "input": { "order_id": "ORD-48213" } }
+    ]
+  },
+  {
+    "role": "user",
+    "content": [
+      { "type": "tool_result", "tool_use_id": "toolu_1", "content": "{\"order_id\":\"ORD-48299\",\"status\":\"delivered\",\"eta\":\"2026-10-18\"}" }
+    ]
+  },
+  { "role": "assistant", "content": [{ "type": "text", "text": "Your order was delivered on 2026-10-18." }] }
+]

agentguard_eval-0.3.0/examples/ok-but-empty.json

@@ -0,0 +1,9 @@
+{
+  "id": "example-ok-but-empty",
+  "steps": [
+    { "role": "user", "content": "Where is my order ORD-48213 and when will it arrive?" },
+    { "role": "tool_call", "id": "c1", "tool": "orders.lookup", "args": { "order_id": "ORD-48213" } },
+    { "role": "tool_result", "id": "c1", "tool": "orders.lookup", "status": "ok", "payload": { "order_id": null, "status": null, "eta": null } },
+    { "role": "assistant", "content": "Order ORD-48213 is in transit with UPS and should arrive on 2026-10-24." }
+  ]
+}

agentguard_eval-0.3.0/examples/openai-log.json

@@ -0,0 +1,12 @@
+[
+  { "role": "user", "content": "What's the balance on account 7731?" },
+  {
+    "role": "assistant",
+    "content": null,
+    "tool_calls": [
+      { "id": "call_1", "type": "function", "function": { "name": "accounts.balance", "arguments": "{\"account\":\"7731\"}" } }
+    ]
+  },
+  { "role": "tool", "tool_call_id": "call_1", "content": "{\"account\":\"7731\",\"balance\":null,\"currency\":null}" },
+  { "role": "assistant", "content": "Account 7731 has a balance of $2,310.40." }
+]

agentguard_eval-0.3.0/pyproject.toml

@@ -0,0 +1,63 @@
+[build-system]
+requires = ["hatchling>=1.25"]
+build-backend = "hatchling.build"
+
+[project]
+name = "agentguard-eval"
+version = "0.3.0"
+description = "Guardrails and evals for AI agents: a policy gate for tool calls, a checker for invented facts, and a benchmark for shell-command gates. Deterministic, no API keys."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Zohaib Qamar" }]
+requires-python = ">=3.10"
+dependencies = []
+keywords = ["ai-safety", "llm", "agents", "guardrails", "evaluation", "hallucination", "pytest"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "Framework :: Pytest",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Security",
+  "Typing :: Typed",
+]
+
+[project.urls]
+Homepage = "https://github.com/zbs-ops/agentguard"
+Source = "https://github.com/zbs-ops/agentguard"
+Issues = "https://github.com/zbs-ops/agentguard/issues"
+
+[project.scripts]
+agentguard = "agentguard.cli:main"
+
+[project.entry-points.pytest11]
+agentguard = "agentguard.pytest_plugin"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/agentguard"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "UP", "B", "SIM", "RUF"]
+# The checker deliberately handles non-ASCII quotes (guillemets, curly quotes) in its patterns
+# and test data, so the ambiguous-character warnings are expected.
+ignore = ["RUF001", "RUF002", "RUF003", "E501"]
+
+[tool.ruff.lint.per-file-ignores]
+# Test data (transcripts, parametrized attack strings) is naturally wide; keep it readable.
+
+[tool.mypy]
+python_version = "3.10"
+strict = true
+files = ["src/agentguard"]

agentguard_eval-0.3.0/src/agentguard/__init__.py

@@ -0,0 +1,3 @@
+"""agentguard: guardrails and evals for AI agents."""
+
+__version__ = "0.3.0"

agentguard_eval-0.3.0/src/agentguard/cli.py

@@ -0,0 +1,101 @@
+"""The ``agentguard`` command line.
+
+- ``agentguard check <log.json>``: check an agent transcript (agentguard, OpenAI or Anthropic format) for invented values. Exit code 1
+  when it's flagged, so it can gate a CI job over saved runs. ``--ids-only`` keeps only
+  high-confidence invented-ID flags.
+- ``agentguard gate <tool> [--cost C] [--amount A] [--limit L] [--spent S]``: ask the policy
+  gate what it would do with a proposed tool call.
+- ``agentguard bench``: run the shell-command gate benchmark and print the scorecard.
+- ``agentguard verify-ledger <ledger.json>``: recompute every hash in an approval ledger and
+  name the first entry that was edited, cut or reordered. Exit code 1 if it doesn't verify.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from collections.abc import Sequence
+
+from agentguard.gate import Budget, evaluate, load_entries, verify_chain
+from agentguard.provenance import check_transcript, id_flags, to_transcript
+from agentguard.shell import format_report, run_bench
+
+
+def _cmd_check(args: argparse.Namespace) -> int:
+    with open(args.log, encoding="utf-8") as fh:
+        transcript = to_transcript(json.load(fh))
+    verdict = check_transcript(transcript)
+    flags = id_flags(verdict) if args.ids_only else list(verdict.flags)
+    if not flags:
+        print("honest: no ungrounded values found.")
+        return 0
+    for flag in flags:
+        values = f" [{', '.join(flag.values)}]" if flag.values else ""
+        print(f"{flag.kind}: {flag.message}{values}")
+    return 1
+
+
+def _cmd_gate(args: argparse.Namespace) -> int:
+    call_args: dict[str, object] = {}
+    if args.amount is not None:
+        call_args["amount_cents"] = args.amount
+    decision = evaluate(
+        args.tool,
+        estimated_cost_cents=args.cost,
+        args=call_args,
+        budget=Budget(spent_cents=args.spent, limit_cents=args.limit),
+    )
+    print(f"{decision.verdict.value}  ({decision.decided_by})")
+    for reason in decision.reasons:
+        print(f"  - {reason}")
+    return 0 if decision.verdict.value == "ALLOW" else 1
+
+
+def _cmd_bench(_args: argparse.Namespace) -> int:
+    print(format_report(run_bench()))
+    return 0
+
+
+def _cmd_verify_ledger(args: argparse.Namespace) -> int:
+    with open(args.ledger, encoding="utf-8") as fh:
+        try:
+            chain = load_entries(json.load(fh))
+        except ValueError as exc:
+            print(f"not a ledger: {exc}")
+            return 2
+    result = verify_chain(chain)
+    print(("valid: " if result.valid else "BROKEN: ") + result.message)
+    return 0 if result.valid else 1
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="agentguard", description="Guardrails and evals for AI agents.")
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p_check = sub.add_parser("check", help="check an agent transcript (JSON) for invented values")
+    p_check.add_argument("log", help="path to a transcript JSON file")
+    p_check.add_argument("--ids-only", action="store_true", help="keep only high-confidence invented-ID flags")
+    p_check.set_defaults(func=_cmd_check)
+
+    p_gate = sub.add_parser("gate", help="ask the policy gate about a proposed tool call")
+    p_gate.add_argument("tool")
+    p_gate.add_argument("--cost", type=int, default=0, help="the agent's estimated cost, in cents")
+    p_gate.add_argument("--amount", type=int, default=None, help="amount_cents the call would move")
+    p_gate.add_argument("--limit", type=int, default=50_000, help="budget cap, in cents")
+    p_gate.add_argument("--spent", type=int, default=0, help="cents already spent")
+    p_gate.set_defaults(func=_cmd_gate)
+
+    p_bench = sub.add_parser("bench", help="run the shell-command gate benchmark")
+    p_bench.set_defaults(func=_cmd_bench)
+
+    p_ledger = sub.add_parser("verify-ledger", help="check an approval ledger for tampering")
+    p_ledger.add_argument("ledger", help="path to a ledger JSON file (a list, or an object with 'chain')")
+    p_ledger.set_defaults(func=_cmd_verify_ledger)
+
+    args = parser.parse_args(argv)
+    return int(args.func(args))
+
+
+if __name__ == "__main__":
+    sys.exit(main())