abrade-cli 0.2.0__tar.gz → 0.3.0__tar.gz

{abrade_cli-0.2.0 → abrade_cli-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: abrade-cli
-Version: 0.2.0
+Version: 0.3.0
 Summary: Abrade's self-hosted CLI — run the injection suite on your own machine, results stay local, no SaaS account required.
 Keywords: ai-agents,security-testing,prompt-injection,red-teaming,llm-security
 Author: Abrade
@@ -16,6 +16,7 @@ Classifier: Topic :: Security
 Classifier: Topic :: Software Development :: Testing
 Requires-Dist: anthropic>=0.119.0
 Requires-Dist: fpdf2>=2.8.7
+Requires-Dist: httpx>=0.28.1,<1
 Requires-Dist: pydantic>=2.13.4
 Requires-Dist: pyyaml>=6.0.3
 Requires-Dist: rich>=15.0.0
@@ -127,6 +128,35 @@ Use any saved run as the baseline. `.abrade/runs/<id>.json` is the file;
 committing one to the repo, or fetching it from an artifact store, is how
 CI gets a baseline on an ephemeral runner.
 
+
+### Gating a release on it
+
+`--baseline` compares whatever two runs share. That is right for seeing what
+changed and wrong for a gate: reword a payload under its old id, stop firing
+the probes that fail, or loosen the rules, and nothing it compared got worse.
+`--release-guard` (CLI 0.3.0+) refuses a comparison a release cannot rest on:
+
+```bash
+abrade run --suite full --baseline .abrade/baseline.json --release-guard
+```
+
+| | |
+|---|---|
+| **The change under evaluation** | system prompt, tool config, posture, model requested, CLI version. Passes, and is named on the record. Testing a change is what a release is. |
+| **A policy** | Refused until you say it is intended with `--allow-change`, and then on the record that you did: `new-probes`, `removed-probes`, `corpus-order`, `scope`, `assertions`, `environment` (agent name, declared evidence level, API format, ledger schema). `new-probes` accepts adding a probe and not what it finds: a new probe that does not hold is still refused. |
+| **Not a decision** | A probe changed under its id, another judge or rubric, a probe that errored on either side, action evidence that went away, an edited or unfinished baseline, or an agent that got worse. No allowance covers these. |
+
+Whatever the contract alone can show is refused before anything is fired, so a
+comparison that cannot be valid costs nothing. A run that fails afterwards is
+still saved, with what the guard found and the baseline's contract in it, so it
+can be read without the baseline file.
+
+Two limits. The baseline has to come from a CLI that records run contracts, so
+the first one needs recording again. And it cannot tell you the two runs hit the
+same endpoint, because an endpoint is never recorded, not even as a digest. If
+neither run had a ledger it passes and says it compared replies only;
+`--action-gate` is what demands one.
+
 ### SARIF, for code scanning
 
 ```bash
@@ -148,6 +178,10 @@ What a finding carries is the payload's name, the verdict, the vector and
 the judge's reasoning, plus a pointer to `.abrade/runs/<id>.json` where the
 evidence actually lives.
 
+A call breaking a declared assertion is always a finding, and a ledger that could
+not be read is reported once per reason. These name the rule and not the call,
+because a call's arguments come from the agent.
+
 Fingerprints are stable across runs, keyed on the payload and the verdict
 rather than the run id — otherwise a nightly scan re-opens every alert
 every night, which is how a scanner gets switched off. `OVERREACH`
@@ -159,7 +193,8 @@ filing it as an error next to a leaked credential misleads about both.
 
 Runs are written to `.abrade/runs/<id>.json` next to wherever you invoke
 the command, the way `.git/` works. The only network calls are to your
-agent's endpoint and to Anthropic for judging.
+agent's endpoint, Anthropic for judging, and — if configured — your action
+ledger.
 
 ### Verdicts
 
@@ -187,6 +222,63 @@ agent controls.
 A payload without a canary reports no canary evidence rather than a pass.
 Coverage is reported, never assumed.
 
+### Optional: requiring action evidence
+
+`--action-gate` exits non-zero unless every probe finished, its actions were
+completely observed, and it passed your declared `assertions`:
+
+```bash
+abrade run --suite quick --gate 80 --action-gate
+```
+
+The score gate cannot answer this. With the ledger down a run keeps its text
+verdicts and clears any threshold, with every assertion `unevaluated`. The gate
+needs assertions and a ledger reporting sequence numbers, and it refuses a config
+without them before firing anything. An older ledger with no sequence numbers
+still works without the flag. See [the ledger contract](../docs/action-ledger-contract.md).
+
+A rule that could never fire stops the run, with or without the flag: a misspelled
+key under `assertions`, or a limit written as text. `abrade verify` reports the
+same thing first. A tool name one typo away from a tool the agent really called
+produces a warning, and fails `--action-gate`, because that rule may never have
+fired.
+
+### Optional: action ledgers in CI
+
+A local run can read `action_ledger_url`, authenticating with the variable
+named by `action_ledger_key_env`. In CI the branch under test cannot choose
+which secret to send or its destination. A configured ledger requires this
+pair in **protected workflow configuration**, separate from the tested YAML:
+
+```yaml
+env:
+  ABRADE_CI_LEDGER_KEY_ENV: LEDGER_KEY
+  ABRADE_CI_LEDGER_ORIGIN: https://ledger.example
+  LEDGER_KEY: ${{ secrets.ABRADE_LEDGER_KEY }}
+```
+
+The agent config may then name `LEDGER_KEY` and
+`https://ledger.example/actions`. The grant covers that HTTP(S) origin
+(scheme, hostname and effective port), not a URL prefix or a single path.
+Do not put a path, query or fragment in the workflow's origin value. AUT
+allowlists do not authorize ledger access. There are no implicit ledger
+defaults; CI without a ledger needs no extra settings.
+
+Only same-origin ledger redirects retain the credential. Different hosts,
+ports or schemes are refused and collection becomes unknown evidence.
+Ledger collection also has 4 MiB raw/decoded body ceilings and a five-second
+elapsed I/O budget shared across redirect responses. DNS resolution remains
+an OS-level limitation; this is not a DNS-inclusive end-to-end deadline.
+See [response collection limits](../docs/response-collection-limits.md).
+Local HTTP(S) ledgers, including loopback lab endpoints, remain supported
+without CI authorization variables. `abrade verify` reports a CI trust
+refusal without sending a request.
+
+Protect the workflow and the CLI installation as well as these variables;
+this boundary addresses config-only changes, not arbitrary code execution
+in a secret-bearing job. See [the ledger contract](../docs/action-ledger-contract.md)
+for evidence limits.
+
 ### Optional: linking a workspace
 
 ```bash
@@ -212,19 +304,31 @@ clearly-marked sample payloads. The curated Abrade corpus is released on
 a responsible-disclosure timeline and is not shipped here. Point
 `--corpus <path>` at your own file in the same format.
 
+**And it has not been measured for discrimination.** The payloads are
+real and they execute, but nobody has measured how often each one lands,
+or whether the set separates a careful agent from a careless one. The
+corpus that *has* been measured — five rounds, up to 63 runs, three
+models — is not the one in this package. So treat a score from the
+bundled sample as a demonstration that the tool works end to end, not as
+a rating of your agent.
+
 **Not built yet.** Pulling a signed corpus artifact from a registry.
 There is no telemetry collection of any kind — not "off by default", it
 does not exist.
 
 ## What leaves your machine
 
-Three destinations, and only the third is optional:
+These are the destinations; the last two are optional:
 
 1. **Your agent's endpoint** — every payload, because firing them at your
    agent is the test.
 2. **Anthropic** — every agent reply, for judging, using your
    `ABRADE_JUDGE_API_KEY`.
-3. **Abrade** — only when you run `abrade push`.
+3. **Abrade** — only when you run `abrade push`. That sends the results, including
+   each probe's reply and any action ledger entries with their arguments, and your
+   workspace stores them.
+4. **Your action ledger** — only when configured, using its separate key
+   to read the agent's reported tool actions.
 
 What stays: execution, and the run files under `.abrade/runs/`.
 
@@ -232,7 +336,9 @@ What stays: execution, and the run files under `.abrade/runs/`.
 mean offline, and this page used to say both.
 
 Your agent's endpoint and credential are read from the environment for
-the duration of a run and written nowhere.
+the duration of a run and written nowhere. A credential follows a redirect only
+within the origin it was configured for. A redirect anywhere else stops the call,
+and the result says so.
 
 ---
 

{abrade_cli-0.2.0 → abrade_cli-0.3.0}/README.md

@@ -100,6 +100,35 @@ Use any saved run as the baseline. `.abrade/runs/<id>.json` is the file;
 committing one to the repo, or fetching it from an artifact store, is how
 CI gets a baseline on an ephemeral runner.
 
+
+### Gating a release on it
+
+`--baseline` compares whatever two runs share. That is right for seeing what
+changed and wrong for a gate: reword a payload under its old id, stop firing
+the probes that fail, or loosen the rules, and nothing it compared got worse.
+`--release-guard` (CLI 0.3.0+) refuses a comparison a release cannot rest on:
+
+```bash
+abrade run --suite full --baseline .abrade/baseline.json --release-guard
+```
+
+| | |
+|---|---|
+| **The change under evaluation** | system prompt, tool config, posture, model requested, CLI version. Passes, and is named on the record. Testing a change is what a release is. |
+| **A policy** | Refused until you say it is intended with `--allow-change`, and then on the record that you did: `new-probes`, `removed-probes`, `corpus-order`, `scope`, `assertions`, `environment` (agent name, declared evidence level, API format, ledger schema). `new-probes` accepts adding a probe and not what it finds: a new probe that does not hold is still refused. |
+| **Not a decision** | A probe changed under its id, another judge or rubric, a probe that errored on either side, action evidence that went away, an edited or unfinished baseline, or an agent that got worse. No allowance covers these. |
+
+Whatever the contract alone can show is refused before anything is fired, so a
+comparison that cannot be valid costs nothing. A run that fails afterwards is
+still saved, with what the guard found and the baseline's contract in it, so it
+can be read without the baseline file.
+
+Two limits. The baseline has to come from a CLI that records run contracts, so
+the first one needs recording again. And it cannot tell you the two runs hit the
+same endpoint, because an endpoint is never recorded, not even as a digest. If
+neither run had a ledger it passes and says it compared replies only;
+`--action-gate` is what demands one.
+
 ### SARIF, for code scanning
 
 ```bash
@@ -121,6 +150,10 @@ What a finding carries is the payload's name, the verdict, the vector and
 the judge's reasoning, plus a pointer to `.abrade/runs/<id>.json` where the
 evidence actually lives.
 
+A call breaking a declared assertion is always a finding, and a ledger that could
+not be read is reported once per reason. These name the rule and not the call,
+because a call's arguments come from the agent.
+
 Fingerprints are stable across runs, keyed on the payload and the verdict
 rather than the run id — otherwise a nightly scan re-opens every alert
 every night, which is how a scanner gets switched off. `OVERREACH`
@@ -132,7 +165,8 @@ filing it as an error next to a leaked credential misleads about both.
 
 Runs are written to `.abrade/runs/<id>.json` next to wherever you invoke
 the command, the way `.git/` works. The only network calls are to your
-agent's endpoint and to Anthropic for judging.
+agent's endpoint, Anthropic for judging, and — if configured — your action
+ledger.
 
 ### Verdicts
 
@@ -160,6 +194,63 @@ agent controls.
 A payload without a canary reports no canary evidence rather than a pass.
 Coverage is reported, never assumed.
 
+### Optional: requiring action evidence
+
+`--action-gate` exits non-zero unless every probe finished, its actions were
+completely observed, and it passed your declared `assertions`:
+
+```bash
+abrade run --suite quick --gate 80 --action-gate
+```
+
+The score gate cannot answer this. With the ledger down a run keeps its text
+verdicts and clears any threshold, with every assertion `unevaluated`. The gate
+needs assertions and a ledger reporting sequence numbers, and it refuses a config
+without them before firing anything. An older ledger with no sequence numbers
+still works without the flag. See [the ledger contract](../docs/action-ledger-contract.md).
+
+A rule that could never fire stops the run, with or without the flag: a misspelled
+key under `assertions`, or a limit written as text. `abrade verify` reports the
+same thing first. A tool name one typo away from a tool the agent really called
+produces a warning, and fails `--action-gate`, because that rule may never have
+fired.
+
+### Optional: action ledgers in CI
+
+A local run can read `action_ledger_url`, authenticating with the variable
+named by `action_ledger_key_env`. In CI the branch under test cannot choose
+which secret to send or its destination. A configured ledger requires this
+pair in **protected workflow configuration**, separate from the tested YAML:
+
+```yaml
+env:
+  ABRADE_CI_LEDGER_KEY_ENV: LEDGER_KEY
+  ABRADE_CI_LEDGER_ORIGIN: https://ledger.example
+  LEDGER_KEY: ${{ secrets.ABRADE_LEDGER_KEY }}
+```
+
+The agent config may then name `LEDGER_KEY` and
+`https://ledger.example/actions`. The grant covers that HTTP(S) origin
+(scheme, hostname and effective port), not a URL prefix or a single path.
+Do not put a path, query or fragment in the workflow's origin value. AUT
+allowlists do not authorize ledger access. There are no implicit ledger
+defaults; CI without a ledger needs no extra settings.
+
+Only same-origin ledger redirects retain the credential. Different hosts,
+ports or schemes are refused and collection becomes unknown evidence.
+Ledger collection also has 4 MiB raw/decoded body ceilings and a five-second
+elapsed I/O budget shared across redirect responses. DNS resolution remains
+an OS-level limitation; this is not a DNS-inclusive end-to-end deadline.
+See [response collection limits](../docs/response-collection-limits.md).
+Local HTTP(S) ledgers, including loopback lab endpoints, remain supported
+without CI authorization variables. `abrade verify` reports a CI trust
+refusal without sending a request.
+
+Protect the workflow and the CLI installation as well as these variables;
+this boundary addresses config-only changes, not arbitrary code execution
+in a secret-bearing job. See [the ledger contract](../docs/action-ledger-contract.md)
+for evidence limits.
+
 ### Optional: linking a workspace
 
 ```bash
@@ -185,19 +276,31 @@ clearly-marked sample payloads. The curated Abrade corpus is released on
 a responsible-disclosure timeline and is not shipped here. Point
 `--corpus <path>` at your own file in the same format.
 
+**And it has not been measured for discrimination.** The payloads are
+real and they execute, but nobody has measured how often each one lands,
+or whether the set separates a careful agent from a careless one. The
+corpus that *has* been measured — five rounds, up to 63 runs, three
+models — is not the one in this package. So treat a score from the
+bundled sample as a demonstration that the tool works end to end, not as
+a rating of your agent.
+
 **Not built yet.** Pulling a signed corpus artifact from a registry.
 There is no telemetry collection of any kind — not "off by default", it
 does not exist.
 
 ## What leaves your machine
 
-Three destinations, and only the third is optional:
+These are the destinations; the last two are optional:
 
 1. **Your agent's endpoint** — every payload, because firing them at your
    agent is the test.
 2. **Anthropic** — every agent reply, for judging, using your
    `ABRADE_JUDGE_API_KEY`.
-3. **Abrade** — only when you run `abrade push`.
+3. **Abrade** — only when you run `abrade push`. That sends the results, including
+   each probe's reply and any action ledger entries with their arguments, and your
+   workspace stores them.
+4. **Your action ledger** — only when configured, using its separate key
+   to read the agent's reported tool actions.
 
 What stays: execution, and the run files under `.abrade/runs/`.
 
@@ -205,7 +308,9 @@ What stays: execution, and the run files under `.abrade/runs/`.
 mean offline, and this page used to say both.
 
 Your agent's endpoint and credential are read from the environment for
-the duration of a run and written nowhere.
+the duration of a run and written nowhere. A credential follows a redirect only
+within the origin it was configured for. A redirect anywhere else stops the call,
+and the result says so.
 
 ---
 

{abrade_cli-0.2.0 → abrade_cli-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "abrade-cli"
-version = "0.2.0"
+version = "0.3.0"
 description = "Abrade's self-hosted CLI — run the injection suite on your own machine, results stay local, no SaaS account required."
 readme = "README.md"
 requires-python = ">=3.12"
@@ -25,6 +25,7 @@ classifiers = [
 dependencies = [
     "anthropic>=0.119.0",
     "fpdf2>=2.8.7",
+    "httpx>=0.28.1,<1",
     "pydantic>=2.13.4",
     "pyyaml>=6.0.3",
     "rich>=15.0.0",

{abrade_cli-0.2.0 → abrade_cli-0.3.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "abrade-cli"
-version = "0.2.0"
+version = "0.3.0"
 description = "Abrade's self-hosted CLI — run the injection suite on your own machine, results stay local, no SaaS account required."
 readme = "README.md"
 # The project, not a personal handle. PyPI renders this as "Author" on
@@ -42,6 +42,7 @@ classifiers = [
 dependencies = [
     "anthropic>=0.119.0",
     "fpdf2>=2.8.7",
+    "httpx>=0.28.1,<1",
     "pydantic>=2.13.4",
     "pyyaml>=6.0.3",
     "rich>=15.0.0",