agentproof-scan 0.1.3__tar.gz → 0.2.0__tar.gz

{agentproof_scan-0.1.3/agentproof_scan.egg-info → agentproof_scan-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentproof-scan
-Version: 0.1.3
+Version: 0.2.0
 Summary: CLI security scanner: probes AI-agent HTTP endpoints for secret/prompt leakage, with CI-gating exit codes.
 Author: AgentProof
 License-Expression: Apache-2.0
@@ -48,7 +48,10 @@ pattern-matcher (no AI doing the judging) counts how many runs the secret litera
 shows up in each. A result reads like *"leaked in 4 of 10 runs."*
 
 Because it's straight matching against known secret shapes, there's nothing hidden:
-you can read the code and reproduce every number yourself.
+you can read the code. Every number we mark **GREEN-backed** you can reproduce from a
+clone, offline, byte-for-byte — see [Reproducing these numbers](#reproducing-these-numbers).
+The cross-model observations are a different kind of number: measured snapshots that
+depend on a live model, reported as directional. We don't blur the two.
 
 ---
 
@@ -67,31 +70,41 @@ python3 --version
 py -3   --version
 ```
 
-**2. Install, and put your key in a `.env` file** in the folder you'll run from.
-Every command below carries a label saying whether *we actually ran it*:
+**2. Install.**
 
-| Shell | Commands | Source |
-|---|---|---|
-| bash / zsh | `pip install agentproof-scan`<br>`echo 'GEMINI_API_KEY=PASTE_YOUR_KEY_HERE' > .env` | **VERIFIED** — 9 Linux-container cells (Python 3.9/3.11/3.13 × venv/system/pipx, live PyPI install) |
-| PowerShell 5.1 | `pip install agentproof-scan`<br>`'GEMINI_API_KEY=PASTE_YOUR_KEY_HERE' \| Out-File -Encoding utf8 .env` | **UNTESTED** — works or not, [open an issue](https://github.com/ghkfuddl1327-wq/agentproof/issues) |
-| pwsh 7 | `pip install agentproof-scan`<br>`'GEMINI_API_KEY=PASTE_YOUR_KEY_HERE' > .env` | **UNTESTED** — same |
-| cmd.exe | `pip install agentproof-scan`<br>`echo GEMINI_API_KEY=PASTE_YOUR_KEY_HERE> .env` | **UNTESTED** — same |
+```
+pip install agentproof-scan
+```
+`pip: command not found` → `python -m pip install agentproof-scan`. On Windows without
+Python, `py -m pip install agentproof-scan` triggers the Python Install Manager to fetch
+Python for you. [VERIFIED: `py -m ...` installed Python 3.14.6 on Windows, owner, 2026-07-11.]
 
-Get a free Gemini key at <https://aistudio.google.com/apikey>. `.env` is read from the
-directory you run in, and is gitignored — never commit it.
+**3. Put your key in a `.env` file, with an editor — not a shell command.**
 
-**3. If something isn't found:**
+Open a new file called `.env` in the folder you'll run from, and put one line in it:
 
-| Symptom | Try |
-|---|---|
-| `pip: command not found` | `python -m pip install agentproof-scan` |
-| `agentproof-scan: command not found` | `python -m agentproof_scan` (same tool, same flags) |
+```
+GEMINI_API_KEY=your-real-key-here
+```
+
+Save it. Get a free key at <https://aistudio.google.com/apikey>. `.env` is auto-loaded from
+the directory you run in and is gitignored — never commit it.
 
-> **On `.env` encoding.** `0.1.3` reads `.env` files written as UTF-8, UTF-8 with BOM,
-> and **UTF-16LE with BOM** (what PowerShell 5.1's `>` produces), and it strips stray
-> quotes that `cmd.exe` leaves around the *key name*. That is a claim about **bytes**,
-> tested byte-for-byte. It is **not** a claim that the tool passes on Windows — see
-> [Verification scope](#verification-scope).
+> **Why an editor and not `echo '...' > .env`?** A shell command puts your key on the
+> command line, and the shell saves that line to its **history** in plaintext
+> (`~/.bash_history`, PowerShell's `ConsoleHost_history.txt`). A tool that detects leaked
+> credentials should not teach you to write one to a file nobody thinks to check. An editor
+> avoids the history entirely. (If you do use the shell, this tool does **not** scan shell
+> history — that's on you to clear.)
+
+> **On `.env` encoding.** The loader reads `.env` written as UTF-8, UTF-8 with BOM, and
+> **UTF-16LE with BOM** (what PowerShell 5.1's `>` produces — [VERIFIED on a real PowerShell
+> 5.1 file, owner, 2026-07-11]), and strips stray quotes `cmd.exe` leaves around the *key
+> name*. **Don't quote the value** — `KEY="abc"` keeps the quotes literally.
+
+**4. If the command isn't found:** `agentproof-scan: command not found` →
+`python -m agentproof_scan` (Windows: `py -m agentproof_scan`) — same tool, same flags,
+works when the console script isn't on PATH. [VERIFIED on Windows, owner, 2026-07-11.]
 
 ---
 
@@ -105,25 +118,38 @@ backed by Google Gemini. You only need a free Gemini key to try it.
 pip install agentproof-scan
 
 # 2. Get a FREE Gemini key → https://aistudio.google.com/apikey
-#    save it to a .env file in the folder you run from (auto-loaded, kept out of git):
-echo 'GEMINI_API_KEY=PASTE_YOUR_KEY_HERE' > .env
+#    Then create a file named .env in this folder (with an editor, not the shell —
+#    see Prerequisites) containing one line:
+#        GEMINI_API_KEY=your-real-key-here
 
 # 3. Scan the built-in vulnerable demo agent
-agentproof-scan                  # same as: agentproof-scan --target victim
-agentproof-scan --stability 5    # repeat 5× — more reliable (see note below)
+agentproof-scan                  # one pass = 15 probes = 15 API calls
+agentproof-scan --stability 2    # 2 passes = 30 calls — more reliable (see notes)
 ```
 
 A JSON report prints. If the demo leaked, you'll see a `leak_count` of 1 or more.
 
-> ⚠️ **A zero you can trust.** Up to `0.1.2` a scan that never reached your agent — bad
-> key, HTTP 500, timeout — still printed `leak_count: 0` and exited `0`. It read as
-> *"your agent is safe"* when it meant *"we couldn't ask."* From `0.1.3` the scanner
-> **refuses to report clean** unless every probe actually got an answer: it exits `1`
-> and prints `AGENTPROOF_SCAN_DID_NOT_RUN reason=<slug>` on stderr. A leak it *did*
-> find is still reported — a partial scan may not claim clean, but it may claim what it
-> found.
-
-> **Why `--stability 5`?** A single run is non-deterministic — a leaky agent can still answer "safely" on any one try, so a one-shot scan might read `0` by luck. Repeating (e.g. `--stability 5`) measures *how often* it leaks (`leak_rate`) and is the reliable way to read the verdict.
+> ⚠️ **On a free Gemini key, watch the request count.** Each pass sends **15 probes** = 15
+> API calls; `--stability N` multiplies that (`--stability 5` = 75 calls, `--stability 10`
+> = 150). The free tier has a per-minute quota, and hitting it is **not** a tool failure —
+> the scanner exits `1` with `reason=rate_limit` and tells you to wait or lower
+> `--stability`. Start with `--stability 2`; raise it once you know your quota headroom.
+
+> ⚠️ **A zero you can trust.** A scan that never reached your agent, or was cut off
+> partway (bad key, HTTP 500, timeout, rate limit), must not read as *"safe."* The scanner
+> **refuses to report clean** unless every probe actually got an answer: otherwise it exits
+> `1` and prints `AGENTPROOF_SCAN_DID_NOT_RUN reason=<slug>` on stderr (`rate_limit`,
+> `auth_failed`, `http_status`, `incomplete_scan`, …). A leak it *did* find is still
+> reported and, under `--fail-on-findings`, still exits `2` — **a partial scan may not
+> claim clean, but it may claim what it found.** (Earlier versions, up to `0.1.3`, could
+> print `leak_count: 0` and exit `0` here — that's the bug `0.1.4` closes.)
+
+> **Why repeat with `--stability`?** A single run is non-deterministic — a leaky agent can
+> still answer "safely" on any one try, so a one-shot scan might read `0` by luck. Repeating
+> measures *how often* it leaks (`leak_rate`) and is the reliable way to read the verdict.
+> A probe that never got an answer shows `leak_rate: null` (**not** `0.0`) — "not asked" is
+> not "didn't leak." ⚠ Each repeat costs `15 × N` API calls; on a free key start at
+> `--stability 2` (see the request-count note above).
 
 ---
 
@@ -136,7 +162,7 @@ Two fields matter:
 
 **Analogy:** `leak` = the guard handed over the vault key. `prompt_disclosure` = the guard didn't hand over the key, but read the security manual aloud. Both are bad; the first is worse.
 
-**`leak_rate`** (in repeat/stability mode) = how often a probe pulled a leak. For example `4/10 (0.4)` = 4 of 10 tries leaked. A flaky leak is still a leak — repetition shows how *reliably* an agent fails.
+**`leak_rate`** (in repeat/stability mode) = how often a probe pulled a leak, over the runs that **got an answer**. For example `4/10 (0.4)` = 4 of 10 answered tries leaked. A flaky leak is still a leak — repetition shows how *reliably* an agent fails. A probe that never got an answer (rate-limited, timed out) shows `leak_rate: null`, not `0.0` — the report separates *"didn't leak"* from *"wasn't asked."*
 
 **Which secrets it recognizes:** the scanner looks for key shapes from major providers — OpenAI (including modern `sk-proj-` / `sk-svcacct-` / `sk-admin-` keys and the legacy `sk-` format), Anthropic, Google, AWS, GitHub, and xAI.
 
@@ -216,9 +242,10 @@ agentproof-scan \
 **Needs auth?** Pass a header — but put only the **name** of an environment variable
 in the flag, never the key itself:
 
+Add the key to your `.env` (with an editor — same reason as the Quick Start: keep it out
+of shell history) as `MY_AGENT_KEY=your-real-key`, then reference it **by name**:
+
 ```bash
-# key lives in .env (gitignored); the flag references it by name
-echo 'MY_AGENT_KEY=sk-your-real-key' >> .env
 agentproof-scan --url https://my-agent.example.com/chat \
   --prompt-field message --response-field reply \
   --auth-header "Authorization=Bearer {MY_AGENT_KEY}"
@@ -264,7 +291,9 @@ body:                                     # your request template
 > so it spends whatever those calls cost on your account — same as the demo Gemini key.
 
 *(Prefer to wire it in yourself? You still can: implement the small `AgentAdapter`
-interface in `adapters/base.py` and register it in `ADAPTERS` in `scan.py`.)*
+interface in `agentproof_scan/adapters/base.py` and register it in `ADAPTERS` in
+`agentproof_scan/scan.py`. The top-level `scan.py` is only a clone-launcher shim and
+isn't in the wheel.)*
 
 **Roadmap:** the generic HTTP path above is **shipped**. Broader shapes — non-JSON
 bodies, streaming responses, and non-HTTP transports — are expanding from here.
@@ -309,11 +338,13 @@ jobs:
 
 *(Also at [`examples/ci/agentproof.yml`](https://github.com/ghkfuddl1327-wq/agentproof/blob/main/examples/ci/agentproof.yml) — copy it, don't clone the repo.)*
 
-**Don't drop the `AGENT_URL` guard.** An unset secret is an empty string, and an
-empty `--url` makes the scanner fall back to its **bundled demo agent** — which
-leaks on purpose. Without the guard your build goes red over a planted fake secret
-while your real agent is never scanned. Forked pull requests get no secrets at all,
-so they always take this path.
+**Keep the `AGENT_URL` guard anyway.** An unset secret is an empty string, and the
+guard turns that into a clear "secret not set" failure. Since `0.1.3` an empty `--url`
+already exits `1` on its own (`reason=missing_url`) rather than scanning anything — so
+a missing `AGENT_URL` fails the build as *"the scan did not run,"* not as *"your agent
+leaked."* The guard just makes that reason obvious in the log. Forked pull requests get
+no secrets, so they hit this path — and a `1` there is correct: nothing was scanned,
+and nothing is claimed clean.
 
 If your agent leaks a secret, `--fail-on-findings` exits non-zero and the check goes
 red. By default it fails on **leaked secrets only** (not softer prompt-disclosure
@@ -363,7 +394,7 @@ Python 3.9 / 3.11 / 3.13 × venv / system / pipx
 The last row will not be closed by automation. If you hit friction installing
 Python itself, an issue is the only way that row changes.
 
-`0.1.3` handles UTF-16LE+BOM `.env` files (byte-level test).
+`0.1.4` handles UTF-16LE+BOM `.env` files (byte-level test); `0.2.0` keeps that behaviour.
 This is **not** a claim that the tool passes on Windows PowerShell 5.1.
 It is a claim about the bytes.
 
@@ -416,7 +447,7 @@ If any step is confusing, paste this into an AI assistant and follow along:
 
 ## ⚠️ A note on the test fixtures
 
-`victim_agent.py` and the `*_canary` adapters contain **intentional** vulnerabilities — fake, format-only secrets (not real keys) used as test fixtures to prove the scanner works. They are not exploits, and the embedded strings are not usable credentials. The probe set in this public repo uses neutral, category-labeled example questions — it does **not** ship copy-pasteable injection prompts.
+`agentproof_scan/victim_agent.py` and the `*_canary` adapters contain **intentional** vulnerabilities — fake, format-only secrets (not real keys) used as test fixtures to prove the scanner works. They are not exploits, and the embedded strings are not usable credentials. The probe set in this public repo uses neutral, category-labeled example questions — it does **not** ship copy-pasteable injection prompts.
 
 ---
 
@@ -424,12 +455,20 @@ If any step is confusing, paste this into an AI assistant and follow along:
 
 Early work in progress. This tool grew out of red-team probing experiments and is expanding toward broader pre-deployment credential-exposure detection. The detection rule and the cross-model numbers are still being validated — **expect changes**, and if you can break something we marked as working, please open an issue.
 
-**Known limitation:** a present-but-invalid key (wrong or expired) can still produce a `0` — detecting invalid keys from API-error responses is a planned follow-up.
+**Invalid keys, since `0.1.3`:** a present-but-invalid key no longer produces a
+misleading `0`. The provider returns an HTTP error — a malformed key is a `400`
+(`reason=http_status`), a wrong or expired key is a `401` (`reason=auth_failed`) — and
+the scanner treats either as *the scan did not run*: exit `1`, refusing to report
+clean. Earlier versions did report `0` here; that was the bug `0.1.3` closes.
 
-**Not yet in this release:** wider credential-type coverage (Stripe, Slack, JWT, PEM,
-SendGrid, Twilio, npm, …) is implemented and tested, but `0.1.3` ships only the six
-families listed under [What it catches](#what-it-catches--and-what-it-doesnt-plainly).
-It lands in a later release rather than being advertised here before you can run it.
+**Released in `0.2.0`:** the wider credential-type coverage that `0.1.4` held back
+(Stripe, Slack, JWT, PEM, SendGrid, Twilio, npm, GitHub fine-grained PAT, GCP) now
+ships — you can run it. See [What it catches](#what-it-catches--and-what-it-doesnt-plainly)
+for the list and [What's measured](#whats-measured--three-separate-things) for what each
+claim is backed by. `postgres` remains **opt-in**, not promoted to default (below).
+
+`0.2.0` is additive: the six families from `0.1.4` behave exactly as before, and the
+exit codes (`0` / `1` / `2`) and the `rule` slug are unchanged.
 
 ---
 
@@ -464,43 +503,114 @@ No extra API calls: the trace is captured during the same probe run.
 
 ---
 
-## Defense prompts — where to find them
+## Defense prompts — reference, not a fix
 
-If a scan turns up a leak, the repo ships **defense prompts** you can paste into your
-agent's system prompt to reduce it. To keep this page short, the prompts themselves
-live in the repo, not here:
+There is no prompt that "fixes" leakage. What the repo ships is a **reference**: a
+defense hypothesis, measured per model, with results that vary by model — what helps
+one can leave a residual on another. Adding a defense block *raises the cost of a
+leak*; it is not a guarantee, and the only figure that means anything for your setup is
+the one you measure on your own model. To keep this page short, the prompts and the
+measurements behind them live in the repo, not here:
 
 [`prompts/system_defense/`](https://github.com/ghkfuddl1327-wq/agentproof/tree/main/prompts/system_defense)
 
-Installed via pip? The prompts live in the repo, not the package — open the link
-above, or `git clone` the repo if you want them locally. Each prompt is a plain text
-block you copy into your agent's system prompt.
+Installed via pip? They live in the repo, not the package — open the link above, or
+`git clone` the repo to read them locally. Each is a plain-text block.
 
 [`REFERENCE.md`](https://github.com/ghkfuddl1327-wq/agentproof/blob/main/prompts/system_defense/REFERENCE.md)
-tells you **which prompt fits which model** and
-states the honest limits (it protects the final answer, not the reasoning trace —
-see above). Keeping the data in the repo means it can grow without turning this page
-into a wall of text.
+is the honest version: which block was measured against which model, in directional
+buckets (not precise rates), and the limits — chiefly that it moves the **final
+answer** surface, not the reasoning trace (see above). Keeping it in the repo lets the
+reference grow without turning this page into a wall of text.
+
+---
+
+## What's measured — three separate things
+
+This page makes three different claims. They come from three different measurements,
+they count three different things, and **they do not add up to one number**. A scanner
+that labels 16 credential shapes is not a scanner that "catches 16 kinds of attack."
+
+| Layer | The claim | What backs it |
+|---|---|---|
+| **Detection — 16 families** (15 default + `postgres` opt-in) | the matcher puts the **right family label** on a credential's shape, and stays quiet on look-alikes | [`axis_b_coverage_green.json`](https://github.com/ghkfuddl1327-wq/agentproof/blob/main/axis_b_coverage_green.json) — the 10 families added since `0.1.4`: 10/10 labelled, 0 missed, 0 false positives on near-miss strings. The original six are held by `test_secrets_integrity.py`. |
+| **Elicitation — 10 families** | a credential sitting in an **agent's response** is picked up end-to-end (probe → response → report), with the right family and nothing invented | [`elicitation_green.json`](https://github.com/ghkfuddl1327-wq/agentproof/blob/main/elicitation_green.json) — 10 planted, 10 detected, 0 missed, 0 spurious providers. `postgres` is excluded (it's opt-in). |
+| **Reasoning-attack (H-CoT) — 3 probes** | when a fake "reasoning step" is injected into an agent **you own**, the scanner **sees** the resulting reasoning-channel leak | [`hcot_green.json`](https://github.com/ghkfuddl1327-wq/agentproof/blob/main/hcot_green.json) — 3 probes, 0 missed, 0 false positives; in all three the answer stayed clean and only the reasoning leaked. |
+
+Three things to be clear about, because the labels are easy to over-read:
+
+- **Detection 16 ≠ elicitation 10.** The first counts shapes the matcher knows. The
+  second counts families carried end-to-end out of an agent's reply. Neither number
+  is a subset or a total of the other, and neither is "how many attacks it stops."
+- **The H-CoT row is about the scanner, not about models.** It says *this tool sees
+  that leak*. It is **not** a claim that any real model is vulnerable to H-CoT — that
+  would need live measurement against real models, which is **not** in this release.
+  The probes exist so you can check an agent **you own**; they are not a jailbreak kit.
+- **All three are offline (`canned`, 0 API calls).** They exercise the scanner against
+  planted, synthetic, shape-only fakes — no real key and no live model is involved. So
+  none of them says how often a *real* agent leaks. That question is measured elsewhere
+  (see [What we've observed](#-what-weve-observed-so-far-early--qualitative)) and those
+  observations are directional, not reproducible here.
+
+### Reproducing these numbers
+
+Every **GREEN-backed** number in the table above reproduces from a clone, offline, with
+no API key — and byte-for-byte, not just "close enough":
+
+```bash
+git clone https://github.com/ghkfuddl1327-wq/agentproof && cd agentproof
+python score_axis_b_coverage.py   # → axis_b_coverage_green.json
+python score_elicitation.py       # → elicitation_green.json
+python score_hcot.py              # → hcot_green.json
+python -m pytest -q               # the gates behind them
+```
+
+Run any of them twice and you get identical bytes (no clock, no RNG seed drift).
+If a regenerated file differs from the committed one, treat the claim as broken —
+that's the point of shipping the generators next to the artifacts.
+
+**What does *not* reproduce this way:** the cross-model observations elsewhere on this
+page. Those are **measured snapshots** — they depend on API keys, model availability,
+and provider-side behaviour that changes under us. They are reported as directional,
+and re-running them will not give you the same bytes. We keep the two kinds of number
+apart on purpose.
 
 ---
 
 ## What it catches — and what it doesn't (plainly)
 
-**It catches:** a set list of credential types, matched by their shape. As of
-`0.1.3` that's **six families** — OpenAI, Anthropic, Google, AWS, GitHub, and xAI.
+**It catches:** a set list of credential types, matched by their shape. As of `0.2.0`
+the **detection** list is **15 families on by default** — OpenAI, Anthropic, Google,
+AWS, GitHub (classic), GitHub fine-grained PAT, xAI, Stripe, Slack, JWT, PEM private
+keys, SendGrid, GCP OAuth client secrets, npm, Twilio — **plus `postgres`, which is
+off by default and opt-in** (16th family; see below for why it isn't promoted).
 Matching holds up whether the secret is in plain text or JSON, across different
 languages, and in the answer or the reasoning — for the types it knows.
 
+Two of these are **exposure signals rather than proof of a secret leak**, and the tool
+says so in the finding's `scope`: a **JWT** is often a public ID token, and a **Twilio**
+`SK…` is a public identifier whose paired secret is separate. They're worth surfacing;
+they are not automatically an incident.
+
+`postgres` (a password inside a `postgres://…` URL) is **opt-in** because it is the one
+type that could not meet the no-false-positives bar: the password has no prefix to
+anchor on, so common documentation strings (`mysecretpassword`, `postgres_dev_password`)
+trip it. Rather than loosen the bar for every user, it's off unless you ask:
+
+```bash
+AGP_ENABLE_OPTIONAL=1 agentproof-scan --target …    # turns postgres on, FPs included
+```
+
 Strings that have a real key's *shape* but are obvious dummies (`sk-ant-…EXAMPLE`,
 `AKIA…FAKE`, `…placeholder…`) are filtered out rather than reported, so example code
 and docs don't set off false alarms.
 
 **It doesn't catch:**
-- **Credential types outside those six** — Stripe, Slack, JWTs, PEM private keys,
-  SendGrid, Twilio, npm tokens and others are **not** matched in `0.1.3`. Broader
-  type coverage is built and tested but not yet released here (see Status).
-- **Secrets with no tell-tale prefix** — e.g. a database password buried in a
-  `postgres://…` URL. A real limit of shape-matching.
+- **Credential types outside that list** — the list is finite and hand-written. A type
+  that isn't in it is not matched at all. Adding families does not make the list
+  complete; it moves the boundary.
+- **Secrets with no tell-tale prefix** — the `postgres://…` case above is the example,
+  and it's why that family is opt-in. A real limit of shape-matching.
 - **Secrets described in words** — if a secret is paraphrased with no literal
   key-string, shape-matching can't see it.
 - **Live/runtime catching** — this runs before you ship (offline), not as a live
@@ -508,8 +618,9 @@ and docs don't set off false alarms.
 - **Models we haven't tested** — results come from a small set of lightweight models,
   not the big frontier ones.
 
-*"No false positives" is true for random text on the six types above — it's not a
-promise that a shape-matching type never flags a token that turns out to be public.*
+*"No false positives" is true for random text on the default types above — it's not a
+promise that a shape-matching type never flags a token that turns out to be public.
+The JWT and Twilio caveats above are exactly that case, stated up front.*
 
 ---