atif-scan 0.7.3__tar.gz

atif_scan-0.7.3/.github/dependabot.yml

@@ -0,0 +1,22 @@
+# Added by hf-security-analysis when this repository was created.
+# Actions are pinned to commit SHAs across the organisation; a pinned
+# SHA is only safe while something raises it, and that something is
+# Dependabot. Edit this file freely — the bot writes it once and does
+# not come back.
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    # Releases younger than this are not adopted. The hours right after a
+    # maintainer account is compromised are exactly when a fresh release
+    # is most dangerous — tj-actions and the reviewdog actions both
+    # travelled that way.
+    cooldown:
+      default-days: 7
+    # One pull request a week for all of them, rather than one per action.
+    groups:
+      actions:
+        patterns:
+          - "*"

atif_scan-0.7.3/.github/workflows/ci.yml

@@ -0,0 +1,21 @@
+name: checks
+on: [push, pull_request]
+permissions:
+  contents: read
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python: ['3.11', '3.14']
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+      - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7  # v10.2.0
+        with:
+          python-version: ${{ matrix.python }}
+      - run: uv sync --locked --group dev
+      - run: uv run pytest -q
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+      - run: uv run ty check
+      - run: uv build

atif_scan-0.7.3/.gitignore

@@ -0,0 +1,23 @@
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+dist/
+.env
+*.auth.json
+traces/
+reports/
+# Harbor writes job output to ./jobs when run from here: real traces.
+/jobs/
+.fast-agent/
+
+# Private review output belongs outside Git; these patterns are only a backstop.
+/review/
+/reviews/
+/judge/
+/judges/
+/j[0-9]*/
+/q/
+*.answer.json
+*.review.atif.json

atif_scan-0.7.3/AGENTS.md

@@ -0,0 +1,21 @@
+# atif-scan development
+
+<!-- fast-agent subagents -->
+
+Read README.md and SECURITY.md first. Keep the core (`data`, `checks`/`rules`/`policy`/`engine`, `detectors`, `packs`) stdlib-only
+and offline; `huggingface_hub` and `rich` are imported lazily, only by `sources` (hf://
+inputs) and `output` (`rich`, `brief_view`). `tests/test_architecture.py` enforces the layers.
+Do not execute trajectory commands or URLs. Never commit real traces, auth files,
+raw findings or copied benchmark solutions. Use synthetic fixtures only.
+
+Keep model/parsing, detector predicates, rule logic, evaluation, and report export
+separate. Plugins return typed results. Reports are explicitly allowlisted and
+contain no snippets. Unknown evidence is not a negative result. Severity is review
+priority, not cheating probability. Make task selection explicit.
+
+Run `uv run pytest -q`, `uv run ruff check .`, `uv run ruff format --check .` and
+`uv run ty check` before committing; all four must be clean (tool versions are pinned
+in pyproject.toml). Fix complexity by splitting into named helpers rather than
+suppressing; a `# noqa`/`# ty: ignore` needs a rule code and a reason. Untrusted JSON is
+narrowed with `jsonval`; report documents are `jsonval.Doc`. Add regression tests whenever a
+new false-positive or evidence gap is found. No external model runs are needed.

atif_scan-0.7.3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hugging Face
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

atif_scan-0.7.3/PKG-INFO

@@ -0,0 +1,12 @@
+Metadata-Version: 2.5
+Name: atif-scan
+Version: 0.7.3
+Summary: Composable, evidence-scoped behavioral checks over ATIF trajectories
+Project-URL: Repository, https://github.com/huggingface/atif-scan
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: huggingface-hub>=2.0.0
+Requires-Dist: rich>=15.0.0
+Provides-Extra: mcp
+Requires-Dist: mcp<2,>=1.2; extra == 'mcp'

atif_scan-0.7.3/README.md

@@ -0,0 +1,111 @@
+# atif-scan
+
+Plugin-style detectors and simple rules over ATIF agent trajectories. The analysis core
+is offline, standard-library Python. The CLI adds `huggingface_hub` (only for Hub
+inputs) and `rich` (text output). It never runs anything found in a trace, and never calls a model unless you
+pass `--image-model`.
+
+> Findings are **review candidates, not verdicts**. A severity is a review priority,
+> not a probability of cheating. Unknown evidence is never treated as a clean result.
+
+## Quick start
+
+```bash
+uv sync
+uv run atif-scan examples/synthetic.json            # text on a terminal, JSON when piped
+uv run atif-scan examples/synthetic.json --format json > report.json
+
+# Directories and Hub paths expand to every trajectory.json below them:
+uv run atif-scan /external/jobs/run-1/
+uv run atif-scan hf://buckets/my-org/traces/run-1/
+uv run atif-scan harbor://jobs/<job-id>
+uv run atif-scan https://huggingface.co/buckets/my-org/traces/tree/run-1   # pasted web URL
+
+# Add a task-specific detector pack, rules and allowances:
+PYTHONPATH=examples uv run atif-scan examples/synthetic.json \
+  --task demo-pytest --plugin demo_pack:checks --rules examples/policy.json
+```
+
+For a run, the default text view is the **brief**: one screen covering coverage, score
+(with a "flagged successes failed" scenario), review candidates, cost and walltime. A
+single trace gets the per-check **detail** view. Remote inputs are mirrored privately to
+the atif-scan home (`~/.cache/atif-scan`), so rescans only fetch what changed.
+
+Then, as needed:
+
+```bash
+atif-scan JOB --cite                         # finding rows with masked trace excerpts
+atif-scan JOB --highlights DIR               # excerpts of where the interesting things happened
+atif-scan JOB --browse                       # private localhost evidence browser + feedback
+atif-scan JOB --questions DIR                # review bundle for the flagged successes
+atif-scan hunt --model MODEL --questions DIR # answer it with fast-agent (a provider call)
+atif-scan JOB --answers DIR --brief          # answers as annotations, never rescoring
+atif-inspect TRIAL_DIR --step 12             # read one step without parsing ATIF
+atif-scan JOB --image-model MODEL            # transcribe images that block a check
+```
+
+`atif-scan labels` manages the label store that measures checks and judges. To scan an
+input actually named `labels` or `hunt`, write `./labels` or `./hunt`.
+
+## How it works
+
+```text
+path / dir / hf:// / harbor:// → sources → loader → Trace → detectors ─┐
+                                                                       ├→ engine → report → JSON | text
+                                                 rules, allowances ────┘         (no text snippets)
+```
+
+Components come in two flavours. **Detectors and rules** flag things worth reviewing
+(outside access, test-path reads, odd timestamps). **Allowances** say a finding is
+expected for this trial ("network access is part of this task"), so it is shown as
+`expected` and left out of the score, but not hidden.
+
+By default detectors only see **agent-authored** text: messages, reasoning, and tool-call
+arguments (commands, paths, URLs, queries). Prompts, copied context, tool outputs and
+argument payloads (file contents, edits) are left out unless a check says otherwise.
+
+## Documentation
+
+| Topic | Where |
+|---|---|
+| Inputs, sync, bench-run cohorts, Harbor Hub, manifests, the atif-scan home | [docs/runs.md](docs/runs.md) |
+| Report views (brief, summary, overview, detail), citations, the browser and the static viewer | [docs/reports.md](docs/reports.md) |
+| Built-in detectors, task packs, writing detectors, rules and allowances | [docs/detectors.md](docs/detectors.md) |
+| Review bundles, the questions, blind hunts, answers, `hunt`, `atif-inspect` | [docs/review.md](docs/review.md) |
+| Usage and cost accounting | [docs/usage-accounting.md](docs/usage-accounting.md) |
+| The private findings browser in depth | [docs/private-browser.md](docs/private-browser.md) |
+| Design rules: evidence, unknowns, layering | [docs/design.md](docs/design.md) |
+| Turning judge findings into checks, and measuring both | [docs/improvement-loop.md](docs/improvement-loop.md) |
+| What is safe to share, and what never is | [SECURITY.md](SECURITY.md) |
+
+## Development
+
+```bash
+uv run pytest -q && uv run ruff check . && uv run ruff format --check . && uv run ty check
+```
+
+All four must be clean. Only synthetic fixtures belong in this repository. Add a
+regression test for every false positive or evidence gap you find. [AGENTS.md](AGENTS.md)
+has the working rules and `tests/test_architecture.py` enforces the package layers.
+
+**Golden prompts.** `tests/golden/prompts/` holds the exact text every review question
+puts in front of a judge. After an intended prompt change, bump the question's version
+and regenerate with `UPDATE_GOLDEN=1 uv run pytest tests/test_prompt_golden.py`, then
+review the diff.
+
+**Viewer screenshots.** Before a release, `uv run python tools/viewer_shots.py` exports the
+synthetic example, screenshots it with headless Chrome (desktop and phone, light and dark)
+and fails on a blank page, a page that doesn't open at its masthead, or a theme that didn't
+apply. Screenshots go to the atif-scan home, for a person to look at.
+
+**Gold masters.** Before changing rules, snapshot scans of real runs and diff after:
+
+```bash
+uv run python tools/gold.py snapshot devin-run harbor://jobs/<id>   # before
+uv run python tools/gold.py diff devin-run harbor://jobs/<id>       # after: per-check ± traces, DQ ±
+uv run python tools/gold.py list
+```
+
+Snapshots keep allowlisted fields only (input label, task, reward, check statuses, DQ
+list), in the atif-scan home's `gold/` folder (or `$ATIF_SCAN_GOLD_DIR`). Labels are real
+run identifiers, so never commit them.

atif_scan-0.7.3/SECURITY.md

@@ -0,0 +1,153 @@
+# Security and evidence handling
+
+- Treat trajectories as untrusted and possibly credential-bearing. Keep traces, reports,
+  auth and env files out of Git. Use only synthetic fixtures, and never copy benchmark
+  solutions into this repository.
+- Built-ins never execute commands, visit URLs, run models or fetch files.
+- **`--image-model MODEL`** is the scan's one model call, and it is opt-in. It sends
+  only the images in trials where an image leaves a check unknown (decoded from the
+  trajectory, written to a `0700` temporary folder and deleted afterwards) to MODEL for
+  transcription, through `fast-agent go --isolated --no-shell --no-subagents` with
+  fixed arguments and no shell (see *fast-agent calls* below). Images may show anything
+  on the agent's screen, credentials included, so only use a provider you may send the
+  traces to. Transcripts are trace content: they are stored in the atif-scan home's
+  `images/` folder (`0700`/`0600`) and never reported (reports carry counts). The loader
+  keeps an inline payload's sha256, never the payload.
+- Reports are an explicit field allowlist: no raw text, paths, URLs, argument values or
+  exception messages. The one report field with trace text is opt-in `--cite`: bounded,
+  secret-masked excerpts of the trace. Masking is best-effort, so treat cited output
+  like the trace itself and keep it out of Git and issues. OpenAI account identifiers
+  (`observation.account_*`) are masked the same way; a finding is a scrub reminder, and
+  its absence does not make a trace safe to publish. Text reports render only that allowlisted document. Manifest IDs,
+  check IDs and check titles must be non-sensitive labels.
+- **Review bundles** (`--questions DIR`, formerly `--judge-prompts`) contain trace text
+  (bounded, masked excerpts) and local paths. Treat them like the trace itself: keep them
+  private and outside Git, in the atif-scan home's `bundles/` folder
+  (`~/.cache/atif-scan/bundles/` by default), not the source tree or the expendable
+  result cache. Use `umask 077` when generating or answering (directories `0700`, files
+  `0600`), and a fresh directory per selection and judge model, so stale questions can't
+  mix into a new review. Generating a bundle never sends data anywhere.
+- **Answering** is the user's decision: whoever answers (e.g. `atif-scan hunt`) sends the
+  prompts to a model provider. Prompts frame trace text as untrusted data. `hunt` disables
+  shell and subagents; the optional `--inspect-tool` grants read-only MCP tools bound to
+  one local trajectory, never arbitrary paths or execution. In reports, `--answers` keeps
+  only the validated answer, confidence, mechanism and steps, never the free-text reason.
+  The judge's reason may quote the trace, so it appears only where trace text already
+  does: the static viewer and the private browser, masked as a whole with the trace's
+  own secrets (best effort), beside a current valid answer. Highlights and blind review
+  exports never carry it.
+- **fast-agent calls** (`--image-model`, `hunt`) run `fast-agent go --isolated`
+  (fast-agent 0.10.43+): config, secrets and model aliases are read from the fast-agent
+  home, but nothing is written there (no session history, file logs or telemetry) and no
+  skills, agent cards, plugins, hooks, or shell, filesystem or subagent tools load. Only
+  the MCP server `hunt --inspect-tool` passes is started. An older fast-agent rejects the
+  flag: atif-scan then warns once and runs without it (`--no-shell --no-subagents`,
+  session history off through the environment), where the home's plugins and hooks may
+  still load. Upgrade rather than rely on that.
+- **Companion history.** Full Harbor archives (`--full`) may include Grok compaction
+  segments and task artifacts. Companion-history tools inventory only fixed local
+  session locations beside the bound trajectory and never follow paths supplied by a
+  summary. Numeric file IDs, size limits, symlink rejection and whole-file masking
+  precede bounded reads and searches. Markdown remains untrusted data, not instructions
+  or validated ATIF reconstruction. Availability does not prove completeness. Only Grok
+  Markdown layouts are checked; other formats, including fast-agent JSON snapshots, are
+  unchecked, not absent or uncollected.
+- `atif-inspect` prints masked trace text: treat its output like the trace.
+- When a directory or `hf://` prefix is expanded, each input is labelled by its path
+  *relative to the root you passed* (e.g. `trial-1/agent/trajectory.json`). The root
+  itself never appears. Don't scan roots whose sub-paths are sensitive; use a manifest.
+- Remote inputs are synced by default to the atif-scan home, `~/.cache/atif-scan` (or
+  `$ATIF_SCAN_HOME`, `$XDG_CACHE_HOME/atif-scan`; `$ATIF_SCAN_SYNC_DIR` or `--sync-dir` for
+  the copies alone). The home also holds the label store and gold snapshots, which name
+  real runs: keep all of it private. Those copies are real, possibly
+  credential-bearing traces: keep the directory private, never inside a repository, and
+  delete it when done (`--no-sync` streams without keeping files). The sync layout is
+  confined to that directory (path traversal and symlink destinations are rejected).
+  Owned download directories are made `0700` and files `0600`, including existing copies.
+  Hugging Face mirrors retain a private `.atif-sync.json` inventory (relative paths and
+  content identities, no trace text). Keep it with the copy: it preserves missing inputs
+  on offline rescans. Failed sync files are reported, never silently dropped or replaced
+  with an old copy; any sync failure returns exit 2.
+- The result cache (`--cache`, default `<sync dir>/results`) stores a subset of the
+  allowlisted JSON report (the trace-derived results; run facts are re-read each scan),
+  never citations or trace text.
+- Besides trajectories, the scanner reads only small, size-capped run files: reward files
+  (`verifier/reward.{json,txt}`, at most 4 KiB), Harbor's `result.json`/`config.json`,
+  `trials.jsonl`, and harbor-hf's `run.json` (declared prices) and
+  `attempt-costs/*.json` (at most 4 KiB). Only allowlisted numbers, codes and labels are
+  extracted from them. `--inspect` reads no file contents.
+- A trial's submitted patch (`<trial>/artifacts/model.patch` beside
+  `<trial>/agent/trajectory.json`, as DeepSWE/Pier record it; at most 8 MiB, symlinks
+  refused) is read for its `diff --git` file paths, change kinds and added lines. They
+  stay in memory for fixed predicates (the DeepSWE pack); reports carry only check IDs,
+  counts and trace step locators, never paths, patch lines or patch text. A missing patch
+  is unknown, never an empty submission. Patch text is the agent's code: treat it like
+  the trace.
+- Harbor Hub jobs are read through the user's own `harbor` CLI: no shell, fixed
+  arguments, a validated UUID, stderr withheld. Downloaded trajectories are real traces;
+  `--sync-to` puts them where you choose, otherwise they go to a temporary folder that's
+  removed afterwards.
+- Network access happens only for `hf://` / huggingface.co inputs you name, via
+  `huggingface_hub` and your saved token. Other URLs are refused. Remote error messages
+  (which may contain URLs) are withheld; files over the size cap are rejected.
+- Bench-run discovery reads only capped cohort catalogs, receipts and receipt-pinned
+  configs. It never imports or executes bench-run code, follows auth/config paths, or
+  contacts a provider. Codex diagnostic receipts are recognized by `kind` and a
+  top-level `run_id` only; `plan_path` is not opened. Job selection is confined to the
+  chosen root's `jobs/` directory; paths escaping the root are rejected. Listing exports
+  only run IDs, allowlisted lifecycle statuses, counts and fixed issue codes.
+  Replacement receipts (`runs/replacements/*/receipt.json`) and release manifests are
+  read the same way: capped, digest-checked against the parent run, confined to the
+  root, failing closed when one names the run but doesn't verify. Reports carry only
+  their codes (error types, phases), IDs and trial folder names, never operator `reason`
+  text.
+- For an errored trial only, fast-agent's `fast-agent-results.json` (beside the
+  trajectory, capped) is read for its provider safety details: `provider`, `reason` and
+  `category`, each kept only as an identifier-like code. The provider's explanation and
+  every message in that file are never reported.
+- Plugins and rule files are **trusted** code/configuration, not a sandbox. A plugin can
+  read raw text and do its own I/O. Only load modules you have reviewed. Plugins are
+  never discovered automatically. The only code loaded without `--plugin` is this
+  package's own bundled packs (`atif_scan.packs`), for runs they recognise; `--packs none`
+  disables that.
+- The loader has a size cap, but there is no isolation for JSON depth, regex runtime or
+  plugin CPU/filesystem use. Use a restricted worker for hostile inputs.
+- `--browse` is private inspection, not report export: a token-authenticated IPv4
+  loopback server with strict Host/Origin checks, fixed assets/routes, CSP, no request
+  logging and no-store responses. Never publish or tunnel its port or share its launch
+  URL. Whole-field masking precedes paging/search; text is rendered literally and no
+  trace commands or URLs are executed. Masking is best effort. Local sources are pinned
+  by digest before scanning, checked again before serving, and revalidated on access;
+  changed sources require a new scan. Browser inputs are opaque IDs, not client paths.
+  Finding-level feedback is separate from reports and trial labels, stored privately in
+  `<atif-scan home>/feedback/` (or `--feedback-dir`). Notes are unmasked reviewer text:
+  keep them and the append-only journal out of Git. Feedback is bound to source/task,
+  trace content and assessment identity; it cannot clear coverage gaps. Latest append
+  wins: this is a single-reviewer POSIX tool, not a multi-user adjudication service.
+- `--viewer DIR` is the one output that deliberately contains trace text: a static
+  folder (fixed HTML/CSS/JS plus `data.js`) for publishing an individually reviewed
+  trajectory. Every field is masked as a whole before export, but masking is best
+  effort, so read the export before publishing it; it is not a clearance. The data is
+  allowlisted (labels, task, reward, coverage, check IDs/titles/priorities, field
+  locators, masked field text and, with `--answers`, the judge's masked reasons) and never includes source paths, citations, feedback
+  or raw metadata; media is not exported. By default only non-info findings, all unknown
+  findings and their related whole steps (including judge-cited steps) are exported;
+  summaries and coverage still describe the full scan. `--viewer-full` includes all
+  findings and steps; blind `--review` exports always retain the full chronology.
+  Filtering is not a safety clearance. Highlights are offsets proven against the
+  masked text, otherwise the whole field is marked as the evidence. Sources are pinned
+  by digest before scanning and must still match when exported. The page loads nothing
+  outside its folder (CSP, no fonts, images or network), renders trace text literally
+  and executes nothing. The folder is created `0700` with `0600` files and must be new
+  or empty.
+- `--viewer DIR --review QUESTION` writes a blind review export (same masking, CSP and
+  file rules; no findings, scores, judge answers or scanner run sections). Verdicts stay
+  in the browser's local storage, keyed by a random export ID, and leave only as a JSON
+  file the reviewer downloads. Notes in it are the reviewer's own unmasked text: keep the
+  file private. `atif-scan labels import-review` keeps answer, mechanism and reward only,
+  never the note, and maps opaque export IDs to trials through the private key.
+- A finding doesn't authorize accusation, disqualification or exclusion. Keep the
+  coverage information and get human review.
+
+Report security concerns privately to the repository owner. Don't post sensitive
+trajectories in an issue.

atif_scan-0.7.3/docs/design.md

@@ -0,0 +1,248 @@
+# Design notes
+
+The fine print behind the [README](../README.md). Read this before writing a
+detector or changing the engine.
+
+## Modules
+
+| Module | Role |
+|---|---|
+| `sources.inputs` | Resolves files, directories, `hf://` paths and Hub URLs to loadable inputs (the only input I/O) |
+| `sources.sync` | Mirrors remote inputs into the private sync folder (inventory, content identities, confinement) |
+| `sources.harbor.runs` | Finds the Harbor files next to each trajectory: reward, trial `result.json` + attempt cost, `exception.txt`, job folders, saved Hub listings and ledgers |
+| `sources.harbor.files` | Parses Harbor's own run files (trial `result.json`, job `config.json`/`result.json`, `trials.jsonl`, harbor-hf `run.json` prices and attempt costs) into allowlisted facts |
+| `data.facts` | The one place that decides which record wins for each per-trial fact (reward, task, error, tokens, cost), and the trace-derived facts that are cached |
+| `data.paths` | The atif-scan home (`$ATIF_SCAN_HOME`) and its folders: `hf/`, `harbor/`, `results/`, `labels/`, `gold/`, `bundles/` |
+| `data.jsonval` | Narrows untrusted JSON values (`as_object`, `count`, `number`, …); wrong shapes become unknown, never zero |
+| `sources.harbor.hub` | `harbor://jobs/<id>`: Hub listing (task/reward/cost) and trajectory downloads via the `harbor` CLI |
+| `sources.harbor.listing` | Hub listing rows: validation and allowlisted run/trial facts, shared by live and saved listings (no I/O) |
+| `sources.layout` | `--inspect`: classifies a listing (Harbor markers, roles, anomalies) without reading traces |
+| `data.model`, `data.loader` | Immutable `Trace → Step → ToolCall / Observation` view of ATIF v1, and the parser that builds it |
+| `data.content`, `data.tools`, `data.pairing`, `data.usage` | What the loader reads: a value's text and media; tool categories and argument channels; call/result pairing; recorded usage and retries |
+| `data.submission` | A trial's submitted git patch (DeepSWE `artifacts/model.patch`): file paths, change kinds and added lines, memory only; read beside the trajectory by `sources.harbor.runs.submission_near` into `Context.submission` |
+| `data.jslit`, `data.shell` | Static readers for Codex code-mode programs and shell commands (never executed); unreadable input is unknown or falls back to text patterns |
+| `checks` | The plugin contract: `CheckSpec`, `Context`, `Detection`, `Status`, `Severity` |
+| `detectors` | Built-in families (`lookup`, `awareness`, `integrity`, `tamper`, `installs`, `side_channel`, …), the shared `vocabulary` packs reuse, the `builtin` registry, and the `RegexDetector` / `SurfaceDetector` / `ObservationDetector` helpers |
+| `rules`, `policy` | Three-valued rule expressions, `Allowance`, and the JSON rule/allow format |
+| `engine` | Dependency ordering, task scope, error isolation, applying allowances |
+| `output.document`, `output.text`, `output.summary`, `output.overview` | The JSON allowlist and filtering; plain-text detail/inspect/citation views; `--summary`; the run overview (accuracy, reruns, finding index, DQ scenario, cost, tokens, walltime), all rendered only from the document |
+| `output.rich`, `output.views` | The rich renderer and its optional entry points |
+| `output.brief`, `output.brief_view` (`words`, `colour`, `run`, `evidence`, `usage`), `output.estimates` | One-screen run integrity report (one renderer per section); token accounting, then cost priced once from given, declared or fitted rates; missing-activity estimates |
+| `cache` | Per-trace result cache keyed by file fingerprint + scanner version + check set + context |
+| `evidence.cite` | Opt-in (`--cite`) masked excerpts with before/after context: the only trace-text output |
+| `evidence.history`, `evidence.extract` | Companion history archives; `atif-inspect` (outline, masked step reads, search) |
+| `review.catalogue`, `review.prompts`, `review.answers` | Follow-up and hunt questions: the catalogue (asks, answer sets, triggers, evidence selection), prompt and schema writing (blind mode shows no findings), answers read back and tallied |
+| `review.labels` | The label store: schema, source precedence, run splits, scanner/Jev evaluation (see improvement-loop.md) |
+| `review.inspect_server` | The read-only MCP server over one trajectory behind `hunt --inspect-tool` (optional `mcp` extra) |
+| `output.bundle` | `--questions`: the review selection (scope, questions, blind mode) written as a question bundle |
+| `cli` | The command line: `args` (parser and option checks), `inputs` (manifests, paths, sync, tasks, plugins), `scan` (cached evaluation, the report document, review bundles), `emit` (brief/overview/summary/detail as text or JSON), `inspect` (`--inspect`), the `labels` and `hunt` commands; `main` (dispatch) and `--submission` in the package |
+
+Keep these separate: parsing doesn't know about detectors, detectors don't know about
+rules, and only `output.document.report` decides what gets written out.
+
+`tests/test_architecture.py` enforces the layering on every runtime import: no cycles;
+the `data` package (`model`, `loader`, `content`, `tools`, `pairing`, `usage`, `jsonval`,
+`shell`, `jslit`, `credentials`, `facts`, `accounting`, `paths`, `web_*`) imports only itself; `sources` only data and itself; analysis
+(`checks`, `rules`, `policy`, `engine`, `access`, `detectors`, `packs`) only data and
+itself.
+
+## Input scope
+
+- Accepts `ATIF-v1.x` (or no version). Other major versions are rejected. The loader
+  checks the structures it uses, ignores unrelated extensions, and doesn't claim full
+  schema validation. Files over 256 MiB are rejected.
+- The `Trace` is an analysis view, not a lossless copy. Top-level metadata and usage
+  stats are dropped.
+- Text fields become `Content`. Formats it can't read get `understood=False`, and
+  detectors report `unknown` for them instead of `no_match`. Images and other binary
+  blocks are excluded from text checks. That doesn't mean they contain nothing relevant.
+  An inline payload is kept only as its sha256 (`Content.media_ids`), so the opt-in
+  `--image-model` answers (`Context.images`) can be matched to it.
+- Every call's arguments go through `tools.classify`. It walks string leaves together
+  with their nearest key and routes each one by key convention and value shape to
+  `PAYLOAD`, `URL`, `COMMAND`, `QUERY`, `PATH` or `ARGUMENTS`, in that order of
+  precedence (see the tool coverage table in detectors.md). Key lists are generic argument conventions, not
+  harness tool names. Extend them with a regression test when a real trace shows a gap.
+- Tool names are *hints* (`tools.TOOLS` → `shell`, `read`, `write`, `search_files`,
+  `web_fetch`, `web_search`, `inert`, `other`). They supply:
+  - meaning (only a known `web_search` tool counts as a web search);
+  - required inputs (a known shell call without a command is `unknown`);
+  - `inert`: orchestration tools whose arguments are ignored, e.g. `ToolSearch`, whose
+    `query` looks up tools, not the web.
+- `SurfaceDetector` is incomplete when a surface on its channels isn't understood, when
+  any call's arguments are unparseable (for tool-input checks), or when its
+  `undecidable(surface)` hook says the predicate needs tool semantics it doesn't have.
+  An unrecognized tool name alone never reduces coverage. Original `name` and deeply
+  immutable `arguments` stay available to plugins (`Surface.tool_name` too).
+- Step metadata (`step_id`, parsed `timestamp`, whether each was recorded), agent-only
+  fields on non-agent steps, and `final_metrics.extra.total_tool_use_tokens` are
+  retained for integrity checks.
+- `Trace.agent_surfaces()` yields agent prose and classified tool-argument surfaces
+  (including `PAYLOAD`, which built-ins don't read).
+  `Trace.observation_surfaces()` is a separate, opt-in API for tool outputs.
+- Content-bearing fields are hidden from `repr`. That prevents accidental logging, but
+  doesn't stop deliberate logging.
+
+## Evidence semantics
+
+A detector returns `Detection(status, evidence, complete)`:
+
+- `match` means the predicate matched recorded text. It can still be `complete=False`.
+- `no_match` means the detector's whole declared scope was readable and nothing
+  matched. It must be `complete=True`.
+- `unknown` and `error` must be `complete=False`.
+
+`Detection.of(hits, complete)` builds the usual result: `match` on any hit (deduplicated,
+order kept), else `no_match` when complete, else `unknown`. `Step.results_for(call)` is
+the one call → result link: observations whose `source_call_id` is the call's result key,
+or, when none is and the step has exactly one call, its unlinked observations.
+
+Engine-level rules:
+
+- A task-scoped check with no task is `unknown`. With a different task it's
+  `not_applicable`.
+- A missing trace makes every check `unknown`.
+- `partial` inputs keep their matches (marked incomplete). Their `no_match` results
+  become `unknown`. A trace with recording gaps (`Trace.recording_gaps`: compacted
+  history, status-only results, claimed but unrecorded actions) is evaluated as partial
+  by `Engine.evaluate` itself (`engine.effective_context`), so library callers get the
+  same result as the CLI.
+- Exceptions, non-`Detection` return values, and evidence pointing outside the trace
+  become `error`. The exception text is never kept.
+
+Every negative is limited to what was observed. If no reasoning was recorded, that
+doesn't prove the agent had no benchmark awareness. If no URL was matched, that doesn't
+prove there was no network access.
+
+## Tool results and integrity
+
+`ObservationDetector` scans recorded tool results. It is `unknown` when the agent
+made calls but no results were recorded. A result is evidence of what the agent
+*received*, not what it authored or used.
+
+Integrity checks (`detectors/integrity.py`) go beyond Harbor's validator, which
+checks schema, sequential `step_id`, call links and timestamp *syntax*, but not
+ordering, smearing or telemetry plausibility. Scopes:
+
+- Timing checks use recorded timestamps on non-copied steps. They are `unknown` only
+  when too few timestamps exist to evaluate. Missing agent-step timestamps are
+  reported positively by `integrity.timestamp_missing`.
+- `integrity.step_sequence` is `unknown` when no step carries a `step_id`.
+- `integrity.tool_token_telemetry` concerns *reported* values; not reporting is
+  not implausible.
+
+## Rules
+
+`Ref`, `All`, `AnyOf`, `Not` and `Requires` use strong Kleene logic: `error` and
+`not_applicable` inputs count as unknown. `All(false, unknown)` is false and
+`AnyOf(true, unknown)` is true. `Requires(A, B)` is `All(A, Not(B))`. It reports a
+**violation**, not a passing implication.
+
+A matched rule cites the evidence of its matched dependencies. When the consequent is
+false there is nothing to cite, so the dependency's `no_match` is the basis instead.
+
+Rules are per-trace co-occurrence checks. They don't encode time order or causation, and
+there are no cross-trial statistics. Write that kind of logic as a Python detector that
+returns the same `Detection` type.
+
+## Context checks and rewards
+
+A `ContextCheck` (kind `context`) states a fact about the run, e.g. `context.rewarded`.
+It is evaluated before rules, so rules and allowances can reference it. It never counts
+towards `score`/`--fail-on`. An unknown fact doesn't make a report incomplete by
+itself; a rule that depends on it is unknown instead.
+
+The Terminal-Bench judge only reviews rewarded trials, and absence rules like "rewarded,
+but no SSH setup" need the reward. `sources` finds it without an extra listing:
+`verifier/reward.json`, then `reward.txt` (Harbor's order), in the trajectory's folder or
+one level up, read only when present in the listing and capped at 4 KiB. Anything that
+isn't a finite number is unknown, never zero.
+
+## LLM judges and classifiers
+
+Some judgments (e.g. "fabricated an answer after abandoning real work", intent language
+that matched 121 of 441 real traces) need a model. Write them as plugins: a detector that
+returns the usual `Detection` with evidence locators. Keep them out of the core:
+
+- Pre-filter with cheap detectors and rules, and judge only candidate surfaces or
+  rewarded trials. That's the Terminal-Bench judge's policy too.
+- Network/model use is the plugin's explicit, trusted I/O. Report only the verdict and
+  locators, never model text. Map refusals, timeouts and unparseable verdicts to
+  `unknown`.
+- Pin the model/prompt in the check `version`.
+
+## Allowances
+
+An `Allowance` is evaluated after every detector and rule. It has a task scope, a set of
+covered check IDs and an optional `when` expression. It applies only when its status is
+`match`: unconditional within scope, or `when` is true. Then each covered assessment
+that matched gets `expected_by`, and `Assessment.counts` becomes false. That single
+predicate drives `score`, `severity` and `--fail-on`.
+
+- An allowance never rewrites a `Detection`. "Expected" is a disposition, not a new status.
+- `unknown` allowances (unknown `when`, missing task, partial trace with an unmet
+  condition) don't apply, and they make the report `incomplete`.
+- Rules evaluate raw results, so excusing a fact doesn't silently disarm rules built on it.
+  This keeps evaluation single-pass and explicit: cover the rule ID too if intended.
+- Allowances can't be referenced by rules or other allowances.
+
+## Report boundary
+
+`report.report` builds its output field by field and never serializes trace objects or
+plugin data. Evidence is `(step, channel, call, observation, field, span)`: **array
+positions** (not ATIF step IDs), the index of the classified call argument, and the
+character span of the match when the detector knows it. A `SurfaceDetector` predicate may
+return a `re.Match` or `(start, end)` to supply the span. `cite` turns these into
+excerpts only when asked; renderers never re-open the trace. Check IDs, versions and manifest IDs must be static, non-sensitive
+identifiers. `CheckSpec.title` is a static, non-sensitive label too (at most 60 printable
+characters, one line, no URL); `report.document` lists every check the engine ran in a
+top-level `checks` catalog (`{id: {severity, title}}`) built from `Engine.catalog()` on each
+scan, never from cached trace results. Built-in titles say what was observed, in sentence
+case, at most 48 characters, without severity words or hedging.
+
+Built-ins are deterministic: no clock, randomness, network or filesystem access. Bump a
+detector's `version` when its meaning changes. Bump the report `schema_version` for
+incompatible output changes.
+
+## Writing and testing a detector
+
+- State what the predicate actually shows: text signature, attempted action, observed
+  tool result, or verified outcome. Keep these distinct. A `pytest` command is not
+  evidence that the tests passed.
+- Return `unknown` when the evidence you need is missing or unreadable, and say where:
+  put the positions you could judge in `evidence` and the context you couldn't read in
+  `Detection.unread` (a fixed reason code plus its locator: `media`, `unreadable`,
+  `compacted`, `web_result_not_recorded`, `prompt_not_recorded`; for a call with no
+  result `run_ended`, `result_compacted` or `result_not_recorded`; for a recorded result that
+  says it's partial `result_truncated` or `terminal_screen_only` (Terminus 2 captures, see
+  `data/terminal.py`); `undecidable` when
+  the predicate can't judge without running it; `prompt_names_benchmark` when the prompt
+  already says what the evidence says, so a hit can't be told apart from the task). `Detection.of(hits, complete, unread)`
+  keeps them only for incomplete results; reports list them as `unread`. If unread context can only *remove* a candidate (priming), a
+  negative that holds even without it is a real `no_match`.
+- A trace-wide check with no single location should say what it compared: put its
+  figures in `Detection.measure` (identifier names with numbers, None or short codes;
+  validated, never text). Reports list them as `measure`; the viewer shows the sum.
+- Never put runtime data into IDs or evidence.
+- Patterns run over megabytes of tool output and minified code. Bound every gap
+  between two literals (`\S{0,256}?`, `[^\n]{0,300}`: document the bound), split
+  `A.*B` into a search for `A` then for `B` from its end, and never lead with `^\s*`
+  under `re.M` (use `^[^\S\n]*`). A literal prefilter (`detectors.text.gated`) is only
+  exact when every alternative contains a hint; test that per alternative. Add a timing
+  guard for anything that was superlinear.
+
+Checklist for tests:
+
+- A positive match and a near-miss negative, across tool aliases.
+- System/user text, copied context and tool outputs don't leak into authored-text checks.
+- Ordinary look-alikes: local tests, perf benchmarks, package installs.
+- Missing trace, malformed arguments, missing task, partial trace.
+- Secrets, signed URLs and multiline commands never appear in the report.
+- Unknown dependencies, cycles and `requires` behave as described above.
+
+## Known limits of the built-in pack
+
+The built-ins are exploratory. Shell comments, heredocs, and program source embedded in
+a command can match. Quoted prose can echo task instructions. Dynamic URLs, hosted
+(native) search and unconventional tool names can be missed. Improve them with versioned
+tests, not by tuning scores after the fact.