extract-cli 0.1.5__tar.gz → 0.1.8__tar.gz

extract_cli-0.1.8/AGENTS.md

@@ -0,0 +1,87 @@
+# Agents
+
+Drive `extract-cli` from an LLM agent or non-interactive client. Same agent
+contract as the rest of the contract-ops suite: a stable machine-readable
+catalog, JSON on stdout, humans on stderr, and a small documented exit-code set.
+
+`extract-cli` is the suite's **open-loop front door**: hand it any contract
+(`.md` / `.txt` / `.html` / `.docx` / `.pdf`, yours or a counterparty's) and it
+returns structured JSON the rest of the pipeline can consume. Every field
+carries a `confidence` and a `source` — **verify, don't trust**.
+
+## Output contract
+
+- **Success**: a single JSON object to **stdout**, exit `0`. This is the machine
+  payload; it's the default (no `--json` needed, though `--json` forces it).
+- Every extracted scalar is the envelope `{value, confidence, source}`;
+  "not found" is the canonical `{value: null, confidence: 0.0, source: "none"}`.
+  Lists (`parties`, `clauses`, `defined_terms`) carry per-item
+  `confidence`/`source`. `source ∈ {deterministic, llm, none}`.
+- `_meta` records `extractor_version`, `tiers_used`, and `llm_used`.
+- The output shape is locked by a JSON Schema —
+  [`docs/spec/extract-output.schema.json`](docs/spec/extract-output.schema.json),
+  also printed by `extract schema`. Validate against it instead of trusting
+  field shapes by convention. (Note: the `--no-confidence` projection is a
+  reduced convenience view, **not** governed by the schema.)
+- **stderr** is for humans only: `--why` rationale, warnings, and errors.
+  stdout stays clean JSON even under `--why`.
+- **Failure**: a one-line `error: <message>` on **stderr**, non-zero exit.
+  The error shape is a flat string (the suite is not uniform on error-object
+  shape) — **branch on the exit code, never on the human-readable message.**
+
+## Exit codes
+
+| Code | Meaning |
+|------|---------|
+| `0` | Success. |
+| `1` | Low-signal document — no high-signal fields (parties/clauses/dates) could be extracted; e.g. a scanned/image-only or empty file. A **finding**, not a crash: valid JSON is still emitted on stdout. |
+| `2` | Bad usage / user-actionable error (unreadable path, bad flag value, unsupported completion shell). |
+
+## Discovery
+
+Never hardcode command or flag names — call the catalog at startup:
+
+```bash
+extract --catalog json    # {name, bin, version, description, commands[], exitCodes}
+```
+
+`--catalog json` is the suite-wide discovery contract (parallel to
+`nda-review-cli --catalog json`, `docx2pdf --catalog json`,
+`sign --catalog json`). It is **complete, accurate, and stable across minor
+versions** — a test asserts it never drifts from the real parser.
+
+Tool-specific discovery extras:
+
+```bash
+extract schema            # the output JSON Schema (the cross-CLI data contract)
+extract fields            # extractable fields and the tier that produces each
+extract fields --json     # ...as JSON
+extract demo              # run on a bundled fixture (zero-config first run)
+extract --version
+```
+
+## Failure → recovery
+
+| Symptom | Diagnose | Recover |
+|---|---|---|
+| Exit `1`, warning "no high-signal fields" | The document is likely scanned/image-only or has no recognizable structure. JSON is still emitted. | OCR the source first, or feed a text/`.docx`/`.md` version. The empty-but-valid JSON is safe to pass downstream. |
+| Exit `2`, `error: ...` | `extract --catalog json` (or `extract <cmd> --help`) for the real surface. | Fix the path/flag and retry. |
+| `clauses: []` on a real contract | The `.docx` likely auto-numbers via Word's numbering with no heading style (its numbers live only in `numbering.xml`), so the deterministic cascade sees no headings. | Re-run with `--llm` (opt-in): when no clauses are detected, the LLM is asked for section headings, normalized through the same canonical vocabulary and emitted with `tier: "llm"`, `source: "llm"`, and a modest confidence. Requires `~/.config/contract-ops/llm.json`. |
+| Low-fidelity `.docx`/`.pdf` text | The stdlib best-effort reader ran (no extras installed). | `pip install "extract-cli[docx]"` and/or `"extract-cli[pdf]"` for higher fidelity. The core always works without them. |
+| `--llm` only printed a warning | No LLM config found. | Copy [`config/llm.json.example`](config/llm.json.example) to `~/.config/contract-ops/llm.json`. Without it, deterministic output is still returned in full. |
+
+## Recommended usage
+
+```bash
+# Inspect any contract's structure, one tool for five formats.
+extract counterparty.docx | jq '{parties: [.parties[].name],
+  governing_law: .governing_law.value, clauses: [.clauses[].canonical_title]}'
+
+# Gate a workflow on extraction confidence (non-zero exit if any clause is shaky).
+extract draft.docx | jq -e '.clauses | all(.confidence > 0.7)' && echo ok
+```
+
+The integration contract is the **output schema** + the **shared canonical
+clause vocabulary** (`canonical_title` values match what `template-vault-cli`
+detects and `nda-review-cli` keys policy on) — not per-tool flags. See
+[`docs/INTEROP.md`](docs/INTEROP.md).

{extract_cli-0.1.5 → extract_cli-0.1.8}/CHANGELOG.md

@@ -6,6 +6,69 @@ to [Semantic Versioning](https://semver.org/). Per the suite convention
 (see [`docs/INTEROP.md`](docs/INTEROP.md)), **backward-incompatible changes to
 the output schema require a major version bump**; new optional fields are minor.
 
+## [0.1.8] - 2026-05-22
+
+Clause-detection breadth, driven by a 58-document real-corpus survey.
+
+### Added
+- **Auto-numbered DOCX clauses.** The DOCX reader now treats `w:numPr` list
+  paragraphs (no heading style; number generated from `numbering.xml`) as
+  clause-heading candidates, run through the same run-in/heading-likeness filter
+  as heading styles. Real agreements that number clauses this way (data
+  processing / design-partner agreements) get a clause map where they previously
+  got none; deep numbered body sentences are still excluded. New `numbered_docx`
+  fixture + tests.
+- **Two-line `ARTICLE N` headings.** A bare `ARTICLE N` / `SECTION N` line whose
+  title sits on the next line (common in formal agreements) is detected as a
+  pair — recovering, e.g., a real SEC services agreement's clause map (0 → 8).
+  Fires only with >= 2 well-formed pairs; reported under the `numbered` tier (no
+  schema change).
+- **Expanded canonical clause vocabulary** from the corpus survey: new canonical
+  clauses `Exclusions`, `Remedies`, `Restrictions`, `Taxes`,
+  `Reservation of Rights`, `Third-Party Beneficiaries`, `Feedback`,
+  `Miscellaneous`, plus aliases for `Compliance with Laws` (anti-bribery, export
+  controls) and `Data Protection` (customer data/content). ~155 more clauses map
+  across the corpus, with no observed over-matching.
+- **`CLAUDE.md`** — codebase development notes (complements AGENTS.md).
+
+No output-schema change.
+
+## [0.1.7] - 2026-05-22
+
+### Added
+- **`extract --catalog json` — the suite's shared discovery contract.** Emits
+  `{name, bin, version, description, commands[], exitCodes}` (mirroring
+  `nda-review-cli --catalog json` / `docx2pdf --catalog json` /
+  `sign --catalog json`) so agents can learn every command and flag at startup
+  instead of hardcoding them. A test asserts the catalog never drifts from the
+  real argparse parser. Also added to the bash/zsh completion flag lists.
+- **`AGENTS.md`** — the agent contract in the suite's canonical section order
+  (output contract / exit codes / discovery / failure → recovery).
+- **`llms.txt`** — machine-readable tool summary at the repo root.
+
+### Changed
+- Packaging: added the suite-standard keywords (`contract-ops`, `agent-first`,
+  `legal-tech`); README now opens with `## Run this` / `## Where to go next`;
+  `--catalog json` documented in the README and `docs/INTEROP.md`. No schema or
+  extraction-logic change (`extractor_version` unchanged).
+
+## [0.1.6] - 2026-05-21
+
+### Docs
+- **Rewrote the README composability section to verified, runnable examples.**
+  Testing extract-cli against the real sibling CLIs (`template-vault-cli`,
+  `nda-review-cli`) showed the previous pipes were aspirational — the siblings
+  expose no `--from-extract`/`--stdin` flag (`nda-review review` takes
+  `--file`/`--text`; `template-vault` reads its own vault). The integration
+  contract is the **output schema + the shared canonical clause vocabulary**,
+  glued by stdout JSON and standard tools (`jq`, `comm`): `extract`'s
+  `canonical_title` values are the same names template-vault detects and
+  nda-review keys policy on, so a foreign document's clauses line up with the
+  suite's with no bespoke adapter. New examples cover clause-coverage gap
+  analysis against a vault template and a combined extract+nda-review intake
+  report — all runnable today. (Also fixed a broken `jq input_filename` in the
+  folder-triage example.) No code or schema change.
+
 ## [0.1.5] - 2026-05-21
 
 ### Added
@@ -181,6 +244,9 @@ Initial release — the open-loop front door of the contract-ops CLI suite.
   intentionally *not* governed by the output schema (the schema describes the
   full default output).
 
+[0.1.8]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.8
+[0.1.7]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.7
+[0.1.6]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.6
 [0.1.5]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.5
 [0.1.4]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.4
 [0.1.3]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.3

{extract_cli-0.1.5 → extract_cli-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: extract-cli
-Version: 0.1.5
+Version: 0.1.8
 Summary: Open-loop front door of the contract-ops CLI suite: ingest any contract (.md/.txt/.html/.docx/.pdf) and emit structured JSON.
 Project-URL: Homepage, https://cli.drbaher.com/
 Project-URL: Repository, https://github.com/DrBaher/extract-cli
@@ -8,7 +8,7 @@ Project-URL: Suite interop, https://github.com/DrBaher/extract-cli/blob/main/doc
 Author-email: DrBaher <Drbaher@gmail.com>
 License: MIT
 License-File: LICENSE
-Keywords: clause,cli,contract,extraction,json,legal,nda
+Keywords: agent-first,clause,cli,contract,contract-ops,extraction,json,legal,legal-tech,nda
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
@@ -61,6 +61,30 @@ ingest (extract) → review → diff → convert → sign
    ^you are here
 ```
 
+## Run this
+
+```bash
+pipx run extract-cli demo        # zero-config: extract a bundled NDA → structured JSON
+# or, installed:  pip install extract-cli && extract demo
+```
+
+That prints the full output contract — parties, dates, term, governing law, and
+a clause map normalized onto the suite's canonical vocabulary — for a bundled
+fixture, with no setup and no network. Point it at your own file with
+`extract path/to/contract.docx`.
+
+## Where to go next
+
+- **New here?** Keep reading — [What it does](#what-it-does) and
+  [The two extraction tiers](#the-two-extraction-tiers).
+- **Driving it from an agent?** See [`AGENTS.md`](AGENTS.md) and call
+  `extract --catalog json` at startup to discover commands/flags. The output
+  shape is locked by [`docs/spec/extract-output.schema.json`](docs/spec/extract-output.schema.json).
+- **Wiring it into the pipeline?** See [`docs/INTEROP.md`](docs/INTEROP.md) — the
+  contract is the output schema + the shared clause vocabulary.
+- **Contributing / building a sibling CLI?** [`CONTRIBUTING.md`](CONTRIBUTING.md)
+  and [ARCHITECTURE.md](ARCHITECTURE.md).
+
 ## What it does
 
 Give it a contract in **`.md` / `.txt` / `.html`** (native), **`.docx`**, or
@@ -115,6 +139,7 @@ for them.
 
 ```bash
 extract <path>            # parse a document → structured JSON on stdout (default)
+extract --catalog json    # machine-readable catalog of commands/flags (agents call at startup)
 extract schema            # print the output JSON Schema (the cross-CLI contract)
 extract fields            # list extractable fields and their tier
 extract demo              # run on a bundled fixture and show the narrative
@@ -125,6 +150,7 @@ extract completion bash   # emit a shell-completion script (bash|zsh)
 
 | Flag | Meaning |
 |---|---|
+| `--catalog json` | Print the machine-readable command/flag catalog and exit (the suite discovery contract; agents call this at startup) |
 | `--llm` | Opt-in LLM enrichment of fuzzy fields (off by default) |
 | `--fields a,b,c` | Emit only a subset of top-level fields (e.g. `parties,clauses`) |
 | `--format json\|table` | Output format (default `json`) |
@@ -171,37 +197,48 @@ extract counterparty.pdf | jq '.clauses[] | {canonical_title, detected_title, ma
 
 ## Composability — piping into the rest of the suite
 
-`extract-cli` is built to be the first stage of a Unix pipe. Its JSON is the
-contract every downstream tool reads.
+`extract-cli` is built to be the first stage of a Unix pipe. The glue is its
+**stdout JSON + standard tools** (`jq`, `comm`) and the **shared clause
+vocabulary** — `extract`'s `canonical_title` values are the same names
+`template-vault-cli` detects and `nda-review-cli` keys policy on, so a foreign
+document's clauses line up with the suite's with no bespoke adapter. Every
+example below is runnable today (verified against the real sibling CLIs).
 
 ```bash
-# 1) Foreign NDA → review. extract normalizes clauses; nda-review runs policy.
-extract counterparty_nda.pdf | nda-review review --from-extract -
-
-# 2) Pull just the clause map and feed compare-cli to diff a foreign doc
-#    against your canonical template's structure.
-extract their_msa.docx --fields clauses | compare-cli align --stdin \
-  --against msa/standard
-
-# 3) Archive structured metadata for any inbound paper into the post-signature
-#    vault, keyed by content hash.
-extract signed_contract.pdf | contract-vault put --from-extract - \
-  --id "$(extract signed_contract.pdf | jq -r .document.sha256)"
-
-# 4) Triage a folder of inbound contracts: list governing law + parties.
-for f in inbox/*.pdf; do
+# 1) Inspect any contract's structure (.md/.txt/.html/.docx/.pdf, one tool).
+extract counterparty.docx | jq '{parties: [.parties[].name],
+  governing_law: .governing_law.value, clauses: [.clauses[].canonical_title]}'
+
+# 2) Clause-coverage gap vs your canonical template in template-vault-cli.
+#    extract normalizes the counterparty's *foreign* headings onto the same
+#    clause vocabulary template-vault detects, so a plain `comm` diffs them.
+template-vault info nda/mutual-standard --json | jq -r '.clauses[].title' | sort > ours.txt
+extract counterparty_nda.docx | jq -r '.clauses[].canonical_title' | sort -u > theirs.txt
+comm -23 ours.txt theirs.txt    # clauses in OUR standard that THEY are missing
+comm -13 ours.txt theirs.txt    # clauses THEY added that we don't have
+
+# 3) Intake: extract for structure, nda-review-cli for a policy verdict on the
+#    same foreign doc; merge both views with jq.
+extract counterparty_nda.docx > extract.json
+nda-review review --file counterparty_nda.docx --playbook output/nda_playbook.json \
+  --out-json review.json
+jq -n --slurpfile e extract.json --slurpfile r review.json \
+  '{parties: [$e[0].parties[].name], governing_law: $e[0].governing_law.value,
+    clauses: ($e[0].clauses | length), decision: $r[0].decision, risk: $r[0].risk_score}'
+
+# 4) Triage a folder of inbound contracts: governing law + parties per file.
+for f in inbox/*; do
   extract "$f" --fields parties,governing_law --no-confidence \
-    | jq -c '{file: input_filename, gov: .governing_law, parties: [.parties[].name]}'
+    | jq -c --arg f "$f" '{file: $f, gov: .governing_law, parties: [.parties[].name]}'
 done
 
-# 5) Gate a workflow on extraction confidence.
+# 5) Gate a workflow on extraction confidence (non-zero exit if any clause is shaky).
 extract draft.docx | jq -e '.clauses | all(.confidence > 0.7)' && echo "ok to review"
 ```
 
-> The `--from-extract`/`--stdin` flags above are the consumption points the
-> sibling CLIs expose (or are adopting) for this contract; see
-> [`docs/INTEROP.md`](docs/INTEROP.md) for the shared conventions and the
-> versioning commitment on the schema.
+> The integration contract is the **output schema** and the **canonical clause
+> vocabulary**, not per-tool flags. See [`docs/INTEROP.md`](docs/INTEROP.md) for
+> the shared conventions and the schema's versioning commitment.
 
 ## LLM configuration (opt-in)
 

{extract_cli-0.1.5 → extract_cli-0.1.8}/README.md

@@ -23,6 +23,30 @@ ingest (extract) → review → diff → convert → sign
    ^you are here
 ```
 
+## Run this
+
+```bash
+pipx run extract-cli demo        # zero-config: extract a bundled NDA → structured JSON
+# or, installed:  pip install extract-cli && extract demo
+```
+
+That prints the full output contract — parties, dates, term, governing law, and
+a clause map normalized onto the suite's canonical vocabulary — for a bundled
+fixture, with no setup and no network. Point it at your own file with
+`extract path/to/contract.docx`.
+
+## Where to go next
+
+- **New here?** Keep reading — [What it does](#what-it-does) and
+  [The two extraction tiers](#the-two-extraction-tiers).
+- **Driving it from an agent?** See [`AGENTS.md`](AGENTS.md) and call
+  `extract --catalog json` at startup to discover commands/flags. The output
+  shape is locked by [`docs/spec/extract-output.schema.json`](docs/spec/extract-output.schema.json).
+- **Wiring it into the pipeline?** See [`docs/INTEROP.md`](docs/INTEROP.md) — the
+  contract is the output schema + the shared clause vocabulary.
+- **Contributing / building a sibling CLI?** [`CONTRIBUTING.md`](CONTRIBUTING.md)
+  and [ARCHITECTURE.md](ARCHITECTURE.md).
+
 ## What it does
 
 Give it a contract in **`.md` / `.txt` / `.html`** (native), **`.docx`**, or
@@ -77,6 +101,7 @@ for them.
 
 ```bash
 extract <path>            # parse a document → structured JSON on stdout (default)
+extract --catalog json    # machine-readable catalog of commands/flags (agents call at startup)
 extract schema            # print the output JSON Schema (the cross-CLI contract)
 extract fields            # list extractable fields and their tier
 extract demo              # run on a bundled fixture and show the narrative
@@ -87,6 +112,7 @@ extract completion bash   # emit a shell-completion script (bash|zsh)
 
 | Flag | Meaning |
 |---|---|
+| `--catalog json` | Print the machine-readable command/flag catalog and exit (the suite discovery contract; agents call this at startup) |
 | `--llm` | Opt-in LLM enrichment of fuzzy fields (off by default) |
 | `--fields a,b,c` | Emit only a subset of top-level fields (e.g. `parties,clauses`) |
 | `--format json\|table` | Output format (default `json`) |
@@ -133,37 +159,48 @@ extract counterparty.pdf | jq '.clauses[] | {canonical_title, detected_title, ma
 
 ## Composability — piping into the rest of the suite
 
-`extract-cli` is built to be the first stage of a Unix pipe. Its JSON is the
-contract every downstream tool reads.
+`extract-cli` is built to be the first stage of a Unix pipe. The glue is its
+**stdout JSON + standard tools** (`jq`, `comm`) and the **shared clause
+vocabulary** — `extract`'s `canonical_title` values are the same names
+`template-vault-cli` detects and `nda-review-cli` keys policy on, so a foreign
+document's clauses line up with the suite's with no bespoke adapter. Every
+example below is runnable today (verified against the real sibling CLIs).
 
 ```bash
-# 1) Foreign NDA → review. extract normalizes clauses; nda-review runs policy.
-extract counterparty_nda.pdf | nda-review review --from-extract -
-
-# 2) Pull just the clause map and feed compare-cli to diff a foreign doc
-#    against your canonical template's structure.
-extract their_msa.docx --fields clauses | compare-cli align --stdin \
-  --against msa/standard
-
-# 3) Archive structured metadata for any inbound paper into the post-signature
-#    vault, keyed by content hash.
-extract signed_contract.pdf | contract-vault put --from-extract - \
-  --id "$(extract signed_contract.pdf | jq -r .document.sha256)"
-
-# 4) Triage a folder of inbound contracts: list governing law + parties.
-for f in inbox/*.pdf; do
+# 1) Inspect any contract's structure (.md/.txt/.html/.docx/.pdf, one tool).
+extract counterparty.docx | jq '{parties: [.parties[].name],
+  governing_law: .governing_law.value, clauses: [.clauses[].canonical_title]}'
+
+# 2) Clause-coverage gap vs your canonical template in template-vault-cli.
+#    extract normalizes the counterparty's *foreign* headings onto the same
+#    clause vocabulary template-vault detects, so a plain `comm` diffs them.
+template-vault info nda/mutual-standard --json | jq -r '.clauses[].title' | sort > ours.txt
+extract counterparty_nda.docx | jq -r '.clauses[].canonical_title' | sort -u > theirs.txt
+comm -23 ours.txt theirs.txt    # clauses in OUR standard that THEY are missing
+comm -13 ours.txt theirs.txt    # clauses THEY added that we don't have
+
+# 3) Intake: extract for structure, nda-review-cli for a policy verdict on the
+#    same foreign doc; merge both views with jq.
+extract counterparty_nda.docx > extract.json
+nda-review review --file counterparty_nda.docx --playbook output/nda_playbook.json \
+  --out-json review.json
+jq -n --slurpfile e extract.json --slurpfile r review.json \
+  '{parties: [$e[0].parties[].name], governing_law: $e[0].governing_law.value,
+    clauses: ($e[0].clauses | length), decision: $r[0].decision, risk: $r[0].risk_score}'
+
+# 4) Triage a folder of inbound contracts: governing law + parties per file.
+for f in inbox/*; do
   extract "$f" --fields parties,governing_law --no-confidence \
-    | jq -c '{file: input_filename, gov: .governing_law, parties: [.parties[].name]}'
+    | jq -c --arg f "$f" '{file: $f, gov: .governing_law, parties: [.parties[].name]}'
 done
 
-# 5) Gate a workflow on extraction confidence.
+# 5) Gate a workflow on extraction confidence (non-zero exit if any clause is shaky).
 extract draft.docx | jq -e '.clauses | all(.confidence > 0.7)' && echo "ok to review"
 ```
 
-> The `--from-extract`/`--stdin` flags above are the consumption points the
-> sibling CLIs expose (or are adopting) for this contract; see
-> [`docs/INTEROP.md`](docs/INTEROP.md) for the shared conventions and the
-> versioning commitment on the schema.
+> The integration contract is the **output schema** and the **canonical clause
+> vocabulary**, not per-tool flags. See [`docs/INTEROP.md`](docs/INTEROP.md) for
+> the shared conventions and the schema's versioning commitment.
 
 ## LLM configuration (opt-in)
 

{extract_cli-0.1.5 → extract_cli-0.1.8}/docs/INTEROP.md

@@ -118,6 +118,7 @@ only stdlib `urllib`, so there is no runtime dependency.
 | Concern | Convention |
 |---|---|
 | Primary result | **stdout** (JSON payload, default) |
+| Discovery | `extract --catalog json` (commands/flags, the suite contract) + `extract schema` / `extract fields --json` |
 | `--why`, warnings, errors | **stderr** |
 | `--why` envelope | plain-text `[why] <header>` block (as in template-vault-cli / draft-cli) |
 | Quiet | `-q` / `--silent` / `--quiet` aliases |