assurance-cli 0.2.0__tar.gz → 0.2.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (25) hide show
  1. assurance_cli-0.2.2/PKG-INFO +150 -0
  2. assurance_cli-0.2.2/README.md +123 -0
  3. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli/cli.py +6 -1
  4. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli/gather.py +13 -4
  5. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli/setdiff.py +36 -3
  6. assurance_cli-0.2.2/assurance_cli.egg-info/PKG-INFO +150 -0
  7. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli.egg-info/requires.txt +1 -1
  8. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/pyproject.toml +2 -2
  9. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/tests/test_cli.py +31 -0
  10. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/tests/test_setdiff.py +86 -0
  11. assurance_cli-0.2.0/PKG-INFO +0 -158
  12. assurance_cli-0.2.0/README.md +0 -131
  13. assurance_cli-0.2.0/assurance_cli.egg-info/PKG-INFO +0 -158
  14. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/LICENSE +0 -0
  15. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli/__init__.py +0 -0
  16. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli/__main__.py +0 -0
  17. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli/baseline.py +0 -0
  18. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli/paths.py +0 -0
  19. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli/profile.py +0 -0
  20. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli.egg-info/SOURCES.txt +0 -0
  21. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli.egg-info/dependency_links.txt +0 -0
  22. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli.egg-info/entry_points.txt +0 -0
  23. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/assurance_cli.egg-info/top_level.txt +0 -0
  24. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/setup.cfg +0 -0
  25. {assurance_cli-0.2.0 → assurance_cli-0.2.2}/tests/test_read_only.py +0 -0
@@ -0,0 +1,150 @@
1
+ Metadata-Version: 2.4
2
+ Name: assurance-cli
3
+ Version: 0.2.2
4
+ Summary: CLI for folder assurance checks — coverage, staleness, and baselines.
5
+ Author-email: "I-Ops Operations Intelligence, LLC" <hello@i-ops.dev>
6
+ License: Apache-2.0
7
+ Project-URL: Homepage, https://i-ops.dev
8
+ Project-URL: Source, https://github.com/i-ops-hq/assurance-cli
9
+ Keywords: cli,ai-agents,governance,auditing,assurance,reproducibility
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: License :: OSI Approved :: Apache Software License
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: Topic :: Software Development :: Quality Assurance
19
+ Requires-Python: >=3.10
20
+ Description-Content-Type: text/markdown
21
+ License-File: LICENSE
22
+ Requires-Dist: assurance-core>=0.3.2
23
+ Requires-Dist: openpyxl>=3.1
24
+ Provides-Extra: dev
25
+ Requires-Dist: pytest>=8.0; extra == "dev"
26
+ Dynamic: license-file
27
+
28
+ # assurance-cli
29
+
30
+ [![PyPI](https://img.shields.io/pypi/v/assurance-cli)](https://pypi.org/project/assurance-cli/)
31
+ [![Tests](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml/badge.svg)](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml)
32
+ [![Python](https://img.shields.io/pypi/pyversions/assurance-cli)](https://pypi.org/project/assurance-cli/)
33
+ [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE)
34
+
35
+ ## Did the job cover everything it was supposed to cover?
36
+
37
+ One command. One honest ratio. An exit code your pipeline can act on.
38
+
39
+ ```bash
40
+ pip install assurance-cli
41
+ ```
42
+
43
+ ```bash
44
+ assurance diff --expected corpus.txt --found retrieved.json \
45
+ --scope "documents the question spans" --where "the retrieved set" --fail-on-gap
46
+ ```
47
+ ```
48
+ 2 of 5 documents the question spans — not in the retrieved set: doc-2, doc-3, doc-5
49
+ also present and not expected: doc-9
50
+ ```
51
+
52
+ That second line matters as much as the ratio: the retriever drew on something the scope never
53
+ allowed. It's reported, and it earns no credit.
54
+
55
+ No account · no API key · no network call · **no model decides any of it**
56
+
57
+ ## Three commands
58
+
59
+ ### `assurance diff` — any two sets of keys
60
+
61
+ ```bash
62
+ # code review agent actually read the diff?
63
+ git diff --name-only origin/main...HEAD > changed.txt
64
+ assurance diff --expected changed.txt --found reviewed.txt --fail-on-gap
65
+
66
+ # eval suite ran every declared case?
67
+ assurance diff --expected cases.json --found ran.json --fail-on-gap
68
+
69
+ # straight from a pipe
70
+ retriever --query "$Q" | jq -r '.chunks[].doc_id' | \
71
+ assurance diff --expected corpus.txt --found - --json
72
+ ```
73
+
74
+ Inputs are whatever you already have: **one key per line**, a **JSON array** (strings, or objects
75
+ with `key`/`id`/`name`/`path`), **`-` for stdin**, or an **inline comma list**.
76
+
77
+ ### `assurance check` — a folder of dated or numbered files
78
+
79
+ ```bash
80
+ assurance check ~/reports
81
+ ```
82
+ ```
83
+ 22 of 24 months from 2024-01 to 2025-12 in reports — not in this folder: March 2025, July 2025
84
+ — Range inferred from filenames: earliest 2024-01, latest 2025-12. Override with --from / --to.
85
+ ```
86
+
87
+ ```bash
88
+ assurance check ~/invoices --expect numbered
89
+ ```
90
+ ```
91
+ 7 of 8 runs from inv_0001 to inv_0008 in invoices — not in this folder: INV-0006
92
+ — Range inferred from filenames: earliest inv_0001, latest inv_0008. Override with --from / --to.
93
+ ```
94
+
95
+ Monthly, quarterly, weekly, daily, numbered. That last line is the **derivation**: it prints with
96
+ every ratio so you can disagree with the denominator, not just the result.
97
+
98
+ ### `assurance init` — did anything change underneath?
99
+
100
+ ```bash
101
+ assurance init ~/thesis-data
102
+ # ... weeks pass, several people touch the folder ...
103
+ assurance check ~/thesis-data --against-baseline
104
+ ```
105
+
106
+ ## Exit codes
107
+
108
+ | | |
109
+ |---|---|
110
+ | `0` | it checked, and either found no gap or wasn't asked to fail on one |
111
+ | `1` | a finding: a gap with `--fail-on-gap`, a stale baseline, or **nothing it could check** |
112
+ | `2` | could not run: bad path, unreadable list, unparseable JSON, a table where keys were expected |
113
+
114
+ **"I couldn't check this" exits 1, not 0.** A folder whose filenames it can't parse must not look
115
+ like a folder it checked and found whole.
116
+
117
+ Diagnostics go to **stderr**, results to **stdout**, so `--json` stays pipeable.
118
+
119
+ ## It expects your files, not tidy ones
120
+
121
+ - **Excel exports work.** UTF-8 BOM and CRLF are handled; a BOM used to glue itself to your first
122
+ key and report it as missing *and* unexpected in the same sentence
123
+ - **Spaces, unicode and month words in filenames** — `Inventory Report August 2024.csv` parses
124
+ - **`.xlsx`, and nested subfolders**
125
+ - **A piped CSV is refused, not misread.** It names the column-picking command instead of quietly
126
+ admitting your header row as a key
127
+ - **When it can't work out a series it says so**, rather than reporting an empty check as a pass
128
+
129
+ ## Use it for
130
+
131
+ | | expected | found |
132
+ |---|---|---|
133
+ | **RAG** | documents the question spans | chunks retrieved |
134
+ | **Code review in CI** | `git diff --name-only` | files reviewed |
135
+ | **ETL / batch** | records or partitions declared | records or partitions loaded |
136
+ | **Compliance** | controls in scope | controls with evidence |
137
+ | **Research data** | the series you expect | what's actually in the folder |
138
+
139
+ ## What it won't do
140
+
141
+ - **Invent your expected set.** `diff` takes your declaration; `check` derives one and prints how
142
+ - **Send anything anywhere.** No network, no telemetry, no keys
143
+ - **Guess.** A JSON object of id → metadata is refused, not interpreted
144
+
145
+ ## Family
146
+
147
+ [assurance-core](https://pypi.org/project/assurance-core/) — the pure arithmetic, zero dependencies ·
148
+ [assurance-mcp](https://pypi.org/project/assurance-mcp/) — the same checks as MCP tools
149
+
150
+ Upstream is [I-Ops](https://i-ops.dev); this repo is a publication, never a source. Apache-2.0.
@@ -0,0 +1,123 @@
1
+ # assurance-cli
2
+
3
+ [![PyPI](https://img.shields.io/pypi/v/assurance-cli)](https://pypi.org/project/assurance-cli/)
4
+ [![Tests](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml/badge.svg)](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml)
5
+ [![Python](https://img.shields.io/pypi/pyversions/assurance-cli)](https://pypi.org/project/assurance-cli/)
6
+ [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE)
7
+
8
+ ## Did the job cover everything it was supposed to cover?
9
+
10
+ One command. One honest ratio. An exit code your pipeline can act on.
11
+
12
+ ```bash
13
+ pip install assurance-cli
14
+ ```
15
+
16
+ ```bash
17
+ assurance diff --expected corpus.txt --found retrieved.json \
18
+ --scope "documents the question spans" --where "the retrieved set" --fail-on-gap
19
+ ```
20
+ ```
21
+ 2 of 5 documents the question spans — not in the retrieved set: doc-2, doc-3, doc-5
22
+ also present and not expected: doc-9
23
+ ```
24
+
25
+ That second line matters as much as the ratio: the retriever drew on something the scope never
26
+ allowed. It's reported, and it earns no credit.
27
+
28
+ No account · no API key · no network call · **no model decides any of it**
29
+
30
+ ## Three commands
31
+
32
+ ### `assurance diff` — any two sets of keys
33
+
34
+ ```bash
35
+ # code review agent actually read the diff?
36
+ git diff --name-only origin/main...HEAD > changed.txt
37
+ assurance diff --expected changed.txt --found reviewed.txt --fail-on-gap
38
+
39
+ # eval suite ran every declared case?
40
+ assurance diff --expected cases.json --found ran.json --fail-on-gap
41
+
42
+ # straight from a pipe
43
+ retriever --query "$Q" | jq -r '.chunks[].doc_id' | \
44
+ assurance diff --expected corpus.txt --found - --json
45
+ ```
46
+
47
+ Inputs are whatever you already have: **one key per line**, a **JSON array** (strings, or objects
48
+ with `key`/`id`/`name`/`path`), **`-` for stdin**, or an **inline comma list**.
49
+
50
+ ### `assurance check` — a folder of dated or numbered files
51
+
52
+ ```bash
53
+ assurance check ~/reports
54
+ ```
55
+ ```
56
+ 22 of 24 months from 2024-01 to 2025-12 in reports — not in this folder: March 2025, July 2025
57
+ — Range inferred from filenames: earliest 2024-01, latest 2025-12. Override with --from / --to.
58
+ ```
59
+
60
+ ```bash
61
+ assurance check ~/invoices --expect numbered
62
+ ```
63
+ ```
64
+ 7 of 8 runs from inv_0001 to inv_0008 in invoices — not in this folder: INV-0006
65
+ — Range inferred from filenames: earliest inv_0001, latest inv_0008. Override with --from / --to.
66
+ ```
67
+
68
+ Monthly, quarterly, weekly, daily, numbered. That last line is the **derivation**: it prints with
69
+ every ratio so you can disagree with the denominator, not just the result.
70
+
71
+ ### `assurance init` — did anything change underneath?
72
+
73
+ ```bash
74
+ assurance init ~/thesis-data
75
+ # ... weeks pass, several people touch the folder ...
76
+ assurance check ~/thesis-data --against-baseline
77
+ ```
78
+
79
+ ## Exit codes
80
+
81
+ | | |
82
+ |---|---|
83
+ | `0` | it checked, and either found no gap or wasn't asked to fail on one |
84
+ | `1` | a finding: a gap with `--fail-on-gap`, a stale baseline, or **nothing it could check** |
85
+ | `2` | could not run: bad path, unreadable list, unparseable JSON, a table where keys were expected |
86
+
87
+ **"I couldn't check this" exits 1, not 0.** A folder whose filenames it can't parse must not look
88
+ like a folder it checked and found whole.
89
+
90
+ Diagnostics go to **stderr**, results to **stdout**, so `--json` stays pipeable.
91
+
92
+ ## It expects your files, not tidy ones
93
+
94
+ - **Excel exports work.** UTF-8 BOM and CRLF are handled; a BOM used to glue itself to your first
95
+ key and report it as missing *and* unexpected in the same sentence
96
+ - **Spaces, unicode and month words in filenames** — `Inventory Report August 2024.csv` parses
97
+ - **`.xlsx`, and nested subfolders**
98
+ - **A piped CSV is refused, not misread.** It names the column-picking command instead of quietly
99
+ admitting your header row as a key
100
+ - **When it can't work out a series it says so**, rather than reporting an empty check as a pass
101
+
102
+ ## Use it for
103
+
104
+ | | expected | found |
105
+ |---|---|---|
106
+ | **RAG** | documents the question spans | chunks retrieved |
107
+ | **Code review in CI** | `git diff --name-only` | files reviewed |
108
+ | **ETL / batch** | records or partitions declared | records or partitions loaded |
109
+ | **Compliance** | controls in scope | controls with evidence |
110
+ | **Research data** | the series you expect | what's actually in the folder |
111
+
112
+ ## What it won't do
113
+
114
+ - **Invent your expected set.** `diff` takes your declaration; `check` derives one and prints how
115
+ - **Send anything anywhere.** No network, no telemetry, no keys
116
+ - **Guess.** A JSON object of id → metadata is refused, not interpreted
117
+
118
+ ## Family
119
+
120
+ [assurance-core](https://pypi.org/project/assurance-core/) — the pure arithmetic, zero dependencies ·
121
+ [assurance-mcp](https://pypi.org/project/assurance-mcp/) — the same checks as MCP tools
122
+
123
+ Upstream is [I-Ops](https://i-ops.dev); this repo is a publication, never a source. Apache-2.0.
@@ -136,7 +136,12 @@ def _has_findings(payload: dict[str, Any]) -> bool:
136
136
  if payload.get("baseline") and not payload["baseline"].get("ok", True):
137
137
  return True
138
138
  coverage = payload.get("coverage") or payload
139
- return bool(coverage.get("error"))
139
+ if coverage.get("error"):
140
+ return True
141
+ # "I could not work out what to check" is a finding, not a success. Exiting 0 here made a folder
142
+ # whose filenames we cannot parse indistinguishable, to a CI job, from a folder we checked and
143
+ # found whole — which is the one thing this tool exists not to do.
144
+ return bool((coverage.get("coverage") or {}).get("undetermined"))
140
145
 
141
146
 
142
147
  def _format_text(payload: dict[str, Any]) -> str:
@@ -73,7 +73,7 @@ def check_coverage(
73
73
  "complete": False,
74
74
  "derivation": "",
75
75
  "coverage": _coverage_to_dict(
76
- Coverage(scope_label=f"items in {root.name}", expected=[])
76
+ Coverage(scope_label=f"items in {root.name}", expected=[], undetermined=_UNDETERMINED)
77
77
  ),
78
78
  }
79
79
 
@@ -88,7 +88,7 @@ def check_coverage(
88
88
  "complete": False,
89
89
  "derivation": "",
90
90
  "coverage": _coverage_to_dict(
91
- Coverage(scope_label=f"items in {root.name}", expected=[])
91
+ Coverage(scope_label=f"items in {root.name}", expected=[], undetermined=_UNDETERMINED)
92
92
  ),
93
93
  }
94
94
 
@@ -122,7 +122,7 @@ def check_coverage(
122
122
  "complete": False,
123
123
  "derivation": "",
124
124
  "coverage": _coverage_to_dict(
125
- Coverage(scope_label=f"items in {root.name}", expected=[])
125
+ Coverage(scope_label=f"items in {root.name}", expected=[], undetermined=_UNDETERMINED)
126
126
  ),
127
127
  }
128
128
 
@@ -264,13 +264,21 @@ def _unit_for_kind(kind: SeriesKind) -> str:
264
264
  }[kind]
265
265
 
266
266
 
267
+ _UNDETERMINED = "no dated or numbered series could be read from these filenames"
268
+ """Why nothing was checked. Carried INTO the coverage record, not just the wrapper around it.
269
+
270
+ Until 0.2.2 these paths emitted an empty `Coverage`, whose `complete` is True by the arithmetic —
271
+ nothing was required, so nothing is missing. So the payload carried `complete: false` at the top and
272
+ `complete: true` one level down, and an integrator reading either one was reading a real field."""
273
+
274
+
267
275
  def _error_result(root: Path, message: str) -> dict[str, Any]:
268
276
  return {
269
277
  "folder": str(root),
270
278
  "summary": message,
271
279
  "complete": False,
272
280
  "derivation": "",
273
- "coverage": _coverage_to_dict(Coverage(scope_label=f"items in {root.name}", expected=[])),
281
+ "coverage": _coverage_to_dict(Coverage(scope_label=f"items in {root.name}", expected=[], undetermined=_UNDETERMINED)),
274
282
  "error": message,
275
283
  }
276
284
 
@@ -290,6 +298,7 @@ def _coverage_to_dict(cov: Coverage) -> dict[str, Any]:
290
298
  "unreadable": dict(cov.unreadable),
291
299
  "unauthorized": dict(cov.unauthorized),
292
300
  "truncated": cov.truncated,
301
+ "undetermined": cov.undetermined,
293
302
  }
294
303
 
295
304
 
@@ -45,7 +45,11 @@ def read_keys(spec: str | None, *, label: str) -> list[str]:
45
45
  if candidate.exists():
46
46
  if candidate.is_dir():
47
47
  raise KeySpecError(f"{label}: {spec} is a directory, not a list of keys")
48
- text = candidate.read_text(encoding="utf-8")
48
+ # `utf-8-sig`, not `utf-8`. Excel writes CSV and TXT with a byte-order mark by default,
49
+ # and reading it as plain utf-8 glues U+FEFF to the FIRST key — so `doc-1` was reported
50
+ # as missing AND as unexpected in the same sentence, which is a confidently wrong answer
51
+ # produced by the most common export path there is.
52
+ text = candidate.read_text(encoding="utf-8-sig")
49
53
 
50
54
  if text is None:
51
55
  return _dedupe(part.strip() for part in spec.split(","))
@@ -64,11 +68,38 @@ def _parse(text: str, *, label: str) -> list[str]:
64
68
  return _keys_from_json(payload, label=label)
65
69
  # A full-line comment is dropped; a `#` inside a key is part of the key, because guessing which
66
70
  # is which would silently change someone's denominator.
67
- return [
71
+ lines = [
68
72
  line.strip()
69
73
  for line in stripped.splitlines()
70
74
  if line.strip() and not line.lstrip().startswith("#")
71
75
  ]
76
+ _refuse_a_table(lines, label=label)
77
+ return lines
78
+
79
+
80
+ def _refuse_a_table(lines: list[str], *, label: str) -> None:
81
+ """A delimited table is not a list of keys, and reading it as one is silent nonsense.
82
+
83
+ Piping a CSV is the obvious thing to try — it is what people have. Read line by line it produced
84
+ `0 of 3 — also present and not expected: doc_id,score, doc-1,0.9, doc-2,0.8`: a header row and
85
+ two score columns admitted as keys, and a confident zero. We cannot pick the column for you, so
86
+ this says which command does.
87
+
88
+ Deliberately narrow, because a key may legitimately contain a comma: it fires only when EVERY
89
+ line carries the SAME delimiter the SAME number of times, which is a table and not a coincidence.
90
+ """
91
+ if len(lines) < 2:
92
+ return
93
+ for delimiter, flag in (("\t", "-f1"), (",", "-d, -f1")):
94
+ counts = {line.count(delimiter) for line in lines}
95
+ if len(counts) == 1 and counts.pop() >= 1:
96
+ name = "tab" if delimiter == "\t" else "comma"
97
+ raise KeySpecError(
98
+ f"{label}: this looks like a {name}-delimited table, not a list of keys — every "
99
+ f"line has the same number of {name}s. Reading it line by line would admit the "
100
+ f"header and every other column as keys. Pick the column you mean, e.g. "
101
+ f"`cut {flag} file | assurance diff ... --{label.lstrip('-')} -`, or pass JSON."
102
+ )
72
103
 
73
104
 
74
105
  def _keys_from_json(payload: Any, *, label: str) -> list[str]:
@@ -106,7 +137,9 @@ def _dedupe(values: Any) -> list[str]:
106
137
  seen: set[str] = set()
107
138
  out: list[str] = []
108
139
  for value in values:
109
- value = str(value).strip()
140
+ # A BOM can still arrive from stdin, where there is no decode step to strip it, and inside a
141
+ # JSON string. Strip it per key too: a stray U+FEFF changes what a key IS.
142
+ value = str(value).replace("\ufeff", "").strip()
110
143
  if value and value not in seen:
111
144
  seen.add(value)
112
145
  out.append(value)
@@ -0,0 +1,150 @@
1
+ Metadata-Version: 2.4
2
+ Name: assurance-cli
3
+ Version: 0.2.2
4
+ Summary: CLI for folder assurance checks — coverage, staleness, and baselines.
5
+ Author-email: "I-Ops Operations Intelligence, LLC" <hello@i-ops.dev>
6
+ License: Apache-2.0
7
+ Project-URL: Homepage, https://i-ops.dev
8
+ Project-URL: Source, https://github.com/i-ops-hq/assurance-cli
9
+ Keywords: cli,ai-agents,governance,auditing,assurance,reproducibility
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: License :: OSI Approved :: Apache Software License
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: Topic :: Software Development :: Quality Assurance
19
+ Requires-Python: >=3.10
20
+ Description-Content-Type: text/markdown
21
+ License-File: LICENSE
22
+ Requires-Dist: assurance-core>=0.3.2
23
+ Requires-Dist: openpyxl>=3.1
24
+ Provides-Extra: dev
25
+ Requires-Dist: pytest>=8.0; extra == "dev"
26
+ Dynamic: license-file
27
+
28
+ # assurance-cli
29
+
30
+ [![PyPI](https://img.shields.io/pypi/v/assurance-cli)](https://pypi.org/project/assurance-cli/)
31
+ [![Tests](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml/badge.svg)](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml)
32
+ [![Python](https://img.shields.io/pypi/pyversions/assurance-cli)](https://pypi.org/project/assurance-cli/)
33
+ [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE)
34
+
35
+ ## Did the job cover everything it was supposed to cover?
36
+
37
+ One command. One honest ratio. An exit code your pipeline can act on.
38
+
39
+ ```bash
40
+ pip install assurance-cli
41
+ ```
42
+
43
+ ```bash
44
+ assurance diff --expected corpus.txt --found retrieved.json \
45
+ --scope "documents the question spans" --where "the retrieved set" --fail-on-gap
46
+ ```
47
+ ```
48
+ 2 of 5 documents the question spans — not in the retrieved set: doc-2, doc-3, doc-5
49
+ also present and not expected: doc-9
50
+ ```
51
+
52
+ That second line matters as much as the ratio: the retriever drew on something the scope never
53
+ allowed. It's reported, and it earns no credit.
54
+
55
+ No account · no API key · no network call · **no model decides any of it**
56
+
57
+ ## Three commands
58
+
59
+ ### `assurance diff` — any two sets of keys
60
+
61
+ ```bash
62
+ # code review agent actually read the diff?
63
+ git diff --name-only origin/main...HEAD > changed.txt
64
+ assurance diff --expected changed.txt --found reviewed.txt --fail-on-gap
65
+
66
+ # eval suite ran every declared case?
67
+ assurance diff --expected cases.json --found ran.json --fail-on-gap
68
+
69
+ # straight from a pipe
70
+ retriever --query "$Q" | jq -r '.chunks[].doc_id' | \
71
+ assurance diff --expected corpus.txt --found - --json
72
+ ```
73
+
74
+ Inputs are whatever you already have: **one key per line**, a **JSON array** (strings, or objects
75
+ with `key`/`id`/`name`/`path`), **`-` for stdin**, or an **inline comma list**.
76
+
77
+ ### `assurance check` — a folder of dated or numbered files
78
+
79
+ ```bash
80
+ assurance check ~/reports
81
+ ```
82
+ ```
83
+ 22 of 24 months from 2024-01 to 2025-12 in reports — not in this folder: March 2025, July 2025
84
+ — Range inferred from filenames: earliest 2024-01, latest 2025-12. Override with --from / --to.
85
+ ```
86
+
87
+ ```bash
88
+ assurance check ~/invoices --expect numbered
89
+ ```
90
+ ```
91
+ 7 of 8 runs from inv_0001 to inv_0008 in invoices — not in this folder: INV-0006
92
+ — Range inferred from filenames: earliest inv_0001, latest inv_0008. Override with --from / --to.
93
+ ```
94
+
95
+ Monthly, quarterly, weekly, daily, numbered. That last line is the **derivation**: it prints with
96
+ every ratio so you can disagree with the denominator, not just the result.
97
+
98
+ ### `assurance init` — did anything change underneath?
99
+
100
+ ```bash
101
+ assurance init ~/thesis-data
102
+ # ... weeks pass, several people touch the folder ...
103
+ assurance check ~/thesis-data --against-baseline
104
+ ```
105
+
106
+ ## Exit codes
107
+
108
+ | | |
109
+ |---|---|
110
+ | `0` | it checked, and either found no gap or wasn't asked to fail on one |
111
+ | `1` | a finding: a gap with `--fail-on-gap`, a stale baseline, or **nothing it could check** |
112
+ | `2` | could not run: bad path, unreadable list, unparseable JSON, a table where keys were expected |
113
+
114
+ **"I couldn't check this" exits 1, not 0.** A folder whose filenames it can't parse must not look
115
+ like a folder it checked and found whole.
116
+
117
+ Diagnostics go to **stderr**, results to **stdout**, so `--json` stays pipeable.
118
+
119
+ ## It expects your files, not tidy ones
120
+
121
+ - **Excel exports work.** UTF-8 BOM and CRLF are handled; a BOM used to glue itself to your first
122
+ key and report it as missing *and* unexpected in the same sentence
123
+ - **Spaces, unicode and month words in filenames** — `Inventory Report August 2024.csv` parses
124
+ - **`.xlsx`, and nested subfolders**
125
+ - **A piped CSV is refused, not misread.** It names the column-picking command instead of quietly
126
+ admitting your header row as a key
127
+ - **When it can't work out a series it says so**, rather than reporting an empty check as a pass
128
+
129
+ ## Use it for
130
+
131
+ | | expected | found |
132
+ |---|---|---|
133
+ | **RAG** | documents the question spans | chunks retrieved |
134
+ | **Code review in CI** | `git diff --name-only` | files reviewed |
135
+ | **ETL / batch** | records or partitions declared | records or partitions loaded |
136
+ | **Compliance** | controls in scope | controls with evidence |
137
+ | **Research data** | the series you expect | what's actually in the folder |
138
+
139
+ ## What it won't do
140
+
141
+ - **Invent your expected set.** `diff` takes your declaration; `check` derives one and prints how
142
+ - **Send anything anywhere.** No network, no telemetry, no keys
143
+ - **Guess.** A JSON object of id → metadata is refused, not interpreted
144
+
145
+ ## Family
146
+
147
+ [assurance-core](https://pypi.org/project/assurance-core/) — the pure arithmetic, zero dependencies ·
148
+ [assurance-mcp](https://pypi.org/project/assurance-mcp/) — the same checks as MCP tools
149
+
150
+ Upstream is [I-Ops](https://i-ops.dev); this repo is a publication, never a source. Apache-2.0.
@@ -1,4 +1,4 @@
1
- assurance-core>=0.3
1
+ assurance-core>=0.3.2
2
2
  openpyxl>=3.1
3
3
 
4
4
  [dev]
@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "assurance-cli"
7
- version = "0.2.0"
7
+ version = "0.2.2"
8
8
  description = "CLI for folder assurance checks — coverage, staleness, and baselines."
9
9
  readme = "README.md"
10
10
  license = { text = "Apache-2.0" }
11
11
  authors = [{ name = "I-Ops Operations Intelligence, LLC", email = "hello@i-ops.dev" }]
12
12
  requires-python = ">=3.10"
13
13
  dependencies = [
14
- "assurance-core>=0.3",
14
+ "assurance-core>=0.3.2",
15
15
  "openpyxl>=3.1",
16
16
  ]
17
17
  keywords = ["cli", "ai-agents", "governance", "auditing", "assurance", "reproducibility"]
@@ -123,3 +123,34 @@ def test_mixed_series_folder_reports_none(tmp_path: Path):
123
123
  (root / name).write_text("v\n1\n", encoding="utf-8")
124
124
  result = check_coverage(str(root))
125
125
  assert "No dated or numbered series detected" in result["summary"]
126
+
127
+
128
+ # --- "I could not check" must not exit like "I checked and it was fine" ---------------------------
129
+
130
+
131
+ def test_a_folder_with_no_readable_series_is_a_finding(tmp_path: Path) -> None:
132
+ """Exit 0 here made an unparseable folder indistinguishable, to a CI job, from a whole one."""
133
+ for name in ("15.01.2024 shipment.csv", "Jan-24 summary.csv", "20240115_dump.csv"):
134
+ (tmp_path / name).write_text("a,b\n1,2\n", encoding="utf-8")
135
+
136
+ assert main(["check", str(tmp_path)]) == 1
137
+
138
+
139
+ def test_the_nested_record_agrees_with_the_wrapper(
140
+ tmp_path: Path, capsys: pytest.CaptureFixture[str]
141
+ ) -> None:
142
+ """The payload carried `complete: false` at the top and `complete: true` one level down, and an
143
+ integrator reading either one was reading a real field."""
144
+ (tmp_path / "Jan-24 summary.csv").write_text("a,b\n1,2\n", encoding="utf-8")
145
+
146
+ main(["check", str(tmp_path), "--json"])
147
+ payload = json.loads(capsys.readouterr().out)["coverage"]
148
+
149
+ assert payload["complete"] is False
150
+ assert payload["coverage"]["complete"] is False
151
+ assert payload["coverage"]["undetermined"]
152
+
153
+
154
+ def test_a_folder_that_checks_out_still_exits_zero(monthly_folder: Path) -> None:
155
+ """The counterweight: the ordinary path must not start failing."""
156
+ assert main(["check", str(monthly_folder)]) == 0
@@ -150,3 +150,89 @@ def test_an_error_is_printed_rather_than_swallowed(capsys: pytest.CaptureFixture
150
150
  captured = capsys.readouterr()
151
151
  assert captured.out.strip() == ""
152
152
  assert "folder does not exist" in captured.err
153
+
154
+
155
+ # --- what an outsider's files actually look like --------------------------------------------------
156
+ #
157
+ # From the 2026-08-29 smoke test against fixtures nobody here designed against: Excel exports,
158
+ # filenames with spaces, month words, six-digit purchase orders. Most of it worked. This did not.
159
+
160
+
161
+ def test_a_byte_order_mark_does_not_change_what_a_key_is(tmp_path: Path) -> None:
162
+ """Excel writes CSV and TXT with a BOM by default. Read as plain utf-8 it glues U+FEFF to the
163
+ FIRST key, so `doc-1` came back reported as missing AND as unexpected in one sentence — a
164
+ confidently wrong answer, from the most common export path there is."""
165
+ listing = tmp_path / "keys.txt"
166
+ listing.write_bytes(b"\xef\xbb\xbfdoc-1\r\ndoc-2\r\ndoc-3\r\n")
167
+
168
+ assert read_keys(str(listing), label="--expected") == ["doc-1", "doc-2", "doc-3"]
169
+
170
+
171
+ def test_a_bom_in_a_json_export_is_stripped_too(tmp_path: Path) -> None:
172
+ listing = tmp_path / "found.json"
173
+ listing.write_bytes(b'\xef\xbb\xbf["doc-1", "doc-2"]')
174
+
175
+ assert read_keys(str(listing), label="--found") == ["doc-1", "doc-2"]
176
+
177
+
178
+ def test_the_bom_case_end_to_end_reports_one_gap_not_two_wrong_ones(tmp_path: Path) -> None:
179
+ listing = tmp_path / "keys.txt"
180
+ listing.write_bytes(b"\xef\xbb\xbfdoc-1\r\ndoc-2\r\ndoc-3\r\n")
181
+
182
+ payload = diff_sets(str(listing), "doc-1,doc-2")
183
+
184
+ assert payload["read"] == 2
185
+ assert [entry["key"] for entry in payload["missing"]] == ["doc-3"]
186
+ assert payload["unexpected"] == []
187
+
188
+
189
+ def test_crlf_line_endings_do_not_become_part_of_the_key(tmp_path: Path) -> None:
190
+ listing = tmp_path / "keys.txt"
191
+ listing.write_bytes(b"doc-1\r\ndoc-2\r\n")
192
+
193
+ assert read_keys(str(listing), label="--expected") == ["doc-1", "doc-2"]
194
+
195
+
196
+ def test_a_bom_arriving_on_stdin_is_stripped(monkeypatch: pytest.MonkeyPatch) -> None:
197
+ """`cat export.txt | assurance diff --found -` has no decode step to strip a BOM: `sys.stdin`
198
+ is already decoded, by the locale's plain utf-8. The README documents piping, so this path is
199
+ real and needs the per-key strip that `utf-8-sig` cannot reach."""
200
+ import io
201
+
202
+ monkeypatch.setattr("sys.stdin", io.StringIO("doc-1\ndoc-2\n"))
203
+
204
+ assert read_keys("-", label="--found") == ["doc-1", "doc-2"]
205
+
206
+
207
+ def test_a_piped_csv_is_refused_rather_than_read_as_keys(tmp_path: Path) -> None:
208
+ """Piping a CSV is the obvious thing to try. Read line by line it produced a confident `0 of 3`
209
+ with the header row and two score columns admitted as keys."""
210
+ listing = tmp_path / "found.csv"
211
+ listing.write_text("doc_id,score\ndoc-1,0.9\ndoc-2,0.8\n", encoding="utf-8")
212
+
213
+ with pytest.raises(KeySpecError, match="comma-delimited table"):
214
+ read_keys(str(listing), label="--found")
215
+
216
+
217
+ def test_the_refusal_names_the_command_that_fixes_it(tmp_path: Path) -> None:
218
+ listing = tmp_path / "found.tsv"
219
+ listing.write_text("doc_id\tscore\ndoc-1\t0.9\n", encoding="utf-8")
220
+
221
+ with pytest.raises(KeySpecError, match=r"cut -f1"):
222
+ read_keys(str(listing), label="--found")
223
+
224
+
225
+ def test_a_key_that_merely_contains_a_comma_is_still_a_key(tmp_path: Path) -> None:
226
+ """The guard fires only when EVERY line has the SAME delimiter count — a table, not a
227
+ coincidence. A false refusal blocks a legitimate user, so it stays narrow."""
228
+ listing = tmp_path / "keys.txt"
229
+ listing.write_text("Smith, John\nDoe, Jane\nplain-key\n", encoding="utf-8")
230
+
231
+ assert read_keys(str(listing), label="--expected") == ["Smith, John", "Doe, Jane", "plain-key"]
232
+
233
+
234
+ def test_a_single_line_is_never_a_table(tmp_path: Path) -> None:
235
+ listing = tmp_path / "keys.txt"
236
+ listing.write_text("a,b,c\n", encoding="utf-8")
237
+
238
+ assert read_keys(str(listing), label="--expected") == ["a,b,c"]
@@ -1,158 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: assurance-cli
3
- Version: 0.2.0
4
- Summary: CLI for folder assurance checks — coverage, staleness, and baselines.
5
- Author-email: "I-Ops Operations Intelligence, LLC" <hello@i-ops.dev>
6
- License: Apache-2.0
7
- Project-URL: Homepage, https://i-ops.dev
8
- Project-URL: Source, https://github.com/i-ops-hq/assurance-cli
9
- Keywords: cli,ai-agents,governance,auditing,assurance,reproducibility
10
- Classifier: Development Status :: 4 - Beta
11
- Classifier: Intended Audience :: Developers
12
- Classifier: License :: OSI Approved :: Apache Software License
13
- Classifier: Programming Language :: Python :: 3
14
- Classifier: Programming Language :: Python :: 3.10
15
- Classifier: Programming Language :: Python :: 3.11
16
- Classifier: Programming Language :: Python :: 3.12
17
- Classifier: Programming Language :: Python :: 3.13
18
- Classifier: Topic :: Software Development :: Quality Assurance
19
- Requires-Python: >=3.10
20
- Description-Content-Type: text/markdown
21
- License-File: LICENSE
22
- Requires-Dist: assurance-core>=0.3
23
- Requires-Dist: openpyxl>=3.1
24
- Provides-Extra: dev
25
- Requires-Dist: pytest>=8.0; extra == "dev"
26
- Dynamic: license-file
27
-
28
- # assurance-cli
29
-
30
- [![PyPI](https://img.shields.io/pypi/v/assurance-cli)](https://pypi.org/project/assurance-cli/)
31
- [![Tests](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml/badge.svg)](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml)
32
- [![Python](https://img.shields.io/pypi/pyversions/assurance-cli)](https://pypi.org/project/assurance-cli/)
33
- [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE)
34
-
35
- ### Did the job cover everything it was supposed to cover?
36
-
37
- One command, one honest ratio, and an exit code your pipeline can act on. No model, no API key, no
38
- network call, nothing uploaded.
39
-
40
- ```bash
41
- pip install assurance-cli
42
- ```
43
-
44
- ## `assurance diff` — coverage over any two sets
45
-
46
- The general form. Give it what a task **required** and what it **actually read**; it tells you the
47
- difference and exits non-zero on a gap. Keys are anything you can name.
48
-
49
- ```bash
50
- # A retrieval agent: did the retriever see the whole question?
51
- assurance diff \
52
- --expected corpus-for-this-question.txt \
53
- --found retrieved.json \
54
- --scope "documents the question spans" \
55
- --where "the retrieved set" \
56
- --fail-on-gap
57
- ```
58
-
59
- ```
60
- 2 of 5 documents the question spans — not in the retrieved set: doc-2, doc-3, doc-5
61
- also present and not expected: doc-9
62
- ```
63
-
64
- That last line matters as much as the ratio: the retriever drew on a document the scope never
65
- allowed. It is reported, and it earns no credit against the denominator.
66
-
67
- **Inputs are whatever you already have** — a file with one key per line, a JSON array of strings or
68
- of objects with a `key`/`id`/`name`/`path` field, `-` for stdin, or a comma-separated list inline.
69
-
70
- ```bash
71
- # Gate an agentic code review on having actually read the diff
72
- git diff --name-only origin/main...HEAD > changed.txt
73
- assurance diff --expected changed.txt --found reviewed.txt \
74
- --scope "files changed in this pull request" --where "the review log" --fail-on-gap
75
-
76
- # Did the eval suite run every declared case?
77
- assurance diff --expected cases.json --found ran.json --scope "declared eval cases" --fail-on-gap
78
-
79
- # Were all the partitions loaded?
80
- aws s3 ls s3://lake/dt=2026-08-14/ | awk '{print $4}' > loaded.txt
81
- assurance diff --expected expected-partitions.txt --found loaded.txt --where "the warehouse"
82
-
83
- # Straight from a pipe
84
- retriever --query "$Q" --json | jq -r '.chunks[].doc_id' | \
85
- assurance diff --expected corpus.txt --found - --json
86
- ```
87
-
88
- `--json` gives you the full record for a CI artefact: `complete`, `read` of `required`, and each way
89
- an expectation failed to be evidence kept separate.
90
-
91
- ## `assurance check` — coverage over a folder of dated files
92
-
93
- When the thing you must account for is a series on disk, the expected set is derived for you from the
94
- filenames. Monthly, quarterly, weekly, daily, or plain numbered.
95
-
96
- ```bash
97
- assurance check ~/reports
98
- # 22 of 24 months from 01/2024 to 12/2025 in reports — not in this folder: 03/2025, 07/2025
99
-
100
- assurance check ~/reports --from 2024-01 --to 2025-12 --fail-on-gap
101
- assurance check ~/invoices --expect numbered # gaps in INV-0001..INV-0450
102
- ```
103
-
104
- The derivation is printed with the ratio, so you can disagree with the **denominator** rather than
105
- only with the result. That is deliberate: a denominator a tool invents for you is a denominator
106
- nobody can argue with.
107
-
108
- ## `assurance init` / `--against-baseline` — did anything change underneath?
109
-
110
- Write a baseline of a folder, then ask later whether it still holds. Catches the file that was
111
- quietly replaced, the row count that moved, the figure that no longer matches its source.
112
-
113
- ```bash
114
- assurance init ~/thesis-data
115
- # ... weeks pass, several people touch the folder ...
116
- assurance check ~/thesis-data --against-baseline
117
- ```
118
-
119
- ## Exit codes
120
-
121
- | Code | Meaning |
122
- |---|---|
123
- | `0` | Ran, and either found no gap or was not asked to fail on one |
124
- | `1` | A finding — a coverage gap with `--fail-on-gap`, or a baseline that no longer holds |
125
- | `2` | Could not run: bad path, unreadable key list, unparseable JSON |
126
-
127
- Diagnostics go to **stderr**, results to **stdout**, so `--json` stays pipeable.
128
-
129
- ## Who this is for
130
-
131
- | If you run | Use it to check |
132
- |---|---|
133
- | RAG or retrieval pipelines | The retrieved set against the documents the question spans |
134
- | Agentic code review in CI | Files reviewed against `git diff --name-only` |
135
- | Batch or ETL jobs | Records processed against records declared |
136
- | Eval harnesses | Cases executed against cases declared |
137
- | Compliance evidence collection | Controls with evidence against controls in scope |
138
- | Research or thesis data | A folder that several people have been editing for months |
139
- | Any reporting series | Months, quarters, weeks, days, or invoice numbers with a hole in them |
140
-
141
- ## What it will not do
142
-
143
- - **It will not invent your expected set.** `diff` takes your declaration; `check` derives one from
144
- filenames and prints how. Both are arguable on purpose.
145
- - **It does not read file contents** except to profile a baseline you asked for.
146
- - **It never sends anything anywhere.** No network calls, no telemetry, no keys.
147
- - **No model decides any of it.** See [`assurance-core`](https://pypi.org/project/assurance-core/)
148
- for the arithmetic; this package owns all filesystem I/O.
149
-
150
- ## Also in this family
151
-
152
- - **[assurance-core](https://pypi.org/project/assurance-core/)** — the pure decision modules, zero dependencies
153
- - **[assurance-mcp](https://pypi.org/project/assurance-mcp/)** — the same checks as MCP tools any agent can call
154
-
155
- ## Licence
156
-
157
- Apache-2.0. See [LICENSE](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE).
158
- Upstream is [I-Ops](https://i-ops.dev); this repo is a publication, never a source.
@@ -1,131 +0,0 @@
1
- # assurance-cli
2
-
3
- [![PyPI](https://img.shields.io/pypi/v/assurance-cli)](https://pypi.org/project/assurance-cli/)
4
- [![Tests](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml/badge.svg)](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml)
5
- [![Python](https://img.shields.io/pypi/pyversions/assurance-cli)](https://pypi.org/project/assurance-cli/)
6
- [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE)
7
-
8
- ### Did the job cover everything it was supposed to cover?
9
-
10
- One command, one honest ratio, and an exit code your pipeline can act on. No model, no API key, no
11
- network call, nothing uploaded.
12
-
13
- ```bash
14
- pip install assurance-cli
15
- ```
16
-
17
- ## `assurance diff` — coverage over any two sets
18
-
19
- The general form. Give it what a task **required** and what it **actually read**; it tells you the
20
- difference and exits non-zero on a gap. Keys are anything you can name.
21
-
22
- ```bash
23
- # A retrieval agent: did the retriever see the whole question?
24
- assurance diff \
25
- --expected corpus-for-this-question.txt \
26
- --found retrieved.json \
27
- --scope "documents the question spans" \
28
- --where "the retrieved set" \
29
- --fail-on-gap
30
- ```
31
-
32
- ```
33
- 2 of 5 documents the question spans — not in the retrieved set: doc-2, doc-3, doc-5
34
- also present and not expected: doc-9
35
- ```
36
-
37
- That last line matters as much as the ratio: the retriever drew on a document the scope never
38
- allowed. It is reported, and it earns no credit against the denominator.
39
-
40
- **Inputs are whatever you already have** — a file with one key per line, a JSON array of strings or
41
- of objects with a `key`/`id`/`name`/`path` field, `-` for stdin, or a comma-separated list inline.
42
-
43
- ```bash
44
- # Gate an agentic code review on having actually read the diff
45
- git diff --name-only origin/main...HEAD > changed.txt
46
- assurance diff --expected changed.txt --found reviewed.txt \
47
- --scope "files changed in this pull request" --where "the review log" --fail-on-gap
48
-
49
- # Did the eval suite run every declared case?
50
- assurance diff --expected cases.json --found ran.json --scope "declared eval cases" --fail-on-gap
51
-
52
- # Were all the partitions loaded?
53
- aws s3 ls s3://lake/dt=2026-08-14/ | awk '{print $4}' > loaded.txt
54
- assurance diff --expected expected-partitions.txt --found loaded.txt --where "the warehouse"
55
-
56
- # Straight from a pipe
57
- retriever --query "$Q" --json | jq -r '.chunks[].doc_id' | \
58
- assurance diff --expected corpus.txt --found - --json
59
- ```
60
-
61
- `--json` gives you the full record for a CI artefact: `complete`, `read` of `required`, and each way
62
- an expectation failed to be evidence kept separate.
63
-
64
- ## `assurance check` — coverage over a folder of dated files
65
-
66
- When the thing you must account for is a series on disk, the expected set is derived for you from the
67
- filenames. Monthly, quarterly, weekly, daily, or plain numbered.
68
-
69
- ```bash
70
- assurance check ~/reports
71
- # 22 of 24 months from 01/2024 to 12/2025 in reports — not in this folder: 03/2025, 07/2025
72
-
73
- assurance check ~/reports --from 2024-01 --to 2025-12 --fail-on-gap
74
- assurance check ~/invoices --expect numbered # gaps in INV-0001..INV-0450
75
- ```
76
-
77
- The derivation is printed with the ratio, so you can disagree with the **denominator** rather than
78
- only with the result. That is deliberate: a denominator a tool invents for you is a denominator
79
- nobody can argue with.
80
-
81
- ## `assurance init` / `--against-baseline` — did anything change underneath?
82
-
83
- Write a baseline of a folder, then ask later whether it still holds. Catches the file that was
84
- quietly replaced, the row count that moved, the figure that no longer matches its source.
85
-
86
- ```bash
87
- assurance init ~/thesis-data
88
- # ... weeks pass, several people touch the folder ...
89
- assurance check ~/thesis-data --against-baseline
90
- ```
91
-
92
- ## Exit codes
93
-
94
- | Code | Meaning |
95
- |---|---|
96
- | `0` | Ran, and either found no gap or was not asked to fail on one |
97
- | `1` | A finding — a coverage gap with `--fail-on-gap`, or a baseline that no longer holds |
98
- | `2` | Could not run: bad path, unreadable key list, unparseable JSON |
99
-
100
- Diagnostics go to **stderr**, results to **stdout**, so `--json` stays pipeable.
101
-
102
- ## Who this is for
103
-
104
- | If you run | Use it to check |
105
- |---|---|
106
- | RAG or retrieval pipelines | The retrieved set against the documents the question spans |
107
- | Agentic code review in CI | Files reviewed against `git diff --name-only` |
108
- | Batch or ETL jobs | Records processed against records declared |
109
- | Eval harnesses | Cases executed against cases declared |
110
- | Compliance evidence collection | Controls with evidence against controls in scope |
111
- | Research or thesis data | A folder that several people have been editing for months |
112
- | Any reporting series | Months, quarters, weeks, days, or invoice numbers with a hole in them |
113
-
114
- ## What it will not do
115
-
116
- - **It will not invent your expected set.** `diff` takes your declaration; `check` derives one from
117
- filenames and prints how. Both are arguable on purpose.
118
- - **It does not read file contents** except to profile a baseline you asked for.
119
- - **It never sends anything anywhere.** No network calls, no telemetry, no keys.
120
- - **No model decides any of it.** See [`assurance-core`](https://pypi.org/project/assurance-core/)
121
- for the arithmetic; this package owns all filesystem I/O.
122
-
123
- ## Also in this family
124
-
125
- - **[assurance-core](https://pypi.org/project/assurance-core/)** — the pure decision modules, zero dependencies
126
- - **[assurance-mcp](https://pypi.org/project/assurance-mcp/)** — the same checks as MCP tools any agent can call
127
-
128
- ## Licence
129
-
130
- Apache-2.0. See [LICENSE](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE).
131
- Upstream is [I-Ops](https://i-ops.dev); this repo is a publication, never a source.
@@ -1,158 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: assurance-cli
3
- Version: 0.2.0
4
- Summary: CLI for folder assurance checks — coverage, staleness, and baselines.
5
- Author-email: "I-Ops Operations Intelligence, LLC" <hello@i-ops.dev>
6
- License: Apache-2.0
7
- Project-URL: Homepage, https://i-ops.dev
8
- Project-URL: Source, https://github.com/i-ops-hq/assurance-cli
9
- Keywords: cli,ai-agents,governance,auditing,assurance,reproducibility
10
- Classifier: Development Status :: 4 - Beta
11
- Classifier: Intended Audience :: Developers
12
- Classifier: License :: OSI Approved :: Apache Software License
13
- Classifier: Programming Language :: Python :: 3
14
- Classifier: Programming Language :: Python :: 3.10
15
- Classifier: Programming Language :: Python :: 3.11
16
- Classifier: Programming Language :: Python :: 3.12
17
- Classifier: Programming Language :: Python :: 3.13
18
- Classifier: Topic :: Software Development :: Quality Assurance
19
- Requires-Python: >=3.10
20
- Description-Content-Type: text/markdown
21
- License-File: LICENSE
22
- Requires-Dist: assurance-core>=0.3
23
- Requires-Dist: openpyxl>=3.1
24
- Provides-Extra: dev
25
- Requires-Dist: pytest>=8.0; extra == "dev"
26
- Dynamic: license-file
27
-
28
- # assurance-cli
29
-
30
- [![PyPI](https://img.shields.io/pypi/v/assurance-cli)](https://pypi.org/project/assurance-cli/)
31
- [![Tests](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml/badge.svg)](https://github.com/i-ops-hq/assurance-cli/actions/workflows/tests.yml)
32
- [![Python](https://img.shields.io/pypi/pyversions/assurance-cli)](https://pypi.org/project/assurance-cli/)
33
- [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE)
34
-
35
- ### Did the job cover everything it was supposed to cover?
36
-
37
- One command, one honest ratio, and an exit code your pipeline can act on. No model, no API key, no
38
- network call, nothing uploaded.
39
-
40
- ```bash
41
- pip install assurance-cli
42
- ```
43
-
44
- ## `assurance diff` — coverage over any two sets
45
-
46
- The general form. Give it what a task **required** and what it **actually read**; it tells you the
47
- difference and exits non-zero on a gap. Keys are anything you can name.
48
-
49
- ```bash
50
- # A retrieval agent: did the retriever see the whole question?
51
- assurance diff \
52
- --expected corpus-for-this-question.txt \
53
- --found retrieved.json \
54
- --scope "documents the question spans" \
55
- --where "the retrieved set" \
56
- --fail-on-gap
57
- ```
58
-
59
- ```
60
- 2 of 5 documents the question spans — not in the retrieved set: doc-2, doc-3, doc-5
61
- also present and not expected: doc-9
62
- ```
63
-
64
- That last line matters as much as the ratio: the retriever drew on a document the scope never
65
- allowed. It is reported, and it earns no credit against the denominator.
66
-
67
- **Inputs are whatever you already have** — a file with one key per line, a JSON array of strings or
68
- of objects with a `key`/`id`/`name`/`path` field, `-` for stdin, or a comma-separated list inline.
69
-
70
- ```bash
71
- # Gate an agentic code review on having actually read the diff
72
- git diff --name-only origin/main...HEAD > changed.txt
73
- assurance diff --expected changed.txt --found reviewed.txt \
74
- --scope "files changed in this pull request" --where "the review log" --fail-on-gap
75
-
76
- # Did the eval suite run every declared case?
77
- assurance diff --expected cases.json --found ran.json --scope "declared eval cases" --fail-on-gap
78
-
79
- # Were all the partitions loaded?
80
- aws s3 ls s3://lake/dt=2026-08-14/ | awk '{print $4}' > loaded.txt
81
- assurance diff --expected expected-partitions.txt --found loaded.txt --where "the warehouse"
82
-
83
- # Straight from a pipe
84
- retriever --query "$Q" --json | jq -r '.chunks[].doc_id' | \
85
- assurance diff --expected corpus.txt --found - --json
86
- ```
87
-
88
- `--json` gives you the full record for a CI artefact: `complete`, `read` of `required`, and each way
89
- an expectation failed to be evidence kept separate.
90
-
91
- ## `assurance check` — coverage over a folder of dated files
92
-
93
- When the thing you must account for is a series on disk, the expected set is derived for you from the
94
- filenames. Monthly, quarterly, weekly, daily, or plain numbered.
95
-
96
- ```bash
97
- assurance check ~/reports
98
- # 22 of 24 months from 01/2024 to 12/2025 in reports — not in this folder: 03/2025, 07/2025
99
-
100
- assurance check ~/reports --from 2024-01 --to 2025-12 --fail-on-gap
101
- assurance check ~/invoices --expect numbered # gaps in INV-0001..INV-0450
102
- ```
103
-
104
- The derivation is printed with the ratio, so you can disagree with the **denominator** rather than
105
- only with the result. That is deliberate: a denominator a tool invents for you is a denominator
106
- nobody can argue with.
107
-
108
- ## `assurance init` / `--against-baseline` — did anything change underneath?
109
-
110
- Write a baseline of a folder, then ask later whether it still holds. Catches the file that was
111
- quietly replaced, the row count that moved, the figure that no longer matches its source.
112
-
113
- ```bash
114
- assurance init ~/thesis-data
115
- # ... weeks pass, several people touch the folder ...
116
- assurance check ~/thesis-data --against-baseline
117
- ```
118
-
119
- ## Exit codes
120
-
121
- | Code | Meaning |
122
- |---|---|
123
- | `0` | Ran, and either found no gap or was not asked to fail on one |
124
- | `1` | A finding — a coverage gap with `--fail-on-gap`, or a baseline that no longer holds |
125
- | `2` | Could not run: bad path, unreadable key list, unparseable JSON |
126
-
127
- Diagnostics go to **stderr**, results to **stdout**, so `--json` stays pipeable.
128
-
129
- ## Who this is for
130
-
131
- | If you run | Use it to check |
132
- |---|---|
133
- | RAG or retrieval pipelines | The retrieved set against the documents the question spans |
134
- | Agentic code review in CI | Files reviewed against `git diff --name-only` |
135
- | Batch or ETL jobs | Records processed against records declared |
136
- | Eval harnesses | Cases executed against cases declared |
137
- | Compliance evidence collection | Controls with evidence against controls in scope |
138
- | Research or thesis data | A folder that several people have been editing for months |
139
- | Any reporting series | Months, quarters, weeks, days, or invoice numbers with a hole in them |
140
-
141
- ## What it will not do
142
-
143
- - **It will not invent your expected set.** `diff` takes your declaration; `check` derives one from
144
- filenames and prints how. Both are arguable on purpose.
145
- - **It does not read file contents** except to profile a baseline you asked for.
146
- - **It never sends anything anywhere.** No network calls, no telemetry, no keys.
147
- - **No model decides any of it.** See [`assurance-core`](https://pypi.org/project/assurance-core/)
148
- for the arithmetic; this package owns all filesystem I/O.
149
-
150
- ## Also in this family
151
-
152
- - **[assurance-core](https://pypi.org/project/assurance-core/)** — the pure decision modules, zero dependencies
153
- - **[assurance-mcp](https://pypi.org/project/assurance-mcp/)** — the same checks as MCP tools any agent can call
154
-
155
- ## Licence
156
-
157
- Apache-2.0. See [LICENSE](https://github.com/i-ops-hq/assurance-cli/blob/main/LICENSE).
158
- Upstream is [I-Ops](https://i-ops.dev); this repo is a publication, never a source.
File without changes
File without changes