fairmind-cli 0.3.1a0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,53 @@
1
+ Metadata-Version: 2.4
2
+ Name: fairmind-cli
3
+ Version: 0.3.1a0
4
+ Summary: Run FairMind's release gate and fairness tests from a terminal or CI.
5
+ Author-email: Adhithya Rajasekaran <adhithya@axonome.xyz>
6
+ License: Apache-2.0
7
+ Requires-Python: >=3.10
8
+ Description-Content-Type: text/markdown
9
+ Requires-Dist: httpx>=0.27
10
+ Provides-Extra: dev
11
+ Requires-Dist: pytest>=7.4; extra == "dev"
12
+
13
+ # fairmind CLI
14
+
15
+ The release gate and fairness tests from a terminal or a CI job. One dependency, `httpx`.
16
+
17
+ ```bash
18
+ pipx install "git+https://github.com/adhit-r/fairmind.git#subdirectory=apps/cli"
19
+ # from a checkout of this repository: pipx install ./apps/cli
20
+ export FAIRMIND_API_URL=https://fairmind.example.com
21
+ export FAIRMIND_API_KEY=... # minted at POST /api/v1/auth/api-keys, valid 365 days
22
+ ```
23
+
24
+ ## Commands
25
+
26
+ ```bash
27
+ fairmind systems
28
+ fairmind gate --system <id> # exit 0 go, 2 conditional go, 1 no go
29
+ fairmind gate --system <id> --allow-conditional # exit 0 on conditional go
30
+ fairmind gate --system <id> --require-approval # exit 1 unless the release decision is approved
31
+ fairmind test --system <id> --file predictions.csv \
32
+ --outcome repaid --prediction approved --score score --attributes gender,region
33
+ fairmind test ... --gate # exit by the gate's verdict after the run
34
+ ```
35
+
36
+ `test` uploads the CSV, runs disparate impact, demographic parity, equal opportunity, equalized odds and calibration per attribute with bootstrap intervals, and stores the result as measured evidence on the system. `gate` reads the release gate: five checks, each proven, attested or open, a confidence, the verdict from the impact-by-confidence matrix, and the latest release decision (`approvalStatus`: not requested, pending, approved or rejected). A rejected decision makes the verdict no go until a reviewer approves a new request. With `--require-approval` an approval with conditions counts as approved: an approved conditional go exits 0.
37
+
38
+ Exit code 3 means the command itself failed: no network, no key, wrong system id, a column that is not in the file. Add `--json` to any command for the raw response.
39
+
40
+ ## In GitHub Actions
41
+
42
+ ```yaml
43
+ - uses: adhit-r/fairmind/.github/actions/fairmind-gate@main # in this repository: ./.github/actions/fairmind-gate
44
+ with:
45
+ system: ${{ vars.FAIRMIND_SYSTEM_ID }}
46
+ api-url: ${{ vars.FAIRMIND_API_URL }}
47
+ api-key: ${{ secrets.FAIRMIND_API_KEY }}
48
+ predictions: artifacts/predictions.csv # optional: run the test first
49
+ outcome: repaid
50
+ prediction: approved
51
+ attributes: gender,region
52
+ require-approval: "true" # optional: fail unless the release decision is approved
53
+ ```
@@ -0,0 +1,41 @@
1
+ # fairmind CLI
2
+
3
+ The release gate and fairness tests from a terminal or a CI job. One dependency, `httpx`.
4
+
5
+ ```bash
6
+ pipx install "git+https://github.com/adhit-r/fairmind.git#subdirectory=apps/cli"
7
+ # from a checkout of this repository: pipx install ./apps/cli
8
+ export FAIRMIND_API_URL=https://fairmind.example.com
9
+ export FAIRMIND_API_KEY=... # minted at POST /api/v1/auth/api-keys, valid 365 days
10
+ ```
11
+
12
+ ## Commands
13
+
14
+ ```bash
15
+ fairmind systems
16
+ fairmind gate --system <id> # exit 0 go, 2 conditional go, 1 no go
17
+ fairmind gate --system <id> --allow-conditional # exit 0 on conditional go
18
+ fairmind gate --system <id> --require-approval # exit 1 unless the release decision is approved
19
+ fairmind test --system <id> --file predictions.csv \
20
+ --outcome repaid --prediction approved --score score --attributes gender,region
21
+ fairmind test ... --gate # exit by the gate's verdict after the run
22
+ ```
23
+
24
+ `test` uploads the CSV, runs disparate impact, demographic parity, equal opportunity, equalized odds and calibration per attribute with bootstrap intervals, and stores the result as measured evidence on the system. `gate` reads the release gate: five checks, each proven, attested or open, a confidence, the verdict from the impact-by-confidence matrix, and the latest release decision (`approvalStatus`: not requested, pending, approved or rejected). A rejected decision makes the verdict no go until a reviewer approves a new request. With `--require-approval` an approval with conditions counts as approved: an approved conditional go exits 0.
25
+
26
+ Exit code 3 means the command itself failed: no network, no key, wrong system id, a column that is not in the file. Add `--json` to any command for the raw response.
27
+
28
+ ## In GitHub Actions
29
+
30
+ ```yaml
31
+ - uses: adhit-r/fairmind/.github/actions/fairmind-gate@main # in this repository: ./.github/actions/fairmind-gate
32
+ with:
33
+ system: ${{ vars.FAIRMIND_SYSTEM_ID }}
34
+ api-url: ${{ vars.FAIRMIND_API_URL }}
35
+ api-key: ${{ secrets.FAIRMIND_API_KEY }}
36
+ predictions: artifacts/predictions.csv # optional: run the test first
37
+ outcome: repaid
38
+ prediction: approved
39
+ attributes: gender,region
40
+ require-approval: "true" # optional: fail unless the release decision is approved
41
+ ```
@@ -0,0 +1,3 @@
1
+ """FairMind command line: the release gate and fairness tests, scriptable."""
2
+
3
+ __version__ = "0.3.1a0"
@@ -0,0 +1,361 @@
1
+ """fairmind: gate, test and systems.
2
+
3
+ fairmind gate --system <id> exit 0 go, 2 conditional go, 1 no go
4
+ fairmind gate --system <id> --require-approval exit 1 unless the release decision is approved; approved, a conditional go exits 0
5
+ fairmind test --system <id> --file p.csv \
6
+ --outcome repaid --attributes gender,region [--prediction approved] [--score score]
7
+ fairmind monitor --system <id> --file decisions.csv --attributes gender [--prediction approved] [--score score]
8
+ fairmind systems
9
+ fairmind verify pack.json offline: recompute the pack hash and chain heads; exit 0 intact, 1 not
10
+
11
+ Configuration comes from flags or the environment: FAIRMIND_API_URL (default
12
+ http://localhost:8010) and FAIRMIND_API_KEY, a key minted at POST /api/v1/auth/api-keys.
13
+ Exit code 3 means the command itself failed (network, auth, bad input).
14
+ """
15
+
16
+ from __future__ import annotations
17
+
18
+ import argparse
19
+ import hashlib
20
+ import hmac
21
+ import json
22
+ import os
23
+ import sys
24
+ from pathlib import Path
25
+ from typing import Any, Callable, Dict, Optional, Sequence
26
+
27
+ import httpx
28
+
29
+ from . import __version__
30
+
31
+ GOV = "/api/v1/ai-governance"
32
+ EXIT_GO, EXIT_NO_GO, EXIT_CONDITIONAL, EXIT_ERROR = 0, 1, 2, 3
33
+
34
+ ClientFactory = Callable[[str, Dict[str, str]], httpx.Client]
35
+
36
+
37
+ class CliError(Exception):
38
+ """Reported to stderr with exit code 3."""
39
+
40
+
41
+ def _default_client(base_url: str, headers: Dict[str, str]) -> httpx.Client:
42
+ return httpx.Client(base_url=base_url, headers=headers, timeout=httpx.Timeout(180.0, connect=10.0))
43
+
44
+
45
+ def _client(args: argparse.Namespace, factory: ClientFactory) -> httpx.Client:
46
+ base_url = (args.api or os.environ.get("FAIRMIND_API_URL") or "http://localhost:8010").rstrip("/")
47
+ key = args.api_key or os.environ.get("FAIRMIND_API_KEY") or ""
48
+ headers = {"Accept": "application/json", "User-Agent": f"fairmind-cli/{__version__}"}
49
+ if key:
50
+ headers["Authorization"] = f"Bearer {key}"
51
+ return factory(base_url, headers)
52
+
53
+
54
+ def _call(client: httpx.Client, method: str, path: str, **kwargs: Any) -> Any:
55
+ try:
56
+ response = client.request(method, path, **kwargs)
57
+ except httpx.HTTPError as exc:
58
+ raise CliError(f"Could not reach FairMind at {client.base_url}: {exc}") from exc
59
+ if response.status_code == 401:
60
+ raise CliError("Not signed in. Set FAIRMIND_API_KEY to a key from POST /api/v1/auth/api-keys.")
61
+ if response.status_code == 404:
62
+ raise CliError(f"Not found: {path}. Check the system id and that your key belongs to its organisation.")
63
+ if response.status_code >= 400:
64
+ try:
65
+ detail = response.json().get("detail") or response.json().get("message") or response.text
66
+ except ValueError:
67
+ detail = response.text
68
+ raise CliError(f"{method} {path} failed with {response.status_code}: {detail}")
69
+ try:
70
+ return response.json()
71
+ except ValueError as exc:
72
+ raise CliError(f"{method} {path} returned something that is not JSON") from exc
73
+
74
+
75
+ # ---------------------------------------------------------------------------
76
+ # gate
77
+ # ---------------------------------------------------------------------------
78
+
79
+ VERDICT_WORD = {"go": "GO", "conditional_go": "CONDITIONAL GO", "no_go": "NO GO"}
80
+ STATE_MARK = {"proven": "+", "attested": "~", "open": "-"}
81
+ APPROVAL_WORD = {"not_requested": "not requested", "pending": "pending", "approved": "approved", "rejected": "rejected"}
82
+
83
+
84
+ def render_gate(gate: Dict[str, Any]) -> str:
85
+ lines = [
86
+ f"{VERDICT_WORD.get(gate['verdict'], gate['verdict'])} confidence {gate['confidence']:.2f} ({gate['band']}) "
87
+ f"{gate['proven']} proven, {gate['attested']} attested, {gate['open']} open",
88
+ "",
89
+ ]
90
+ for check in gate.get("checks", []):
91
+ conf = f" {check['confidence']:.2f}" if check.get("confidence") else ""
92
+ lines.append(f" {STATE_MARK.get(check['state'], '?')} {check['label']:<22} {check['state']}{conf}")
93
+ if "approvalStatus" in gate:
94
+ lines.append(f" {'Release decision':<22} {APPROVAL_WORD.get(gate['approvalStatus'], gate['approvalStatus'])}")
95
+ if gate.get("reasons"):
96
+ lines.append("")
97
+ lines.extend(f" {reason}" for reason in gate["reasons"])
98
+ return "\n".join(lines)
99
+
100
+
101
+ def cmd_gate(args: argparse.Namespace, factory: ClientFactory, out: Callable[[str], None]) -> int:
102
+ with _client(args, factory) as client:
103
+ gate = _call(client, "GET", f"{GOV}/systems/{args.system}/gate")
104
+ out(json.dumps(gate, indent=2) if args.json else render_gate(gate))
105
+ if args.require_approval and gate.get("approvalStatus") != "approved":
106
+ status = APPROVAL_WORD.get(gate.get("approvalStatus"), "not reported by this server")
107
+ print(f"fairmind: --require-approval: the release decision is {status}", file=sys.stderr)
108
+ return EXIT_NO_GO
109
+ verdict = gate.get("verdict")
110
+ if verdict == "go":
111
+ return EXIT_GO
112
+ if verdict == "conditional_go":
113
+ # Past --require-approval the decision is approved, and approving a conditional go records the accepted conditions.
114
+ return EXIT_GO if args.allow_conditional or args.require_approval else EXIT_CONDITIONAL
115
+ return EXIT_NO_GO
116
+
117
+
118
+ # ---------------------------------------------------------------------------
119
+ # test
120
+ # ---------------------------------------------------------------------------
121
+
122
+ def render_run(run: Dict[str, Any]) -> str:
123
+ result = run["result"]
124
+ lines = [
125
+ f"{'All checks within thresholds' if result['all_checks_pass'] else 'At least one check outside thresholds'} "
126
+ f"{result['rows']} rows evidence confidence {result['evidence_confidence']:.2f} (measured) evidence {run['evidenceId']}",
127
+ "",
128
+ f" {'attribute':<14}{'impact ratio':>13}{'eq. opp. gap':>14}{'eq. odds gap':>14}{'calibration':>13}",
129
+ ]
130
+ for attr in result["attributes"]:
131
+ checks = attr["checks"]
132
+ def cell(value: Optional[float], ok: Optional[bool], width: int) -> str:
133
+ if value is None:
134
+ return f"{'n/a':>{width}}"
135
+ mark = "" if ok else "!"
136
+ return f"{value:.3f}{mark}".rjust(width)
137
+ cal = attr.get("calibration")
138
+ lines.append(
139
+ f" {attr['attribute']:<14}"
140
+ + cell(attr["disparate_impact_ratio"], checks.get("disparate_impact"), 13)
141
+ + cell(attr["equal_opportunity"]["value"], checks.get("equal_opportunity"), 14)
142
+ + cell(attr["equalized_odds"]["value"], checks.get("equalized_odds"), 14)
143
+ + cell(cal["value"] if cal else None, checks.get("calibration"), 13)
144
+ )
145
+ if attr.get("small_groups"):
146
+ lines.append(f" {'':<14}small groups: {', '.join(attr['small_groups'])}")
147
+ lines.append("")
148
+ lines.append(" ! marks a check outside its threshold (impact ratio below 0.80, gap above 0.05)")
149
+ lines.append("")
150
+ lines.append(render_gate(run["gate"]))
151
+ return "\n".join(lines)
152
+
153
+
154
+ def cmd_test(args: argparse.Namespace, factory: ClientFactory, out: Callable[[str], None]) -> int:
155
+ path = Path(args.file)
156
+ if not path.is_file():
157
+ raise CliError(f"No such file: {path}")
158
+ data = {"outcome": args.outcome, "attributes": args.attributes}
159
+ if args.prediction:
160
+ data["prediction"] = args.prediction
161
+ if args.score:
162
+ data["score"] = args.score
163
+ if args.favorable_label:
164
+ data["favorable_label"] = args.favorable_label
165
+ if not args.prediction and not args.score:
166
+ raise CliError("Give --prediction (a 0/1 column) or --score (a 0 to 1 column)")
167
+ with _client(args, factory) as client, path.open("rb") as handle:
168
+ run = _call(client, "POST", f"{GOV}/systems/{args.system}/fairness-runs", data=data, files={"file": (path.name, handle, "text/csv")})
169
+ out(json.dumps(run, indent=2) if args.json else render_run(run))
170
+ if args.gate:
171
+ verdict = run["gate"]["verdict"]
172
+ if verdict == "go":
173
+ return EXIT_GO
174
+ if verdict == "conditional_go":
175
+ return EXIT_GO if args.allow_conditional else EXIT_CONDITIONAL
176
+ return EXIT_NO_GO
177
+ return EXIT_GO if run["result"]["all_checks_pass"] else EXIT_NO_GO
178
+
179
+
180
+ # ---------------------------------------------------------------------------
181
+ # monitor
182
+ # ---------------------------------------------------------------------------
183
+
184
+ def render_monitoring(run: Dict[str, Any]) -> str:
185
+ result = run["result"]
186
+ base = result["baseline"]
187
+ lines = [
188
+ f"{'Stable against the baseline' if result['all_checks_pass'] else 'Drift or disparity outside thresholds'} "
189
+ f"{result['rows']} rows overall score PSI {('%.3f' % result['overall_psi']) if result.get('overall_psi') is not None else 'n/a'} "
190
+ f"baseline {'present' if base.get('present') else 'none (run a fairness test first)'} evidence {run['evidenceId']}",
191
+ "",
192
+ f" {'attribute':<14}{'max rate drift':>15}{'impact ratio':>13}{'worst PSI':>11}",
193
+ ]
194
+ for attr in result["attributes"]:
195
+ checks = attr["checks"]
196
+ def cell(value: Optional[float], ok: Optional[bool], width: int) -> str:
197
+ if value is None:
198
+ return f"{'n/a':>{width}}"
199
+ return f"{value:.3f}{'' if ok in (None, True) else '!'}".rjust(width)
200
+ lines.append(f" {attr['attribute']:<14}" + cell(attr.get("max_rate_drift"), checks.get("selection_rate_stable"), 15) + cell(attr["disparate_impact_ratio"], checks.get("disparate_impact"), 13) + cell(attr.get("worst_psi"), checks.get("score_stable"), 11))
201
+ lines += ["", " ! marks a check outside its threshold (rate drift above 0.05, impact ratio below 0.80, PSI at or above 0.10)", "", render_gate(run["gate"])]
202
+ return "\n".join(lines)
203
+
204
+
205
+ def cmd_monitor(args: argparse.Namespace, factory: ClientFactory, out: Callable[[str], None]) -> int:
206
+ path = Path(args.file)
207
+ if not path.is_file():
208
+ raise CliError(f"No such file: {path}")
209
+ if not args.prediction and not args.score:
210
+ raise CliError("Give --prediction (a 0/1 column) or --score (a 0 to 1 column)")
211
+ data = {"attributes": args.attributes}
212
+ if args.prediction:
213
+ data["prediction"] = args.prediction
214
+ if args.score:
215
+ data["score"] = args.score
216
+ with _client(args, factory) as client, path.open("rb") as handle:
217
+ run = _call(client, "POST", f"{GOV}/systems/{args.system}/monitoring-runs", data=data, files={"file": (path.name, handle, "text/csv")})
218
+ out(json.dumps(run, indent=2) if args.json else render_monitoring(run))
219
+ return EXIT_GO if run["result"]["all_checks_pass"] else EXIT_NO_GO
220
+
221
+
222
+ # ---------------------------------------------------------------------------
223
+ # systems
224
+ # ---------------------------------------------------------------------------
225
+
226
+ def cmd_systems(args: argparse.Namespace, factory: ClientFactory, out: Callable[[str], None]) -> int:
227
+ with _client(args, factory) as client:
228
+ systems = _call(client, "GET", f"{GOV}/systems")
229
+ if args.json:
230
+ out(json.dumps(systems, indent=2))
231
+ return EXIT_GO
232
+ if not systems:
233
+ out("No systems in your organisation yet. Connect a source or register one in FairMind.")
234
+ return EXIT_GO
235
+ width = max(len(s.get("name", "")) for s in systems)
236
+ out(f" {'name':<{width}} {'tier':<9}{'stage':<10}id")
237
+ for s in systems:
238
+ out(f" {s.get('name', ''):<{width}} {s.get('riskTier', ''):<9}{s.get('lifecycleStage', ''):<10}{s.get('id', '')}")
239
+ return EXIT_GO
240
+
241
+
242
+ # ---------------------------------------------------------------------------
243
+ # verify (offline)
244
+ # ---------------------------------------------------------------------------
245
+
246
+ CANONICAL_FORM = (
247
+ 'Canonical form: sha256(json.dumps(manifest, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()). '
248
+ 'Chain head per system: h = sha256("fairmind-evidence-chain-v1"); for each evidence entry in order, '
249
+ 'h = sha256(f"{h}:{entry.hash}"). The file is {"manifest": {...}, "manifestHash": "..."}.'
250
+ )
251
+
252
+
253
+ def _chain_head(entries: Sequence[Dict[str, Any]]) -> str:
254
+ head = hashlib.sha256(b"fairmind-evidence-chain-v1").hexdigest()
255
+ for entry in entries:
256
+ head = hashlib.sha256(f"{head}:{entry['hash']}".encode()).hexdigest()
257
+ return head
258
+
259
+
260
+ def verify_pack(document: Dict[str, Any]) -> Dict[str, Any]:
261
+ """Pure check of a downloaded pack; no network."""
262
+ manifest, expected = document.get("manifest"), document.get("manifestHash")
263
+ if not isinstance(manifest, dict) or not isinstance(expected, str):
264
+ raise CliError('Not an inspection pack: expected {"manifest": {...}, "manifestHash": "..."}')
265
+ actual = hashlib.sha256(json.dumps(manifest, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest()
266
+ systems = []
267
+ for item in manifest.get("systems", []):
268
+ try:
269
+ recomputed = _chain_head(item.get("evidence", []))
270
+ except (KeyError, TypeError) as exc:
271
+ raise CliError(f"Malformed evidence entry: {exc!r}") from exc
272
+ system = item.get("system") or {}
273
+ systems.append({"name": system.get("name") or system.get("id"), "evidence": len(item.get("evidence", [])), "intact": hmac.compare_digest(str(item.get("chainHead")), recomputed)})
274
+ hash_ok = hmac.compare_digest(expected, actual)
275
+ return {"title": manifest.get("title"), "hashOk": hash_ok, "expected": expected, "actual": actual, "systems": systems, "intact": hash_ok and all(s["intact"] for s in systems)}
276
+
277
+
278
+ def cmd_verify(args: argparse.Namespace, factory: ClientFactory, out: Callable[[str], None]) -> int:
279
+ path = Path(args.pack)
280
+ if not path.is_file():
281
+ raise CliError(f"No such file: {path}")
282
+ try:
283
+ document = json.loads(path.read_text(encoding="utf-8"))
284
+ except ValueError as exc:
285
+ raise CliError(f"{path} is not valid JSON: {exc}") from exc
286
+ result = verify_pack(document)
287
+ if args.json:
288
+ out(json.dumps(result, indent=2))
289
+ else:
290
+ out(f"{result['title']}")
291
+ out(f" manifest hash {'ok' if result['hashOk'] else 'MISMATCH'} {result['actual']}")
292
+ if not result["hashOk"]:
293
+ out(f" recorded {result['expected']}")
294
+ for s in result["systems"]:
295
+ out(f" {'ok' if s['intact'] else 'BROKEN':<9}{s['name']} {s['evidence']} evidence records, chain head {'matches' if s['intact'] else 'does not match'}")
296
+ out("")
297
+ out("Pack intact" if result["intact"] else "Pack does NOT verify: it was changed after it was built")
298
+ return EXIT_GO if result["intact"] else EXIT_NO_GO
299
+
300
+
301
+ # ---------------------------------------------------------------------------
302
+ # entry point
303
+ # ---------------------------------------------------------------------------
304
+
305
+ def build_parser() -> argparse.ArgumentParser:
306
+ parser = argparse.ArgumentParser(prog="fairmind", description="FairMind release gate and fairness tests.")
307
+ parser.add_argument("--version", action="version", version=f"fairmind-cli {__version__}")
308
+ common = argparse.ArgumentParser(add_help=False)
309
+ common.add_argument("--api", help="FairMind API base URL (default: FAIRMIND_API_URL or http://localhost:8010)")
310
+ common.add_argument("--api-key", help="API key (default: FAIRMIND_API_KEY)")
311
+ common.add_argument("--json", action="store_true", help="print the raw response")
312
+
313
+ sub = parser.add_subparsers(dest="command", required=True)
314
+
315
+ gate = sub.add_parser("gate", parents=[common], help="read the release gate; exit 0 go, 2 conditional, 1 no go")
316
+ gate.add_argument("--system", required=True, help="AI system id")
317
+ gate.add_argument("--allow-conditional", action="store_true", help="exit 0 on a conditional go")
318
+ gate.add_argument("--require-approval", action="store_true", help="exit 1 unless the release decision is approved (with or without conditions)")
319
+ gate.set_defaults(func=cmd_gate)
320
+
321
+ test = sub.add_parser("test", parents=[common], help="run a fairness test from a predictions CSV and store it as evidence")
322
+ test.add_argument("--system", required=True, help="AI system id")
323
+ test.add_argument("--file", required=True, help="CSV with one row per decision")
324
+ test.add_argument("--outcome", required=True, help="column with what happened (0/1)")
325
+ test.add_argument("--attributes", required=True, help="comma-separated protected attribute columns")
326
+ test.add_argument("--prediction", help="column with the model's decision (0/1)")
327
+ test.add_argument("--score", help="column with the model's score (0 to 1)")
328
+ test.add_argument("--favorable-label", help="value that counts as the favourable outcome when columns are not 0/1")
329
+ test.add_argument("--gate", action="store_true", help="exit by the release gate's verdict instead of the test's checks")
330
+ test.add_argument("--allow-conditional", action="store_true", help="with --gate: exit 0 on a conditional go")
331
+ test.set_defaults(func=cmd_test)
332
+
333
+ monitor = sub.add_parser("monitor", parents=[common], help="record a monitoring snapshot from recent decisions; exit 1 on drift or disparity")
334
+ monitor.add_argument("--system", required=True, help="AI system id")
335
+ monitor.add_argument("--file", required=True, help="CSV of recent decisions")
336
+ monitor.add_argument("--attributes", required=True, help="comma-separated protected attribute columns")
337
+ monitor.add_argument("--prediction", help="column with the model's decision (0/1)")
338
+ monitor.add_argument("--score", help="column with the model's score (0 to 1)")
339
+ monitor.set_defaults(func=cmd_monitor)
340
+
341
+ systems = sub.add_parser("systems", parents=[common], help="list the AI systems your key can see")
342
+ systems.set_defaults(func=cmd_systems)
343
+
344
+ verify = sub.add_parser("verify", help="verify a downloaded inspection pack offline (no network); exit 0 intact, 1 not", description=CANONICAL_FORM)
345
+ verify.add_argument("pack", help="pack JSON downloaded from FairMind or an auditor link")
346
+ verify.add_argument("--json", action="store_true", help="print the result as JSON")
347
+ verify.set_defaults(func=cmd_verify)
348
+ return parser
349
+
350
+
351
+ def main(argv: Optional[Sequence[str]] = None, *, client_factory: ClientFactory = _default_client, out: Callable[[str], None] = print) -> int:
352
+ args = build_parser().parse_args(argv)
353
+ try:
354
+ return int(args.func(args, client_factory, out))
355
+ except CliError as exc:
356
+ print(f"fairmind: {exc}", file=sys.stderr)
357
+ return EXIT_ERROR
358
+
359
+
360
+ if __name__ == "__main__": # pragma: no cover
361
+ sys.exit(main())
@@ -0,0 +1,53 @@
1
+ Metadata-Version: 2.4
2
+ Name: fairmind-cli
3
+ Version: 0.3.1a0
4
+ Summary: Run FairMind's release gate and fairness tests from a terminal or CI.
5
+ Author-email: Adhithya Rajasekaran <adhithya@axonome.xyz>
6
+ License: Apache-2.0
7
+ Requires-Python: >=3.10
8
+ Description-Content-Type: text/markdown
9
+ Requires-Dist: httpx>=0.27
10
+ Provides-Extra: dev
11
+ Requires-Dist: pytest>=7.4; extra == "dev"
12
+
13
+ # fairmind CLI
14
+
15
+ The release gate and fairness tests from a terminal or a CI job. One dependency, `httpx`.
16
+
17
+ ```bash
18
+ pipx install "git+https://github.com/adhit-r/fairmind.git#subdirectory=apps/cli"
19
+ # from a checkout of this repository: pipx install ./apps/cli
20
+ export FAIRMIND_API_URL=https://fairmind.example.com
21
+ export FAIRMIND_API_KEY=... # minted at POST /api/v1/auth/api-keys, valid 365 days
22
+ ```
23
+
24
+ ## Commands
25
+
26
+ ```bash
27
+ fairmind systems
28
+ fairmind gate --system <id> # exit 0 go, 2 conditional go, 1 no go
29
+ fairmind gate --system <id> --allow-conditional # exit 0 on conditional go
30
+ fairmind gate --system <id> --require-approval # exit 1 unless the release decision is approved
31
+ fairmind test --system <id> --file predictions.csv \
32
+ --outcome repaid --prediction approved --score score --attributes gender,region
33
+ fairmind test ... --gate # exit by the gate's verdict after the run
34
+ ```
35
+
36
+ `test` uploads the CSV, runs disparate impact, demographic parity, equal opportunity, equalized odds and calibration per attribute with bootstrap intervals, and stores the result as measured evidence on the system. `gate` reads the release gate: five checks, each proven, attested or open, a confidence, the verdict from the impact-by-confidence matrix, and the latest release decision (`approvalStatus`: not requested, pending, approved or rejected). A rejected decision makes the verdict no go until a reviewer approves a new request. With `--require-approval` an approval with conditions counts as approved: an approved conditional go exits 0.
37
+
38
+ Exit code 3 means the command itself failed: no network, no key, wrong system id, a column that is not in the file. Add `--json` to any command for the raw response.
39
+
40
+ ## In GitHub Actions
41
+
42
+ ```yaml
43
+ - uses: adhit-r/fairmind/.github/actions/fairmind-gate@main # in this repository: ./.github/actions/fairmind-gate
44
+ with:
45
+ system: ${{ vars.FAIRMIND_SYSTEM_ID }}
46
+ api-url: ${{ vars.FAIRMIND_API_URL }}
47
+ api-key: ${{ secrets.FAIRMIND_API_KEY }}
48
+ predictions: artifacts/predictions.csv # optional: run the test first
49
+ outcome: repaid
50
+ prediction: approved
51
+ attributes: gender,region
52
+ require-approval: "true" # optional: fail unless the release decision is approved
53
+ ```
@@ -0,0 +1,11 @@
1
+ README.md
2
+ pyproject.toml
3
+ fairmind_cli/__init__.py
4
+ fairmind_cli/main.py
5
+ fairmind_cli.egg-info/PKG-INFO
6
+ fairmind_cli.egg-info/SOURCES.txt
7
+ fairmind_cli.egg-info/dependency_links.txt
8
+ fairmind_cli.egg-info/entry_points.txt
9
+ fairmind_cli.egg-info/requires.txt
10
+ fairmind_cli.egg-info/top_level.txt
11
+ tests/test_cli.py
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ fairmind = fairmind_cli.main:main
@@ -0,0 +1,4 @@
1
+ httpx>=0.27
2
+
3
+ [dev]
4
+ pytest>=7.4
@@ -0,0 +1 @@
1
+ fairmind_cli
@@ -0,0 +1,25 @@
1
+ [project]
2
+ name = "fairmind-cli"
3
+ version = "0.3.1a0"
4
+ description = "Run FairMind's release gate and fairness tests from a terminal or CI."
5
+ readme = "README.md"
6
+ requires-python = ">=3.10"
7
+ license = { text = "Apache-2.0" }
8
+ authors = [{ name = "Adhithya Rajasekaran", email = "adhithya@axonome.xyz" }]
9
+ dependencies = ["httpx>=0.27"]
10
+
11
+ [project.optional-dependencies]
12
+ dev = ["pytest>=7.4"]
13
+
14
+ [project.scripts]
15
+ fairmind = "fairmind_cli.main:main"
16
+
17
+ [build-system]
18
+ requires = ["setuptools>=68"]
19
+ build-backend = "setuptools.build_meta"
20
+
21
+ [tool.setuptools.packages.find]
22
+ include = ["fairmind_cli*"]
23
+
24
+ [tool.pytest.ini_options]
25
+ testpaths = ["tests"]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,203 @@
1
+ """The CLI's contract: exit codes follow the gate, uploads carry the mapping, errors are named."""
2
+
3
+ import json
4
+
5
+ import httpx
6
+ import pytest
7
+
8
+ from fairmind_cli.main import EXIT_CONDITIONAL, EXIT_ERROR, EXIT_GO, EXIT_NO_GO, main
9
+
10
+ GATE = {
11
+ "verdict": "conditional_go",
12
+ "confidence": 0.58,
13
+ "band": "estimated",
14
+ "proven": 1,
15
+ "attested": 2,
16
+ "open": 2,
17
+ "checks": [
18
+ {"key": "fairness", "label": "Fairness measured", "state": "proven", "confidence": 0.95, "evidenceId": "e1", "detail": None},
19
+ {"key": "monitoring", "label": "Monitoring in place", "state": "open", "confidence": 0.0, "evidenceId": None, "detail": None},
20
+ ],
21
+ "reasons": ["Nothing on file for: Monitoring in place"],
22
+ }
23
+
24
+
25
+ def factory_for(handler, seen):
26
+ def factory(base_url, headers):
27
+ seen["base_url"] = base_url
28
+ seen["headers"] = headers
29
+ return httpx.Client(base_url=base_url, headers=headers, transport=httpx.MockTransport(handler))
30
+ return factory
31
+
32
+
33
+ def run_cli(argv, handler, env=None, monkeypatch=None):
34
+ seen = {}
35
+ printed = []
36
+ if monkeypatch is not None:
37
+ for key in ("FAIRMIND_API_URL", "FAIRMIND_API_KEY"):
38
+ monkeypatch.delenv(key, raising=False)
39
+ for key, value in (env or {}).items():
40
+ monkeypatch.setenv(key, value)
41
+ code = main(argv, client_factory=factory_for(handler, seen), out=printed.append)
42
+ return code, "\n".join(printed), seen
43
+
44
+
45
+ @pytest.mark.parametrize("verdict,allow,expected", [
46
+ ("go", False, EXIT_GO),
47
+ ("conditional_go", False, EXIT_CONDITIONAL),
48
+ ("conditional_go", True, EXIT_GO),
49
+ ("no_go", True, EXIT_NO_GO),
50
+ ])
51
+ def test_gate_exit_codes(verdict, allow, expected, monkeypatch):
52
+ def handler(request):
53
+ assert request.url.path == "/api/v1/ai-governance/systems/sys-1/gate"
54
+ assert request.headers["Authorization"] == "Bearer key-123"
55
+ return httpx.Response(200, json={**GATE, "verdict": verdict})
56
+ argv = ["gate", "--system", "sys-1"] + (["--allow-conditional"] if allow else [])
57
+ code, text, seen = run_cli(argv, handler, env={"FAIRMIND_API_KEY": "key-123", "FAIRMIND_API_URL": "https://fm.example"}, monkeypatch=monkeypatch)
58
+ assert code == expected
59
+ assert seen["base_url"] == "https://fm.example"
60
+ assert "Fairness measured" in text and "Monitoring in place" in text
61
+
62
+
63
+ @pytest.mark.parametrize("approval,verdict,expected", [
64
+ ("approved", "go", EXIT_GO),
65
+ ("approved", "conditional_go", EXIT_GO),
66
+ ("pending", "go", EXIT_NO_GO),
67
+ ("not_requested", "go", EXIT_NO_GO),
68
+ ("rejected", "no_go", EXIT_NO_GO),
69
+ (None, "go", EXIT_NO_GO),
70
+ ])
71
+ def test_gate_require_approval(approval, verdict, expected, monkeypatch, capsys):
72
+ body = {**GATE, "verdict": verdict, **({"approvalStatus": approval} if approval else {})}
73
+ code, text, _ = run_cli(["gate", "--system", "sys-1", "--api-key", "k", "--require-approval"], lambda r: httpx.Response(200, json=body), monkeypatch=monkeypatch)
74
+ assert code == expected
75
+ if approval:
76
+ assert "Release decision" in text
77
+ if expected == EXIT_NO_GO and approval != "rejected":
78
+ assert "--require-approval" in capsys.readouterr().err
79
+
80
+
81
+ def test_gate_json_and_auth_errors(monkeypatch, capsys):
82
+ code, text, _ = run_cli(["gate", "--system", "sys-1", "--json", "--api-key", "k"], lambda r: httpx.Response(200, json=GATE), monkeypatch=monkeypatch)
83
+ assert code == EXIT_CONDITIONAL and json.loads(text)["verdict"] == "conditional_go"
84
+ code, _, _ = run_cli(["gate", "--system", "sys-1"], lambda r: httpx.Response(401, json={"detail": "nope"}), monkeypatch=monkeypatch)
85
+ assert code == EXIT_ERROR and "FAIRMIND_API_KEY" in capsys.readouterr().err
86
+ code, _, _ = run_cli(["gate", "--system", "missing", "--api-key", "k"], lambda r: httpx.Response(404, json={"detail": "AI system not found"}), monkeypatch=monkeypatch)
87
+ assert code == EXIT_ERROR
88
+
89
+
90
+ def test_test_uploads_the_mapping_and_follows_the_gate(tmp_path, monkeypatch):
91
+ csv = tmp_path / "preds.csv"
92
+ csv.write_text("gender,repaid,approved\nF,1,1\nM,0,0\n")
93
+ captured = {}
94
+
95
+ def handler(request):
96
+ captured["path"] = request.url.path
97
+ body = request.read()
98
+ captured["body"] = body
99
+ run = {
100
+ "runId": "r1", "evidenceId": "e1", "systemId": "sys-1", "title": "t",
101
+ "result": {
102
+ "rows": 2, "all_checks_pass": False, "evidence_confidence": 0.85, "smallest_group": 1,
103
+ "thresholds": {"disparate_impact_ratio": 0.8, "gap": 0.05},
104
+ "attributes": [{
105
+ "attribute": "gender", "groups": {"F": 1, "M": 1}, "selection_rates": {"F": 1.0, "M": 0.0},
106
+ "disparate_impact_ratio": 0.0,
107
+ "demographic_parity": {"value": 1.0, "confidence_interval": [1, 1], "is_fair": False},
108
+ "equal_opportunity": {"value": 0.0, "confidence_interval": [0, 0], "is_fair": True},
109
+ "equalized_odds": {"value": 0.0, "confidence_interval": [0, 0], "is_fair": True},
110
+ "calibration": None,
111
+ "checks": {"disparate_impact": False, "equal_opportunity": True, "equalized_odds": True},
112
+ "small_groups": ["F", "M"],
113
+ }],
114
+ },
115
+ "gate": {**GATE, "verdict": "no_go"},
116
+ }
117
+ return httpx.Response(200, json=run)
118
+
119
+ argv = ["test", "--system", "sys-1", "--file", str(csv), "--outcome", "repaid", "--prediction", "approved", "--attributes", "gender", "--api-key", "k"]
120
+ code, text, _ = run_cli(argv, handler, monkeypatch=monkeypatch)
121
+ assert code == EXIT_NO_GO
122
+ assert captured["path"] == "/api/v1/ai-governance/systems/sys-1/fairness-runs"
123
+ for part in (b'name="outcome"\r\n\r\nrepaid', b'name="attributes"\r\n\r\ngender', b'name="prediction"\r\n\r\napproved', b'filename="preds.csv"'):
124
+ assert part in captured["body"]
125
+ assert "0.000!" in text and "small groups: F, M" in text
126
+ code, _, _ = run_cli(argv + ["--gate"], handler, monkeypatch=monkeypatch)
127
+ assert code == EXIT_NO_GO
128
+
129
+
130
+ def test_test_requires_a_prediction_or_score(tmp_path, monkeypatch, capsys):
131
+ csv = tmp_path / "p.csv"; csv.write_text("a,b\n1,2\n")
132
+ code, _, _ = run_cli(["test", "--system", "s", "--file", str(csv), "--outcome", "a", "--attributes", "b", "--api-key", "k"], lambda r: httpx.Response(500), monkeypatch=monkeypatch)
133
+ assert code == EXIT_ERROR and "--prediction" in capsys.readouterr().err
134
+
135
+
136
+ def test_systems_lists_what_the_key_can_see(monkeypatch):
137
+ systems = [{"id": "s1", "name": "Loan scoring", "riskTier": "high", "lifecycleStage": "assess"}]
138
+ code, text, _ = run_cli(["systems", "--api-key", "k"], lambda r: httpx.Response(200, json=systems), monkeypatch=monkeypatch)
139
+ assert code == EXIT_GO and "Loan scoring" in text and "s1" in text
140
+
141
+
142
+ def test_monitor_posts_the_snapshot_and_reports_drift(tmp_path, monkeypatch):
143
+ csv = tmp_path / "d.csv"; csv.write_text("gender,approved,score\nF,1,0.7\nM,0,0.2\n")
144
+ seen = {}
145
+ def handler(request):
146
+ seen["path"] = request.url.path; seen["body"] = request.read()
147
+ run = {"evidenceId": "m1", "systemId": "s", "title": "t", "gate": {**GATE, "verdict": "no_go"}, "result": {
148
+ "rows": 2, "all_checks_pass": False, "overall_psi": 0.31, "evidence_confidence": 0.9, "baseline": {"present": True, "rows": 600},
149
+ "thresholds": {"disparate_impact_ratio": 0.8, "rate_drift": 0.05, "psi_moderate": 0.1, "psi_significant": 0.25},
150
+ "attributes": [{"attribute": "gender", "groups": {"F": 1, "M": 1}, "selection_rates": {"F": 1.0, "M": 0.0}, "baseline_selection_rates": {"F": 0.5, "M": 0.5},
151
+ "rate_drift": {"F": 0.5, "M": 0.5}, "max_rate_drift": 0.5, "disparate_impact_ratio": 0.0, "psi_by_group": {"F": 0.31}, "worst_psi": 0.31,
152
+ "checks": {"disparate_impact": False, "selection_rate_stable": False, "score_stable": False}}]}}
153
+ return httpx.Response(200, json=run)
154
+ code, text, _ = run_cli(["monitor", "--system", "s", "--file", str(csv), "--attributes", "gender", "--prediction", "approved", "--score", "score", "--api-key", "k"], handler, monkeypatch=monkeypatch)
155
+ assert code == EXIT_NO_GO and seen["path"].endswith("/systems/s/monitoring-runs") and b'name="score"' in seen["body"]
156
+ assert "0.500!" in text and "0.310!" in text and "baseline present" in text
157
+
158
+
159
+ def _pack_document():
160
+ import hashlib
161
+
162
+ entries = [{"id": "e1", "type": "fairness_test", "hash": "a" * 64, "createdAt": "2026-01-01"}, {"id": "e2", "type": "attestation", "hash": "b" * 64, "createdAt": "2026-01-02"}]
163
+ head = hashlib.sha256(b"fairmind-evidence-chain-v1").hexdigest()
164
+ for entry in entries:
165
+ head = hashlib.sha256(f"{head}:{entry['hash']}".encode()).hexdigest()
166
+ manifest = {"version": 1, "title": "Q3 inspection", "frameworks": ["EUAIA"], "createdAt": "2026-01-03", "createdBy": "a", "systems": [
167
+ {"system": {"id": "s1", "name": "Loan scoring", "riskTier": "high"}, "gate": {"verdict": "go"}, "evidence": entries, "chainHead": head, "seals": []}]}
168
+ digest = hashlib.sha256(json.dumps(manifest, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest()
169
+ return {"manifest": manifest, "manifestHash": digest}
170
+
171
+
172
+ def _no_network(request):
173
+ raise AssertionError("verify must not touch the network")
174
+
175
+
176
+ def _write(tmp_path, doc):
177
+ path = tmp_path / "pack.json"
178
+ path.write_text(json.dumps(doc))
179
+ return str(path)
180
+
181
+
182
+ def test_verify_accepts_an_intact_pack_offline(tmp_path):
183
+ code, text, _ = run_cli(["verify", _write(tmp_path, _pack_document())], _no_network)
184
+ assert code == EXIT_GO and "Pack intact" in text and "Loan scoring" in text
185
+
186
+
187
+ def test_verify_catches_a_tampered_evidence_hash(tmp_path):
188
+ doc = _pack_document()
189
+ doc["manifest"]["systems"][0]["evidence"][0]["hash"] = "c" * 64
190
+ code, text, _ = run_cli(["verify", _write(tmp_path, doc)], _no_network)
191
+ assert code == EXIT_NO_GO and "BROKEN" in text and "MISMATCH" in text
192
+
193
+
194
+ def test_verify_catches_a_tampered_manifest_field(tmp_path):
195
+ doc = _pack_document()
196
+ doc["manifest"]["systems"][0]["gate"]["verdict"] = "no_go"
197
+ code, text, _ = run_cli(["verify", _write(tmp_path, doc)], _no_network)
198
+ assert code == EXIT_NO_GO and "MISMATCH" in text and "does NOT verify" in text
199
+
200
+
201
+ def test_verify_rejects_a_file_that_is_not_a_pack(tmp_path, capsys):
202
+ code, _, _ = run_cli(["verify", _write(tmp_path, {"hello": 1})], _no_network)
203
+ assert code == EXIT_ERROR and "Not an inspection pack" in capsys.readouterr().err