decisioncontrol 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- decisioncontrol-0.1.0/CHANGELOG.md +41 -0
- decisioncontrol-0.1.0/CONTRIBUTING.md +18 -0
- decisioncontrol-0.1.0/LICENSE +21 -0
- decisioncontrol-0.1.0/MANIFEST.in +4 -0
- decisioncontrol-0.1.0/PKG-INFO +333 -0
- decisioncontrol-0.1.0/README.md +308 -0
- decisioncontrol-0.1.0/SECURITY.md +38 -0
- decisioncontrol-0.1.0/decisioncontrol/__init__.py +15 -0
- decisioncontrol-0.1.0/decisioncontrol/__main__.py +3 -0
- decisioncontrol-0.1.0/decisioncontrol/_outputs.py +28 -0
- decisioncontrol-0.1.0/decisioncontrol/cli.py +116 -0
- decisioncontrol-0.1.0/decisioncontrol/demo.py +70 -0
- decisioncontrol-0.1.0/decisioncontrol/evaluation.py +281 -0
- decisioncontrol-0.1.0/decisioncontrol/evidence.py +147 -0
- decisioncontrol-0.1.0/decisioncontrol/models.py +171 -0
- decisioncontrol-0.1.0/decisioncontrol/policy.py +32 -0
- decisioncontrol-0.1.0/decisioncontrol/store.py +317 -0
- decisioncontrol-0.1.0/decisioncontrol.egg-info/PKG-INFO +333 -0
- decisioncontrol-0.1.0/decisioncontrol.egg-info/SOURCES.txt +36 -0
- decisioncontrol-0.1.0/decisioncontrol.egg-info/dependency_links.txt +1 -0
- decisioncontrol-0.1.0/decisioncontrol.egg-info/entry_points.txt +2 -0
- decisioncontrol-0.1.0/decisioncontrol.egg-info/top_level.txt +1 -0
- decisioncontrol-0.1.0/examples/evaluation-cases.jsonl +8 -0
- decisioncontrol-0.1.0/examples/evaluation-requirements.json +29 -0
- decisioncontrol-0.1.0/examples/policy.json +7 -0
- decisioncontrol-0.1.0/examples/proposal.json +9 -0
- decisioncontrol-0.1.0/pyproject.toml +45 -0
- decisioncontrol-0.1.0/scripts/smoke-wheel.sh +32 -0
- decisioncontrol-0.1.0/setup.cfg +4 -0
- decisioncontrol-0.1.0/tests/fixtures/approved-v1.json +38 -0
- decisioncontrol-0.1.0/tests/fixtures/approved-v2.json +60 -0
- decisioncontrol-0.1.0/tests/test_cli.py +133 -0
- decisioncontrol-0.1.0/tests/test_evaluation.py +190 -0
- decisioncontrol-0.1.0/tests/test_evaluation_cli.py +102 -0
- decisioncontrol-0.1.0/tests/test_evidence.py +171 -0
- decisioncontrol-0.1.0/tests/test_hardening.py +229 -0
- decisioncontrol-0.1.0/tests/test_policy.py +76 -0
- decisioncontrol-0.1.0/tests/test_store.py +558 -0
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here.
|
|
4
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
5
|
+
and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [0.1.0] - 2026-10-13
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
- Independent standard-library SDK, console CLI and module entry point for local
|
|
11
|
+
ticket reassignment; strict policy/recommendation inputs and ALLOW/REVIEW/DENY/ABSTAIN.
|
|
12
|
+
- SQLite frozen decisions, durable review, revision conflicts, atomic idempotent
|
|
13
|
+
local execution, and authorized pre-execution cancellation preserving approvals.
|
|
14
|
+
- Checksummed decision export and non-executing semantic replay; credential-free
|
|
15
|
+
demo with hand-authored inputs, reopened approval and real local queue changes.
|
|
16
|
+
- Independently labeled policy evaluation, exposure-specific metrics, Wilson
|
|
17
|
+
intervals, explicit sample/metric gates, frozen JSON/Markdown evidence and replay.
|
|
18
|
+
Missing exposure cannot pass; ungated reports are not approvals.
|
|
19
|
+
- Adverse synthetic evaluation fixtures, concurrency/rollback regressions and
|
|
20
|
+
Python 3.11/3.13 CI with installed-wheel lifecycle and evaluation smoke checks.
|
|
21
|
+
- Release metadata, dynamic package version, MIT SPDX license and project links.
|
|
22
|
+
- SQLite store schema 1 stamp and refusal of newer unknown database versions.
|
|
23
|
+
|
|
24
|
+
### Changed
|
|
25
|
+
- New decision records use schema 3 with required proposer identity, producer
|
|
26
|
+
metadata and canonical policy digest. V1/v2 replay retains historical rules;
|
|
27
|
+
cancelling a stored v1 decision promotes it to v2, preserving prior approval.
|
|
28
|
+
- Replay mismatches exit 1; malformed input and unsupported schemas exit 2.
|
|
29
|
+
- Demo, evaluation and file export publish complete staged outputs atomically.
|
|
30
|
+
|
|
31
|
+
### Fixed
|
|
32
|
+
- Corrupt pending rows no longer block valid queue entries; SDK diagnostics and
|
|
33
|
+
CLI warnings expose skipped rows (exit 1).
|
|
34
|
+
- Identical rejection retries return the immutable recorded rejection.
|
|
35
|
+
- CLI execution of a stale decision exits 1 because no action was performed.
|
|
36
|
+
|
|
37
|
+
### Security
|
|
38
|
+
- Schema 3 REVIEW proposers cannot approve, reject or cancel their own decisions.
|
|
39
|
+
Identities remain caller-asserted labels, not authentication; hashes detect
|
|
40
|
+
change, not forgery. Local atomicity does not cover external actions.
|
|
41
|
+
- No provider calls, external executor, signed attestation or RELIA integration.
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Work within `ai-workflows/decision-control`; this is an independently installable
|
|
4
|
+
Python 3.11+ standard-library package. Do not import sibling projects or copy RELIA
|
|
5
|
+
implementation. Keep changes to RELIA out of decision-control work.
|
|
6
|
+
|
|
7
|
+
Install with `python -m pip install -e .`, then run
|
|
8
|
+
`python -m unittest discover -s tests`. Exercise the console command and
|
|
9
|
+
`python -m decisioncontrol demo --out` with a fresh temporary directory. Tests
|
|
10
|
+
should verify persisted behavior, lifecycle rejection, atomic mutation and evidence
|
|
11
|
+
replay, not just strings or forwarding. Update README and CHANGELOG when behavior
|
|
12
|
+
changes. Preserve strict schema validation, frozen snapshots, local transaction
|
|
13
|
+
boundaries and idempotency; do not introduce provider calls or claim authenticated
|
|
14
|
+
review without implementing an explicit new trust boundary.
|
|
15
|
+
|
|
16
|
+
Report security issues privately as described in SECURITY.md. Contributions to
|
|
17
|
+
this original component are under its MIT license; other repository components
|
|
18
|
+
retain their own licensing and behavior.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Bicheng Wang
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,333 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: decisioncontrol
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Policy-constrained local ticket actions with durable review and replayable evidence
|
|
5
|
+
Author: Bicheng Wang
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/BichengWang/ai-decision-system/tree/main/ai-workflows/decision-control
|
|
8
|
+
Project-URL: Source, https://github.com/BichengWang/ai-decision-system
|
|
9
|
+
Project-URL: Issues, https://github.com/BichengWang/ai-decision-system/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/BichengWang/ai-decision-system/blob/main/ai-workflows/decision-control/CHANGELOG.md
|
|
11
|
+
Keywords: decision-records,policy,human-review,replay,go-no-go
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Intended Audience :: Science/Research
|
|
15
|
+
Classifier: Operating System :: OS Independent
|
|
16
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
21
|
+
Requires-Python: >=3.11
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
License-File: LICENSE
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
# decision-control
|
|
27
|
+
|
|
28
|
+
Replayable decision records for AI-assisted go/no-go decisions, demonstrated by
|
|
29
|
+
**local ticket reassignment**. An independent Python 3.11+ standard-library SDK and
|
|
30
|
+
JSON CLI. It evaluates a supplied recommendation, stores frozen evidence in SQLite, persists human review, and atomically changes the local ticket
|
|
31
|
+
queue. It does not call a model, authenticate people, or operate an external service.
|
|
32
|
+
This original MIT-licensed project is separate from RELIA: no RELIA imports, copied
|
|
33
|
+
implementation, integration, benchmark changes, or provider-quality claims.
|
|
34
|
+
|
|
35
|
+
## Install and run
|
|
36
|
+
|
|
37
|
+
Install the PyPI distribution (the 0.1.0 release is prepared but not yet published):
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
pip install decisioncontrol
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
For a source installation, run from this directory:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
uv venv --python 3.11 .venv
|
|
47
|
+
uv pip install --python .venv/bin/python .
|
|
48
|
+
. .venv/bin/activate
|
|
49
|
+
decision-control demo --out /tmp/decision-control-demo
|
|
50
|
+
python -m decisioncontrol replay --record /tmp/decision-control-demo/record-review.json
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Choose a new or empty demo directory; existing output is never overwritten. The demo
|
|
54
|
+
publishes a complete directory using staging and `os.replace`, so failed writes
|
|
55
|
+
leave the target new or empty and retryable. It creates `tickets.sqlite3`,
|
|
56
|
+
policy/proposal JSON, four exported records, and
|
|
57
|
+
`summary.json`. It exercises ALLOW, REVIEW, DENY and ABSTAIN; closes/reopens the store
|
|
58
|
+
before approval; executes the approved reassignment twice safely; and replays all
|
|
59
|
+
serialized records. ALLOW and approved REVIEW tickets have revision 1 and new
|
|
60
|
+
departments; DENY and ABSTAIN tickets remain in intake at revision 0. Probabilities
|
|
61
|
+
are **hand-authored illustrative inputs**, not measured confidence or model calls.
|
|
62
|
+
|
|
63
|
+
## Persisted operator workflow
|
|
64
|
+
|
|
65
|
+
The bundled example requests engineering, which requires review. Run from this
|
|
66
|
+
directory after installation (use a fresh database for the initial ticket):
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
decision-control --db queue.sqlite3 ticket-create --ticket-id T-100 --department intake
|
|
70
|
+
decision-control --db queue.sqlite3 submit --proposal examples/proposal.json --policy examples/policy.json --proposed-by local-proposer
|
|
71
|
+
decision-control --db queue.sqlite3 pending
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Copy the returned decision `id` into `DECISION_ID`, then run:
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
decision-control --db queue.sqlite3 show "$DECISION_ID"
|
|
78
|
+
decision-control --db queue.sqlite3 approve "$DECISION_ID" --reviewer local-reviewer --note 'Checked routing'
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
The decision is now READY, not yet executed. Choose execution or withdrawal.
|
|
82
|
+
|
|
83
|
+
To execute:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
decision-control --db queue.sqlite3 execute "$DECISION_ID"
|
|
87
|
+
decision-control --db queue.sqlite3 execute "$DECISION_ID"
|
|
88
|
+
decision-control --db queue.sqlite3 ticket-show T-100
|
|
89
|
+
decision-control --db queue.sqlite3 export "$DECISION_ID" --out approved-record.json
|
|
90
|
+
decision-control replay --record approved-record.json
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
To withdraw **before executing**, use a reviewer from the decision's frozen policy
|
|
94
|
+
and a nonempty reason. This also works for pending reviews and automatically
|
|
95
|
+
allowed READY decisions:
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
decision-control --db queue.sqlite3 cancel "$DECISION_ID" --reviewer local-reviewer --note 'Approval was mistaken'
|
|
99
|
+
decision-control --db queue.sqlite3 export "$DECISION_ID" --out cancelled-record.json
|
|
100
|
+
decision-control replay --record cancelled-record.json
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Cancellation leaves the ticket unchanged and permanently blocks execution and
|
|
104
|
+
further review. The record retains any original approval and adds a CANCELLED
|
|
105
|
+
event with the cancelling reviewer, reason and time. SDK callers use
|
|
106
|
+
`store.cancel(decision_id, reviewer, note)`. Repeating the same cancellation is
|
|
107
|
+
idempotent; a different actor or reason cannot rewrite it. A cancelled pending
|
|
108
|
+
review leaves the pending queue. If execution commits first, cancellation fails:
|
|
109
|
+
it is not an undo operation. Both operations use the same SQLite writer
|
|
110
|
+
transaction, so overlapping callers cannot both commit their conflicting actions.
|
|
111
|
+
|
|
112
|
+
Alternatively `reject ID --reviewer local-reviewer --note TEXT` permanently rejects
|
|
113
|
+
a pending review. Identical repeated approvals, rejections and cancellations return
|
|
114
|
+
the recorded result; conflicting retries fail. For new REVIEW decisions, the
|
|
115
|
+
proposer cannot approve, reject or cancel, even if listed as a policy reviewer.
|
|
116
|
+
`export ID` without `--out` prints the envelope; file export
|
|
117
|
+
refuses overwrite. All successful command results are JSON. `--db` is a global
|
|
118
|
+
option placed **before** storage commands. `demo` and `replay` need no database
|
|
119
|
+
option. Exit codes are:
|
|
120
|
+
|
|
121
|
+
- **0**: command succeeded, gate passed, or evidence replayed successfully. Valid
|
|
122
|
+
replay succeeds even for a recorded FAIL; ungated evaluation also exits 0.
|
|
123
|
+
- **1**: gate failed, replay checksum/semantic mismatch, prohibited lifecycle action,
|
|
124
|
+
unauthorized/self review, execution returned STALE (no action performed), or
|
|
125
|
+
`pending` skipped corrupt rows.
|
|
126
|
+
- **2**: invalid input/usage, unreadable file, unsupported record/store schema, or IO error.
|
|
127
|
+
|
|
128
|
+
`pending()` returns a list-compatible result whose `.skipped` contains `{id, error}`
|
|
129
|
+
diagnostics. The CLI still prints valid pending records as JSON, warns on stderr
|
|
130
|
+
and exits 1 if it skipped any rows. It does not repair corrupt rows. Each submitted
|
|
131
|
+
record (including DENY/ABSTAIN) is persisted successfully; submission exit 0 means
|
|
132
|
+
recorded, and execution is a separate command.
|
|
133
|
+
|
|
134
|
+
Errors go to stderr; argparse help is ordinary
|
|
135
|
+
human-readable help. See `decision-control --help` and command `--help`.
|
|
136
|
+
|
|
137
|
+
## SDK
|
|
138
|
+
|
|
139
|
+
```python
|
|
140
|
+
from decisioncontrol import Recommendation, RoutingPolicy, TicketStore, export_record, replay
|
|
141
|
+
|
|
142
|
+
policy = RoutingPolicy('sdk-1', ('support',), (), ('local-reviewer',), 0.8)
|
|
143
|
+
proposal = Recommendation('reassign_ticket', 'SDK-1', 'support', 0.95,
|
|
144
|
+
'hand-authored-example', 'routing-question-1')
|
|
145
|
+
with TicketStore('sdk-queue.sqlite3') as store:
|
|
146
|
+
store.create_ticket('SDK-1', 'intake')
|
|
147
|
+
submitted = store.submit(proposal, policy, proposed_by="local-proposer")
|
|
148
|
+
executed = store.execute(submitted['id'])
|
|
149
|
+
assert store.execute(submitted['id']) == executed
|
|
150
|
+
envelope = export_record(executed)
|
|
151
|
+
print(replay(envelope))
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Model output is input, not authority. Only `reassign_ticket` is supported. Invalid
|
|
155
|
+
actions, mismatched tickets, prohibited destinations and no-op assignments are
|
|
156
|
+
DENY. Refusal, missing probability or probability below the threshold is ABSTAIN
|
|
157
|
+
when hard constraints otherwise permit the action. Review destinations yield
|
|
158
|
+
REVIEW; other permitted recommendations yield ALLOW. JSON inputs are strictly
|
|
159
|
+
validated; no missing probability is invented.
|
|
160
|
+
|
|
161
|
+
## Evaluate supplied recommendations
|
|
162
|
+
|
|
163
|
+
Evaluate a candidate routing policy on independently labeled JSON Lines before
|
|
164
|
+
allowing it to automate actions. This is an offline evaluator: it neither calls a
|
|
165
|
+
provider nor executes actions or approves pending decisions.
|
|
166
|
+
|
|
167
|
+
```bash
|
|
168
|
+
decision-control evaluate --cases examples/evaluation-cases.jsonl \
|
|
169
|
+
--policy examples/policy.json --out /tmp/decision-control-evaluation
|
|
170
|
+
decision-control evaluate --cases examples/evaluation-cases.jsonl \
|
|
171
|
+
--policy examples/policy.json --require examples/evaluation-requirements.json \
|
|
172
|
+
--out /tmp/decision-control-gated
|
|
173
|
+
# The supplied adverse fixture intentionally exits 1 and records FAIL.
|
|
174
|
+
decision-control replay-evaluation --record /tmp/decision-control-gated/evaluation.json
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Each output directory must be new or empty. Evaluation and demo directories, and
|
|
178
|
+
file exports, are built beside the target and published with `os.replace`; failed
|
|
179
|
+
writes leave no partial target. Existing nonempty outputs are refused.
|
|
180
|
+
`evaluation.json` retains the full
|
|
181
|
+
cases, labels, policy, requirements, per-case verdicts, metrics and gate outcomes
|
|
182
|
+
in a checksummed envelope. `EVALUATION.md` is the human-readable report. A failed
|
|
183
|
+
gate still writes evidence and exits 1; invalid input exits 2 without excluding
|
|
184
|
+
bad rows from the score. Omitting `--require` gives `UNGATED`, **not PASS**.
|
|
185
|
+
Replay returns success for valid evidence even when its recorded gate is FAIL;
|
|
186
|
+
the returned disposition remains FAIL.
|
|
187
|
+
|
|
188
|
+
The example cases and review outcomes are **hand-authored software fixtures**.
|
|
189
|
+
They include an unsafe automatic allowance, an incorrect review approval and an
|
|
190
|
+
unobserved review. Their small sample intentionally fails the example gates;
|
|
191
|
+
neither that failure nor an ungated report measures any live model.
|
|
192
|
+
|
|
193
|
+
### Case labels
|
|
194
|
+
|
|
195
|
+
Each JSON Lines row has `id`, `ticket`, `recommendation`, `labels`, and an optional
|
|
196
|
+
`review_outcome`. Ticket and recommendation use the existing SDK schemas.
|
|
197
|
+
|
|
198
|
+
- `labels.safe_to_execute`: human judgment of whether the proposed action would
|
|
199
|
+
be safe if the required approval were obtained.
|
|
200
|
+
- `labels.requires_review`: human judgment of whether approval is required.
|
|
201
|
+
- `labels.correct_destination`: independently labeled routing destination.
|
|
202
|
+
- `review_outcome`: supplied `APPROVED`, `REJECTED`, or null/absent. It is an
|
|
203
|
+
observation, not a request to approve anything or proof of reviewer identity.
|
|
204
|
+
|
|
205
|
+
Duplicate case identifiers, empty datasets, malformed inputs and unknown fields
|
|
206
|
+
are rejected. Labels must not be generated by the policy being evaluated.
|
|
207
|
+
|
|
208
|
+
### Metrics and denominators
|
|
209
|
+
|
|
210
|
+
| Metric | Numerator | Denominator |
|
|
211
|
+
| --- | --- | --- |
|
|
212
|
+
| `auto_allow_rate` | ALLOW cases | All cases |
|
|
213
|
+
| `unsafe_auto_allow_rate` | ALLOW despite an unsafe or required-review label | ALLOW cases |
|
|
214
|
+
| `auto_route_error_rate` | ALLOW with an incorrect destination | ALLOW cases |
|
|
215
|
+
| `review_rate` | REVIEW cases | All cases |
|
|
216
|
+
| `abstain_rate` | ABSTAIN cases | All cases |
|
|
217
|
+
| `deny_rate` | DENY cases | All cases |
|
|
218
|
+
| `required_review_recall` | REVIEW with a required-review label | All required-review cases |
|
|
219
|
+
| `review_precision` | REVIEW with a required-review label | REVIEW cases |
|
|
220
|
+
| `incorrect_approval_rate` | Observed APPROVED with an unsafe label or incorrect destination | Observed APPROVED on REVIEW cases |
|
|
221
|
+
| `review_observation_rate` | Supplied review outcome on REVIEW cases | REVIEW cases |
|
|
222
|
+
|
|
223
|
+
Every metric includes its numerator, denominator, point estimate (`value`) and
|
|
224
|
+
95% Wilson bounds (`lower`, `upper`). A zero denominator yields null estimates,
|
|
225
|
+
not zero errors, and cannot satisfy a required bound. A safe DENY/ABSTAIN does not
|
|
226
|
+
count as REVIEW recall. Review observations are scored only for cases this policy
|
|
227
|
+
routes to REVIEW; approval error is conditional on observed approvals. Require
|
|
228
|
+
enough observations rather than assuming missing approvals were correct.
|
|
229
|
+
|
|
230
|
+
### Gate contract and statistical limits
|
|
231
|
+
|
|
232
|
+
Requirements contain `min_cases` and a nonempty `bounds` object keyed by the
|
|
233
|
+
metric names above. Each bound has `min` and/or `max`, an optional `statistic`
|
|
234
|
+
(`value`, `lower`, `upper`), and optional positive `min_denominator`.
|
|
235
|
+
|
|
236
|
+
Use `upper` for error ceilings and `lower` for recall floors, plus enough sample
|
|
237
|
+
support. A point-estimate gate does not establish a confidence guarantee:
|
|
238
|
+
zero observed errors in ten allowed cases still has a 95% upper bound near 0.278.
|
|
239
|
+
Freeze policies and requirements before examining a representative holdout.
|
|
240
|
+
Intervals assume independent observations, are per-metric rather than simultaneous
|
|
241
|
+
guarantees, and do not correct for threshold tuning or repeated looks. Synthetic,
|
|
242
|
+
correlated or selectively observed cases do not establish deployed safety,
|
|
243
|
+
calibrated provider probabilities or legal compliance.
|
|
244
|
+
|
|
245
|
+
SDK entry points are `EvaluationCase`, `load_cases(path)`,
|
|
246
|
+
`evaluate_cases(cases, policy, requirements=None)`, `write_evaluation(report, out)`
|
|
247
|
+
and `replay_evaluation(envelope)`. Evaluation evidence is distinct from the
|
|
248
|
+
executed-decision records below. Both checksums detect corruption, not forgery.
|
|
249
|
+
|
|
250
|
+
## Record schema v3 (v1/v2 readable)
|
|
251
|
+
|
|
252
|
+
An export contains `record` and `sha256`. The record includes:
|
|
253
|
+
|
|
254
|
+
- integer `schema_version`, `produced_by` distribution name/version, decision `id`,
|
|
255
|
+
and UTC `created_at`;
|
|
256
|
+
- caller-asserted `proposed_by` and lowercase `policy_sha256` over canonical policy JSON;
|
|
257
|
+
- the supplied `recommendation` with model/question identifiers, probability and refusal;
|
|
258
|
+
- the trusted `ticket` snapshot (identifier, department and revision);
|
|
259
|
+
- the frozen `policy`, evaluated verdict and machine-readable reason codes;
|
|
260
|
+
- lifecycle `status`, authorized `review`, execution `result`, and transition `events`.
|
|
261
|
+
|
|
262
|
+
The full record is checksummed, including review and execution history. Replay
|
|
263
|
+
rejects inconsistent verdicts, illegal transitions, unauthorized recorded reviewers
|
|
264
|
+
and results that disagree with the frozen action. Schema 3 additionally validates
|
|
265
|
+
the policy digest and forbids proposer review/cancellation of REVIEW decisions.
|
|
266
|
+
New decisions use `schema_version: 3`; a CANCELLED event has `event`, `at`,
|
|
267
|
+
`reviewer` and `note` fields. Other event shapes remain unchanged.
|
|
268
|
+
|
|
269
|
+
Archived v1/v2 evidence and stored decisions remain valid under their original
|
|
270
|
+
rules, which have no proposer separation check. They do not gain invented identity
|
|
271
|
+
provenance on replay. Cancelling a stored v1 decision explicitly promotes that record
|
|
272
|
+
to v2, preserving its inputs and prior approval. Unknown schema versions are
|
|
273
|
+
rejected rather than silently reinterpreted. Policy-evaluation reports use their
|
|
274
|
+
own unchanged schema.
|
|
275
|
+
|
|
276
|
+
Supply the probability of the proposed choice, not an unrelated provider confidence
|
|
277
|
+
score. The host owns provider adaptation, model/question identity and threshold
|
|
278
|
+
validation on labeled application data. This library does not calibrate probabilities
|
|
279
|
+
or turn model recommendations into trusted permissions.
|
|
280
|
+
|
|
281
|
+
## Evidence and trust boundary
|
|
282
|
+
|
|
283
|
+
* Proposer and reviewer identifiers are caller-asserted labels supplied by a
|
|
284
|
+
**trusted local host/operator**, not authenticated identities. There is no login, server, RBAC service or network
|
|
285
|
+
boundary. Separation of duties compares labels, so a caller can bypass it by
|
|
286
|
+
claiming another identity. The host must authenticate people, bind labels to
|
|
287
|
+
principals and control who supplies the policy/reviewer list. A policy digest
|
|
288
|
+
freezes the supplied content; it does not establish that policy as trusted.
|
|
289
|
+
A malicious process with file access can modify the database/evidence.
|
|
290
|
+
* Policies and ticket snapshots are frozen at submission. Updating a policy file
|
|
291
|
+
does not retroactively revoke existing decisions or approvals. Use `cancel` for
|
|
292
|
+
explicit pre-execution withdrawal, not an assumed current-policy check.
|
|
293
|
+
* Execution checks the frozen ticket department/revision against current state.
|
|
294
|
+
A competing reassignment makes the decision STALE without changing the ticket.
|
|
295
|
+
The SDK returns STALE; the CLI exits 1. SQLite commits queue mutation, revision
|
|
296
|
+
increment, result and event together.
|
|
297
|
+
Repeating execution of the same executed decision returns its stored result.
|
|
298
|
+
This is **local atomicity**, not exactly-once execution of external actions.
|
|
299
|
+
* Exported envelopes contain SHA-256 over canonical JSON. Replay checks corruption,
|
|
300
|
+
recomputes the frozen policy evaluation and validates lifecycle evidence. Canonical
|
|
301
|
+
JSON uses sorted keys, `separators=(",", ":")`, `ensure_ascii=False`, UTF-8, and
|
|
302
|
+
lowercase hexadecimal SHA-256. Replay neither calls a model nor executes an action. Checksums are not signatures:
|
|
303
|
+
attackers can recompute them. Records are not provider attestations, identity
|
|
304
|
+
proof, calibrated model evidence or proof of outside-world execution.
|
|
305
|
+
* Protect database files, exports and notes using operating-system permissions;
|
|
306
|
+
they can contain sensitive ticket metadata. Avoid untrusted database/JSON files
|
|
307
|
+
and unbounded inputs. See [SECURITY.md](SECURITY.md).
|
|
308
|
+
|
|
309
|
+
SQLite store schema 1 is stamped in `PRAGMA user_version`. Legacy unversioned
|
|
310
|
+
stores are accepted and stamped without changing their records; newer unknown
|
|
311
|
+
store versions are refused before creating tables. Journal mode is unchanged.
|
|
312
|
+
|
|
313
|
+
## Development
|
|
314
|
+
|
|
315
|
+
```bash
|
|
316
|
+
python -m unittest discover -s tests
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
The package has no runtime dependencies or sibling-project imports. Setuptools is
|
|
320
|
+
an installation/build dependency. Repository CI uses the registered
|
|
321
|
+
`decision-control-ci.yml`; [CONTRIBUTING.md](CONTRIBUTING.md) describes scope.
|
|
322
|
+
[CHANGELOG.md](CHANGELOG.md) records changes. New project code is licensed under
|
|
323
|
+
[MIT](LICENSE); this does not relicense RELIA or any other component.
|
|
324
|
+
|
|
325
|
+
## Verification status
|
|
326
|
+
|
|
327
|
+
Behavioral tests cover restart-safe review, concurrent local execution, rollback,
|
|
328
|
+
stale decisions, separation of duties, corrupt queue rows, idempotent rejection,
|
|
329
|
+
retryable output failures, and v1/v2/v3 replay. CI builds and validates the wheel,
|
|
330
|
+
installs it into a fresh environment, then runs demo, decision replay, an adverse
|
|
331
|
+
fixture evaluation, evaluation replay, and a rejected self-approval outside the
|
|
332
|
+
source tree. These are software checks on hand-authored fixtures, not live-model,
|
|
333
|
+
calibration, adoption or production evidence. PyPI publication is a separate step.
|