@optitransfer/proofline 0.0.0-stage → 0.2.0-next.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +32 -0
- package/CHANGELOG.md +56 -0
- package/CONTRIBUTING.md +29 -0
- package/LICENSE +154 -0
- package/NOTICE +10 -0
- package/PATENTS-NONASSERT.md +1 -0
- package/README.md +285 -2
- package/bin/proofline.mjs +292 -0
- package/docs/00-index.md +43 -0
- package/docs/01-scope.md +70 -0
- package/docs/02-requirements.md +64 -0
- package/docs/03-domain-model.md +43 -0
- package/docs/04-architecture.md +79 -0
- package/docs/05-lifecycle.md +47 -0
- package/docs/06-verification.md +57 -0
- package/docs/07-evidence.md +69 -0
- package/docs/08-adapters.md +61 -0
- package/docs/09-security.md +64 -0
- package/docs/10-conformance.md +59 -0
- package/docs/11-development-pipeline.md +50 -0
- package/docs/12-roadmap.md +104 -0
- package/docs/13-interdependencies.md +53 -0
- package/docs/14-metrics.md +40 -0
- package/docs/adr/0001-product-boundary.md +28 -0
- package/docs/adr/0002-evidence-and-gates.md +22 -0
- package/docs/adr/0003-standalone-first.md +18 -0
- package/docs/receipt.md +25 -0
- package/package.json +60 -4
- package/src/cache.mjs +81 -0
- package/src/checks.mjs +276 -0
- package/src/claims.mjs +60 -0
- package/src/contract.mjs +404 -0
- package/src/engine.mjs +378 -0
- package/src/evidence.mjs +45 -0
- package/src/forbidden.mjs +57 -0
- package/src/guided.mjs +240 -0
- package/src/hook-stop.mjs +105 -0
- package/src/index.mjs +431 -0
- package/src/infer.mjs +173 -0
- package/src/inference.json +20 -0
- package/src/isolate.mjs +123 -0
- package/src/mcp.mjs +70 -0
- package/src/receipt.mjs +17 -0
- package/src/signature.mjs +110 -0
- package/src/util.mjs +92 -0
- package/src/vault.mjs +109 -0
- package/src/verify-receipt.mjs +109 -0
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# 06 — Verification and reconciliation
|
|
2
|
+
|
|
3
|
+
## Verification gates
|
|
4
|
+
|
|
5
|
+
| Gate | Question | Minimum evidence |
|
|
6
|
+
|---|---|---|
|
|
7
|
+
| G1 — Contract | Is the accepted scope complete and testable? | Validated, frozen contract; criteria and exclusions |
|
|
8
|
+
| G2 — Traceability | Does each requirement map to a change and/or verification record? | Requirement-to-evidence matrix |
|
|
9
|
+
| G3 — Execution | Were required checks actually run against the relevant revision? | Command identity, environment, revision, exit status, bounded output |
|
|
10
|
+
| G4 — Independent challenge | What plausible failure modes or regressions are not covered by the agent's own checks? | Diff review, negative/edge-case tests, regression challenges |
|
|
11
|
+
| G5 — Reconciliation | Do requested scope, observed changes, results, and agent claims agree? | Discrepancy list with dispositions |
|
|
12
|
+
| G6 — Delivery | Is the report accurate and are artifacts present? | Final revision, report, mandatory gate status, exceptions |
|
|
13
|
+
|
|
14
|
+
## Reconciliation dimensions
|
|
15
|
+
|
|
16
|
+
For each requirement, compare:
|
|
17
|
+
|
|
18
|
+
1. **Requested:** what the accepted contract requires.
|
|
19
|
+
2. **Changed:** what the workspace diff actually changed.
|
|
20
|
+
3. **Tested:** what the recorded checks actually exercised.
|
|
21
|
+
4. **Observed:** what adapter and repository events show.
|
|
22
|
+
5. **Claimed:** what the agent says it completed.
|
|
23
|
+
6. **Accepted:** what a human explicitly waived or changed through contract amendment.
|
|
24
|
+
|
|
25
|
+
A mismatch must create a finding. The system must not silently choose the most optimistic interpretation.
|
|
26
|
+
|
|
27
|
+
## Finding format
|
|
28
|
+
|
|
29
|
+
Each finding should include:
|
|
30
|
+
- stable finding ID and severity;
|
|
31
|
+
- affected requirement IDs;
|
|
32
|
+
- category (`omission`, `scope_drift`, `failed_check`, `stale_evidence`, `unsupported_claim`, `security`, `ambiguity`, `policy_denial`);
|
|
33
|
+
- concise statement;
|
|
34
|
+
- supporting evidence IDs and revision;
|
|
35
|
+
- confidence/verification method where probabilistic;
|
|
36
|
+
- recommended next action;
|
|
37
|
+
- disposition (`open`, `resolved`, `accepted_exception`, `not_applicable`) and who recorded it.
|
|
38
|
+
|
|
39
|
+
## Independence model
|
|
40
|
+
|
|
41
|
+
Independence is layered, not absolute:
|
|
42
|
+
- **Deterministic:** schema validation, status rules, diff paths, command exit codes, freshness checks, traceability completeness.
|
|
43
|
+
- **Test-based:** tests generated or selected from acceptance criteria and run independently.
|
|
44
|
+
- **Heuristic:** static analysis, diff heuristics, suspicious claim checks.
|
|
45
|
+
- **Model-assisted:** a separate reviewer/model may challenge the work, but its output is a finding to corroborate, not proof by itself.
|
|
46
|
+
- **Human:** domain judgement for ambiguous requirements, high-risk changes, or inadequate automated oracles.
|
|
47
|
+
|
|
48
|
+
## Decision matrix
|
|
49
|
+
|
|
50
|
+
| Condition | Permitted outcome |
|
|
51
|
+
|---|---|
|
|
52
|
+
| All mandatory requirements have fresh sufficient evidence; no open mandatory findings | `verified` |
|
|
53
|
+
| Work is useful but one or more requirements remain incomplete or explicitly waived | `partial` |
|
|
54
|
+
| Mandatory check failed, policy denied a required action, or dependency blocks required validation | `blocked` |
|
|
55
|
+
| Evidence is absent, stale, contradictory, or too weak to decide | `unverified` |
|
|
56
|
+
|
|
57
|
+
No status may be inferred from an agent's final message alone.
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# 07 — Evidence model
|
|
2
|
+
|
|
3
|
+
## Evidence record fields
|
|
4
|
+
|
|
5
|
+
The exact schema will be versioned in implementation. Every record should support the following fields:
|
|
6
|
+
|
|
7
|
+
| Field | Purpose |
|
|
8
|
+
|---|---|
|
|
9
|
+
| `schema_version` | Parse/migration contract |
|
|
10
|
+
| `evidence_id` | Stable unique identifier |
|
|
11
|
+
| `run_id` | Run that produced or collected it |
|
|
12
|
+
| `kind` | `command`, `test`, `diff`, `agent_claim`, `review`, `approval`, `environment`, `observation`, etc. |
|
|
13
|
+
| `source` | Adapter, runner, repository inspector, human, or verifier |
|
|
14
|
+
| `created_at` | Timestamp with timezone |
|
|
15
|
+
| `requirement_ids` | Related requirements where known |
|
|
16
|
+
| `revision` | Commit/tree/workspace identifier, where applicable |
|
|
17
|
+
| `environment` | Relevant runtime/tool versions and configuration fingerprint |
|
|
18
|
+
| `result` | `passed`, `failed`, `skipped`, `not_run`, `inconclusive`, `stale`, or `waived` |
|
|
19
|
+
| `command` | Structured argv, working directory, and policy ID for executed checks |
|
|
20
|
+
| `exit_code` | Process result where applicable |
|
|
21
|
+
| `output_ref` | Reference to bounded/sanitized output artifact |
|
|
22
|
+
| `digest` | Content digest for integrity and deduplication |
|
|
23
|
+
| `redactions` | Fields/content omitted or redacted and reason |
|
|
24
|
+
| `supersedes` | Prior record IDs replaced by a corrected observation |
|
|
25
|
+
| `notes` | Short explanation, not a substitute for structured fields |
|
|
26
|
+
|
|
27
|
+
## Command evidence
|
|
28
|
+
|
|
29
|
+
A `command` criterion records one evidence object per run. A run that failed is re-run (`defaults.flaky_reruns`), and the evidence then holds both runs as `first` and `rerun`, so a flaky result shows which tests failed on the first run and passed on the re-run.
|
|
30
|
+
|
|
31
|
+
| Field | Purpose |
|
|
32
|
+
|---|---|
|
|
33
|
+
| `cmd`, `exit`, `ms`, `timedOut` | The command as run, its exit code, its duration and whether it hit `timeout_s` |
|
|
34
|
+
| `counts` | `tests`, `pass`, `fail`, `skipped`, `todo` and `cancelled` from node's summary (`# fail 3` or `ℹ fail 3`) |
|
|
35
|
+
| `out_tail` | The last 1,200 characters of stdout and stderr |
|
|
36
|
+
| `failing` | The names of the failed tests found anywhere in the output, not only in `out_tail`: TAP `not ok <n> - <name>` lines, spec and dot reporter `✖ <name> (<ms>ms)` lines, and the `failing tests:` summary node prints at the end. Tests marked TODO or SKIP (in the spec reporter, a `# <reason>` after the duration) are left out. Repeats are dropped, the order is that of first appearance, and the list holds at most 100 names. Output in another format gives an empty list; `out_tail` still holds its end |
|
|
37
|
+
| `failing_total` | Present only when more than 100 names were found: how many there were |
|
|
38
|
+
|
|
39
|
+
The FAIL detail names the first 10 failing tests (`9 tests failed: a, b, c`, then `and <n> more` past 10), and the gate report lists every recorded name under **Failing tests**, per run.
|
|
40
|
+
|
|
41
|
+
## Evidence rules
|
|
42
|
+
|
|
43
|
+
- Capture the exact revision against which each check ran.
|
|
44
|
+
- A check run before a relevant change is stale for the changed behaviour unless freshness policy proves otherwise.
|
|
45
|
+
- With `defaults.reuse_results` on, a `command` or `metric` PASS is recorded as a `result` entry, and a later verify or gate on the same HEAD with a clean tree may reuse it (key: verifier, HEAD, node version, the env vars the command names) for up to `reuse_max_age_h` hours. The reused result is read from that entry, only while it is in an intact chain with the hash the cache file stored, and the report names it: `reused from <time> (evidence entry <n>)`.
|
|
46
|
+
- Preserve failed results after a later retry passes; final state may improve but history must not be rewritten.
|
|
47
|
+
- Bound output size and execution time.
|
|
48
|
+
- Redact secrets before persistence; do not assume command output is safe.
|
|
49
|
+
- Record why a check was skipped or could not run.
|
|
50
|
+
- Separate raw source evidence from verifier interpretation.
|
|
51
|
+
- A hash protects integrity, not semantic validity.
|
|
52
|
+
- Evidence may be exported without including full source code or secrets.
|
|
53
|
+
|
|
54
|
+
## Evidence lifecycle
|
|
55
|
+
|
|
56
|
+
```mermaid
|
|
57
|
+
flowchart LR
|
|
58
|
+
S[Source event / check] --> N[Normalize + validate]
|
|
59
|
+
N --> R[Redact + bound]
|
|
60
|
+
R --> H[Digest + assign ID]
|
|
61
|
+
H --> L[Append to ledger]
|
|
62
|
+
L --> F[Link to requirements/findings]
|
|
63
|
+
F --> D[Gate decision]
|
|
64
|
+
D --> P[Portable report]
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Report portability
|
|
68
|
+
|
|
69
|
+
The report must contain sufficient metadata to interpret the decision after the local process exits: contract version, run ID, revision, policy/verifier versions, requirement status, check status, exceptions, findings, and artifact references. Any omitted raw data must be clearly marked as omitted/redacted and not silently treated as inspected.
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# 08 — Agent adapters
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
Adapters translate tool-specific lifecycle signals into a stable event model. They do not make final verification decisions and must not have authority to weaken the contract.
|
|
6
|
+
|
|
7
|
+
## Adapter boundary
|
|
8
|
+
|
|
9
|
+
```mermaid
|
|
10
|
+
flowchart LR
|
|
11
|
+
AGENT[Agent / IDE / CLI] <--> AD[Vendor adapter]
|
|
12
|
+
AD --> EV[Normalized event schema]
|
|
13
|
+
EV --> CORE[Proofline core]
|
|
14
|
+
CORE --> FB[Policy-approved feedback/action]
|
|
15
|
+
FB --> AD
|
|
16
|
+
AD --> AGENT
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## Initial adapter selection
|
|
20
|
+
|
|
21
|
+
The first adapter should be chosen by a short technical spike using:
|
|
22
|
+
- quality of lifecycle signals (start, tool use, stop/completion);
|
|
23
|
+
- ability to capture the changed revision and command results;
|
|
24
|
+
- stability and documented extension points;
|
|
25
|
+
- safe failure behaviour when the agent or hook crashes;
|
|
26
|
+
- ease of local installation and reproducible integration tests.
|
|
27
|
+
|
|
28
|
+
Claude Code and Codex are candidate first adapters because they are widely used and expose lifecycle hooks. The choice must be recorded after testing rather than assumed in advance. Do not implement many thin adapters before one end-to-end path is trustworthy.
|
|
29
|
+
|
|
30
|
+
## Normalized event categories
|
|
31
|
+
|
|
32
|
+
- `run_started`
|
|
33
|
+
- `agent_message` (claim, not proof)
|
|
34
|
+
- `tool_invocation_requested`
|
|
35
|
+
- `tool_invocation_completed`
|
|
36
|
+
- `workspace_changed`
|
|
37
|
+
- `check_requested`
|
|
38
|
+
- `check_completed`
|
|
39
|
+
- `agent_paused`
|
|
40
|
+
- `agent_completed`
|
|
41
|
+
- `approval_requested`
|
|
42
|
+
- `approval_granted`
|
|
43
|
+
- `approval_denied`
|
|
44
|
+
- `adapter_error`
|
|
45
|
+
|
|
46
|
+
Each event includes schema version, event ID, run ID, source adapter/version, timestamp, and any relevant revision or correlation ID.
|
|
47
|
+
|
|
48
|
+
## Adapter requirements
|
|
49
|
+
|
|
50
|
+
- Validate and normalize all incoming data.
|
|
51
|
+
- Treat tool arguments and agent text as untrusted.
|
|
52
|
+
- Do not leak secrets in logs or feedback.
|
|
53
|
+
- Preserve ordering/correlation where the host supports it; otherwise report the limitation.
|
|
54
|
+
- Fail closed for enforcement-critical events when the adapter cannot establish a required fact.
|
|
55
|
+
- Never mark a gate verified merely because the host emitted a completion event.
|
|
56
|
+
- Support `observe` mode without taking control of the agent.
|
|
57
|
+
- Document host-specific gaps and version compatibility.
|
|
58
|
+
|
|
59
|
+
## Hook versus core
|
|
60
|
+
|
|
61
|
+
Hooks are an integration mechanism, not the product differentiator. The durable value is the contract, evidence model, independent checks, reconciliation logic, and enforceable delivery semantics behind those hooks.
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# 09 — Security and trust boundaries
|
|
2
|
+
|
|
3
|
+
## Threat model
|
|
4
|
+
|
|
5
|
+
Assets:
|
|
6
|
+
- source code and working-tree state;
|
|
7
|
+
- credentials, environment variables, and private tool output;
|
|
8
|
+
- task contracts, evidence, findings, and approvals;
|
|
9
|
+
- integrity of the delivery decision;
|
|
10
|
+
- ability to execute commands or modify external systems.
|
|
11
|
+
|
|
12
|
+
Threat actors and failure sources:
|
|
13
|
+
- malicious or compromised agent output;
|
|
14
|
+
- prompt injection embedded in repository files, issue text, tool output, or dependency docs;
|
|
15
|
+
- unsafe project scripts or malicious test commands;
|
|
16
|
+
- compromised adapter or model provider;
|
|
17
|
+
- accidental secret capture;
|
|
18
|
+
- confused-deputy behaviour where an agent tricks the verifier into using broader permissions;
|
|
19
|
+
- false confidence from stale tests, mocked checks, or weak acceptance criteria.
|
|
20
|
+
|
|
21
|
+
## Trust boundaries
|
|
22
|
+
|
|
23
|
+
```mermaid
|
|
24
|
+
flowchart TB
|
|
25
|
+
subgraph Untrusted
|
|
26
|
+
A[Agent output]
|
|
27
|
+
R[Repository files / tool output]
|
|
28
|
+
X[External model review]
|
|
29
|
+
end
|
|
30
|
+
subgraph Controlled
|
|
31
|
+
AD[Validated adapter]
|
|
32
|
+
P[Command policy]
|
|
33
|
+
L[Evidence ledger]
|
|
34
|
+
V[Deterministic verifier]
|
|
35
|
+
G[Gate engine]
|
|
36
|
+
end
|
|
37
|
+
H[Human approval] --> G
|
|
38
|
+
A --> AD
|
|
39
|
+
R --> AD
|
|
40
|
+
AD --> L
|
|
41
|
+
P --> L
|
|
42
|
+
L --> V
|
|
43
|
+
X -. advisory findings only .-> V
|
|
44
|
+
V --> G
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Required controls for MVP
|
|
48
|
+
|
|
49
|
+
- Explicit command allowlist/policy; no arbitrary command execution by default.
|
|
50
|
+
- Command argv should be structured; avoid shell interpolation where possible.
|
|
51
|
+
- Working-directory containment and path traversal protections.
|
|
52
|
+
- Process timeouts, output-size limits, and environment filtering.
|
|
53
|
+
- Secret redaction before evidence persistence.
|
|
54
|
+
- Read-only observe mode as the safest starting point.
|
|
55
|
+
- Separate permission for applying corrective edits versus reporting findings.
|
|
56
|
+
- Explicit approval for destructive or external side effects.
|
|
57
|
+
- Adapter event schema validation and compatibility checks.
|
|
58
|
+
- Report that identifies policy and verifier versions.
|
|
59
|
+
- No default telemetry or background upload.
|
|
60
|
+
- Audit trail for policy changes, exceptions, and approvals.
|
|
61
|
+
|
|
62
|
+
## Security non-goals
|
|
63
|
+
|
|
64
|
+
A local verifier cannot guarantee that arbitrary code is harmless, that every malicious change will be detected, or that a model reviewer is free from manipulation. The system reduces specific classes of process failure and makes evidence and residual uncertainty visible; it is not a formal proof of software safety.
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# 10 — Test and conformance strategy
|
|
2
|
+
|
|
3
|
+
## Test layers
|
|
4
|
+
|
|
5
|
+
1. **Schema tests:** valid/invalid contracts, event records, evidence records, and reports.
|
|
6
|
+
2. **Unit tests:** state machine, freshness, gate decisions, retry limits, traceability, redaction.
|
|
7
|
+
3. **Property tests:** deterministic decisions; evidence append/supersede rules; no `verified` state with unmet mandatory gates.
|
|
8
|
+
4. **Adapter contract tests:** event normalization, failure behaviour, version compatibility.
|
|
9
|
+
5. **Integration tests:** CLI + local workspace + check runner + ledger + report.
|
|
10
|
+
6. **End-to-end tests:** realistic agent session with controlled fixtures; no live provider required in default CI.
|
|
11
|
+
7. **Adversarial fixtures:** false completion claims, skipped tests, stale test results, scope drift, malformed events, secret output, repeated corrective loops.
|
|
12
|
+
8. **Security tests:** command injection attempts, path traversal, environment leakage, oversized output, unsafe symlinks, permission denial.
|
|
13
|
+
9. **Usability/pilot tests:** interruption count, false blocks, caught omissions, latency, user review burden.
|
|
14
|
+
|
|
15
|
+
## Mandatory negative fixtures
|
|
16
|
+
|
|
17
|
+
| Fixture | Expected result |
|
|
18
|
+
|---|---|
|
|
19
|
+
| Agent says tests passed; no test evidence exists | Not `verified`; report missing evidence |
|
|
20
|
+
| Mandatory test exits non-zero | `blocked` unless authorized policy explicitly defines a different non-verified disposition |
|
|
21
|
+
| Mandatory test was skipped | Not `verified` |
|
|
22
|
+
| Tests pass on old revision, then relevant code changes | Mark evidence stale and re-run or block |
|
|
23
|
+
| Required acceptance criterion has no evidence mapping | Finding; not `verified` |
|
|
24
|
+
| Diff changes excluded path or unrelated behaviour | Scope-drift finding |
|
|
25
|
+
| Same finding repeats without meaningful change | Stop retry loop |
|
|
26
|
+
| Agent claims a file changed but diff does not contain it | Unsupported-claim finding |
|
|
27
|
+
| Check output contains a secret-shaped value | Redact before persistence; test redaction metadata |
|
|
28
|
+
| Adapter drops an enforcement-critical event | Enforcement cannot silently pass |
|
|
29
|
+
| Gate in a fresh clone with no pinned base, where the default base is HEAD | `BLOCKED` with a corrective naming `--base`; never a pass on an empty diff |
|
|
30
|
+
| Same gate with `--base <parent>` and a change outside the task's scope | Scope-drift finding |
|
|
31
|
+
| Repository already has a pre-push hook that refuses the push | After `init` the push is still refused by that hook; a passing hook hands over to the gate |
|
|
32
|
+
| Two builders share a contract with owners | `next --owner a` never returns a task owned by `b`; an owner outside `defaults.owners` fails validation |
|
|
33
|
+
| Metric run while the machine's load per CPU is above `defaults.max_load_per_cpu` | `BLOCKED` (machine busy, with the numbers); no measured value recorded |
|
|
34
|
+
| Commit adds or amends `proofline.json` with the default deny-list | No forbidden finding in the contract itself; the change is judged by the weakening check |
|
|
35
|
+
| `defaults.reuse_results` on (`true`/`false`, default `false`; `defaults.reuse_max_age_h`, a positive number, default 24): `verify` then `gate` on one clean commit | A `command` or `metric` PASS runs once; the gate report lists it under Reused results as `reused from <time> (evidence entry <n>)`, and the gate entry says the same |
|
|
36
|
+
| Same, with an uncommitted edit or untracked file, a new commit, `--fresh`, a changed env var the command names, a PASS older than `reuse_max_age_h`, or a log that no longer holds the recording entry intact with its hash | The check runs again |
|
|
37
|
+
| Same, where the check FAILs, is flaky, is blocked or cannot run | Nothing is stored; it runs every time |
|
|
38
|
+
| `reuse_results` left out or `false` | Every check runs every time; nothing is stored |
|
|
39
|
+
|
|
40
|
+
## Conformance fixture format
|
|
41
|
+
|
|
42
|
+
Each fixture should include:
|
|
43
|
+
- contract;
|
|
44
|
+
- initial and final workspace state or synthetic diff;
|
|
45
|
+
- event stream;
|
|
46
|
+
- check results;
|
|
47
|
+
- agent delivery claims;
|
|
48
|
+
- expected findings;
|
|
49
|
+
- expected terminal status;
|
|
50
|
+
- rationale.
|
|
51
|
+
|
|
52
|
+
## Release gate
|
|
53
|
+
|
|
54
|
+
No release candidate may be described as verified until:
|
|
55
|
+
- all required CI checks pass;
|
|
56
|
+
- all mandatory conformance fixtures pass;
|
|
57
|
+
- security-sensitive changes have explicit review;
|
|
58
|
+
- release artifacts and checksums are recorded;
|
|
59
|
+
- documentation and known limitations match the actual build.
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# 11 — Development and release pipeline
|
|
2
|
+
|
|
3
|
+
## Development loop
|
|
4
|
+
|
|
5
|
+
```mermaid
|
|
6
|
+
flowchart TD
|
|
7
|
+
I[Issue / requirement] --> C[Contract and acceptance criteria]
|
|
8
|
+
C --> D[Design / ADR if needed]
|
|
9
|
+
D --> IMP[Small implementation slice]
|
|
10
|
+
IMP --> U[Unit + schema tests]
|
|
11
|
+
U --> INT[Integration + adapter contract tests]
|
|
12
|
+
INT --> ADV[Adversarial conformance fixtures]
|
|
13
|
+
ADV --> REV[Independent diff / requirement review]
|
|
14
|
+
REV --> REC[Reconcile claims, changes, evidence]
|
|
15
|
+
REC -->|gap| IMP
|
|
16
|
+
REC -->|clean| CI[CI release gates]
|
|
17
|
+
CI --> REL[Versioned release + report]
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Branch and review policy
|
|
21
|
+
|
|
22
|
+
- `main` should represent a coherent, CI-checked baseline.
|
|
23
|
+
- Use short-lived feature branches and pull requests.
|
|
24
|
+
- Each PR names requirement IDs and updates tests/docs/changelog as appropriate.
|
|
25
|
+
- Do not merge a change that weakens a mandatory gate without an ADR and explicit rationale.
|
|
26
|
+
- Require at least one review for changes to command execution, evidence integrity, gate logic, permissions, or schema migrations.
|
|
27
|
+
- Protect `main` with required CI and review rules once repository settings are available.
|
|
28
|
+
|
|
29
|
+
## CI stages
|
|
30
|
+
|
|
31
|
+
1. Checkout with minimal permissions.
|
|
32
|
+
2. Static formatting and lint.
|
|
33
|
+
3. Type/schema checks.
|
|
34
|
+
4. Unit tests.
|
|
35
|
+
5. Integration tests using synthetic/local fixtures.
|
|
36
|
+
6. Adapter contract tests.
|
|
37
|
+
7. Adversarial conformance suite.
|
|
38
|
+
8. Security and dependency checks.
|
|
39
|
+
9. Build/package.
|
|
40
|
+
10. Generate provenance/checksum and release report.
|
|
41
|
+
|
|
42
|
+
The workflow in `.github/workflows/proofline.yml` runs the self-gate (`node bin/proofline.mjs gate --ci`) on every push to `main` and on pull requests, with the branch tip before the push as the base, and uploads the evidence report. Documentation CI must not be misrepresented as software verification.
|
|
43
|
+
|
|
44
|
+
## Release discipline
|
|
45
|
+
|
|
46
|
+
- Version schemas independently or document compatibility in the package versioning policy.
|
|
47
|
+
- Record commit, build inputs, test results, and artifact digests.
|
|
48
|
+
- Publish release notes with added behaviour, limitations, migration notes, and known issues.
|
|
49
|
+
- Never publish a release based only on the agent's assertion that it is ready.
|
|
50
|
+
- Keep a release checklist and a rollback/recovery plan.
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
# 12 — Roadmap
|
|
2
|
+
|
|
3
|
+
This is a dependency-ordered roadmap, not a promise of calendar dates. A phase is complete only when its exit criteria are evidenced.
|
|
4
|
+
|
|
5
|
+
## Phase 0 — Contract and feasibility spike
|
|
6
|
+
|
|
7
|
+
**Goal:** remove ambiguity before building a broad framework.
|
|
8
|
+
|
|
9
|
+
Deliverables:
|
|
10
|
+
- Agree initial adapter choice based on a small integration spike.
|
|
11
|
+
- Define task-contract, event, evidence, finding, and report schemas.
|
|
12
|
+
- Select implementation language/runtime based on the adapter and existing project conventions.
|
|
13
|
+
- Validate that required lifecycle events and revision identity can be captured.
|
|
14
|
+
- Write 5–10 adversarial scenarios.
|
|
15
|
+
|
|
16
|
+
Exit criteria:
|
|
17
|
+
- A real agent session can be correlated with a workspace revision.
|
|
18
|
+
- The team can state which lifecycle facts are available and which are not.
|
|
19
|
+
- The MVP scope and first adapter are recorded in an ADR.
|
|
20
|
+
|
|
21
|
+
## Phase 1 — Local contract and evidence core
|
|
22
|
+
|
|
23
|
+
**Goal:** establish the trustworthy core without agent integration complexity.
|
|
24
|
+
|
|
25
|
+
Deliverables:
|
|
26
|
+
- Contract validation and versioning.
|
|
27
|
+
- Local run state machine.
|
|
28
|
+
- Evidence schema and append-oriented local ledger.
|
|
29
|
+
- Deterministic gate engine.
|
|
30
|
+
- JSON/Markdown report.
|
|
31
|
+
- Synthetic fixtures and negative tests.
|
|
32
|
+
|
|
33
|
+
Exit criteria:
|
|
34
|
+
- Missing, stale, skipped, and failed mandatory evidence cannot produce `verified`.
|
|
35
|
+
- A report is reproducible from the same contract, policy, and evidence set.
|
|
36
|
+
|
|
37
|
+
## Phase 2 — First real adapter
|
|
38
|
+
|
|
39
|
+
**Goal:** observe one supported agent end-to-end.
|
|
40
|
+
|
|
41
|
+
Deliverables:
|
|
42
|
+
- Adapter event normalization.
|
|
43
|
+
- Workspace/diff observation.
|
|
44
|
+
- Start, observe, verify, report workflow.
|
|
45
|
+
- Safe failure and adapter diagnostics.
|
|
46
|
+
|
|
47
|
+
Exit criteria:
|
|
48
|
+
- Real task can be observed in `observe` mode without breaking the agent workflow.
|
|
49
|
+
- Event gaps and adapter limitations are documented.
|
|
50
|
+
|
|
51
|
+
## Phase 3 — Verification and corrective guidance
|
|
52
|
+
|
|
53
|
+
**Goal:** catch useful omissions without creating noisy loops.
|
|
54
|
+
|
|
55
|
+
Deliverables:
|
|
56
|
+
- Requirement traceability.
|
|
57
|
+
- Independent configured checks.
|
|
58
|
+
- Scope drift detection.
|
|
59
|
+
- Unsupported completion-claim detection.
|
|
60
|
+
- Bounded corrective feedback and loop detection.
|
|
61
|
+
|
|
62
|
+
Exit criteria:
|
|
63
|
+
- Mandatory adversarial fixtures pass.
|
|
64
|
+
- Pilot measures both true catches and false blocks.
|
|
65
|
+
|
|
66
|
+
## Phase 4 — Enforce mode and hardening
|
|
67
|
+
|
|
68
|
+
**Goal:** allow policy-based delivery gates.
|
|
69
|
+
|
|
70
|
+
Deliverables:
|
|
71
|
+
- Enforce-mode gate integration.
|
|
72
|
+
- Human approval/exception records.
|
|
73
|
+
- Command policy and security hardening.
|
|
74
|
+
- CI/release report integration.
|
|
75
|
+
|
|
76
|
+
Exit criteria:
|
|
77
|
+
- Security review and failure-mode tests pass.
|
|
78
|
+
- Enforcement does not silently fail open when required signals are missing.
|
|
79
|
+
|
|
80
|
+
## Phase 5 — Optional ecosystem integrations
|
|
81
|
+
|
|
82
|
+
**Goal:** add value without compromising standalone operation.
|
|
83
|
+
|
|
84
|
+
Candidates:
|
|
85
|
+
- HiveVault connector for prior decisions and accepted exceptions.
|
|
86
|
+
- Additional agent adapters.
|
|
87
|
+
- CI provider integrations.
|
|
88
|
+
- Export/import and evidence replay tooling.
|
|
89
|
+
|
|
90
|
+
Exit criteria:
|
|
91
|
+
- Core remains standalone; integrations are optional and cannot bypass evidence gates.
|
|
92
|
+
|
|
93
|
+
## Deferred until justified
|
|
94
|
+
|
|
95
|
+
- Dashboard.
|
|
96
|
+
- Hosted control plane.
|
|
97
|
+
- Multi-agent orchestration.
|
|
98
|
+
- Broad domain packs outside software engineering.
|
|
99
|
+
- CRDT-backed replicated evidence store.
|
|
100
|
+
- Commercial/team policy management.
|
|
101
|
+
|
|
102
|
+
## Go/no-go criteria
|
|
103
|
+
|
|
104
|
+
Continue beyond the pilot only if measured omission detection and false-completion prevention justify added latency and review overhead. The key metric is useful, verified catches per interruption—not the number of events collected.
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# 13 — Interdependencies
|
|
2
|
+
|
|
3
|
+
## Dependency map
|
|
4
|
+
|
|
5
|
+
```mermaid
|
|
6
|
+
flowchart TD
|
|
7
|
+
CONTRACT[Contract schema] --> STATE[Run state machine]
|
|
8
|
+
EVENTS[Normalized event schema] --> ADAPTER[First agent adapter]
|
|
9
|
+
ADAPTER --> EVENTS
|
|
10
|
+
STATE --> LEDGER[Evidence ledger]
|
|
11
|
+
EVENTS --> LEDGER
|
|
12
|
+
GIT[Workspace / revision inspector] --> LEDGER
|
|
13
|
+
RUNNER[Command policy + check runner] --> LEDGER
|
|
14
|
+
CONTRACT --> TRACE[Requirement traceability]
|
|
15
|
+
LEDGER --> TRACE
|
|
16
|
+
TRACE --> VERIFY[Verifier]
|
|
17
|
+
VERIFY --> RECON[Reconciliation]
|
|
18
|
+
CONTRACT --> RECON
|
|
19
|
+
RECON --> GATE[Gate decision]
|
|
20
|
+
POLICY[Approval / exception policy] --> GATE
|
|
21
|
+
GATE --> REPORT[Portable reports]
|
|
22
|
+
REPORT --> PILOT[Pilot metrics]
|
|
23
|
+
HV[Optional context connector] -. non-authoritative context .-> CONTRACT
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## Dependency register
|
|
27
|
+
|
|
28
|
+
| ID | Dependency | Needed by | Risk if delayed | Mitigation |
|
|
29
|
+
|---|---|---|---|---|
|
|
30
|
+
| DEP-01 | Contract schema and stable requirement IDs | State machine, traceability, reports | Requirements become untestable prose | Define schema and examples first |
|
|
31
|
+
| DEP-02 | Adapter lifecycle capability | Observer, end-to-end workflow | Cannot reliably correlate task to events | Spike adapter early; document unsupported signals |
|
|
32
|
+
| DEP-03 | Workspace/revision identity | Evidence freshness, diff review | Stale evidence may appear current | Make revision identity mandatory where available |
|
|
33
|
+
| DEP-04 | Safe command runner | Independent checks | Arbitrary execution risk | Allowlist, structured argv, timeouts, isolated environment |
|
|
34
|
+
| DEP-05 | Evidence record model | All verification and reporting | Lost provenance and non-reproducible decisions | Version schema before multiple consumers |
|
|
35
|
+
| DEP-06 | Deterministic gate rules | Enforce mode | False pass or inconsistent status | Implement and test gate logic before hooks can block |
|
|
36
|
+
| DEP-07 | Conformance fixtures | Verifier confidence | Regressions in safety semantics | Build adversarial fixtures alongside core |
|
|
37
|
+
| DEP-08 | Pilot instrumentation | Product go/no-go | No evidence product helps users | Capture catches, false blocks, latency, interruptions |
|
|
38
|
+
| DEP-09 | An optional context connector, for example a project-memory store such as HiveVault | Cross-run context only | Integration may couple core to memory | Keep adapter boundary optional and read-only initially |
|
|
39
|
+
|
|
40
|
+
## Critical path
|
|
41
|
+
|
|
42
|
+
`contract schema -> evidence schema -> deterministic gate engine -> synthetic conformance suite -> first adapter -> pilot -> enforce mode`
|
|
43
|
+
|
|
44
|
+
The first adapter spike can run in parallel with schema work, but it must not force vendor-specific concepts into the core model.
|
|
45
|
+
|
|
46
|
+
## Integration rule for context connectors
|
|
47
|
+
|
|
48
|
+
A context connector may store or retrieve prior decisions, project constraints, known failure patterns, and accepted exceptions. Proofline must:
|
|
49
|
+
- mark imported context as historical context;
|
|
50
|
+
- show source and timestamp;
|
|
51
|
+
- ask for confirmation when a historical decision conflicts with the current contract;
|
|
52
|
+
- never treat a memory entry as current-run test evidence;
|
|
53
|
+
- work with the connector absent.
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# 14 — Metrics and evaluation
|
|
2
|
+
|
|
3
|
+
## Primary product question
|
|
4
|
+
|
|
5
|
+
Does Proofline catch meaningful omissions and unsupported completion claims at an acceptable cost in latency, false blocks, and user attention?
|
|
6
|
+
|
|
7
|
+
## Metrics
|
|
8
|
+
|
|
9
|
+
| Metric | Definition | Why it matters |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| Verified omission catches | Confirmed missing requirements caught before delivery | Core value |
|
|
12
|
+
| False completion claims blocked | Delivery claims contradicted by evidence | Trust value |
|
|
13
|
+
| False-positive blocks | Blocks later judged unnecessary or incorrect | Adoption risk |
|
|
14
|
+
| Missed defects/omissions | Known pilot issues not detected | Reliability risk |
|
|
15
|
+
| Mandatory evidence coverage | Mandatory criteria with fresh, sufficient evidence / all mandatory criteria | Traceability |
|
|
16
|
+
| Stale-evidence rejection rate | Stale evidence correctly prevented from satisfying gates | Evidence freshness |
|
|
17
|
+
| Mean verification latency | Time added by verification, split by deterministic and model-assisted steps | Workflow cost |
|
|
18
|
+
| Interruptions per task | Agent/user interruptions introduced by the tool | Usability |
|
|
19
|
+
| Corrective cycles | Number of guidance/retry cycles per run | Loop/noise risk |
|
|
20
|
+
| Human review burden | Minutes spent reviewing findings and reports | Adoption |
|
|
21
|
+
| Reproducible decision rate | Decisions replayed from captured contract/policy/evidence with same result | Auditability |
|
|
22
|
+
|
|
23
|
+
## Pilot design
|
|
24
|
+
|
|
25
|
+
- Select a representative set of real tasks, including known failure examples.
|
|
26
|
+
- Establish expected acceptance criteria before running the agent.
|
|
27
|
+
- Compare normal agent workflow with observe mode first.
|
|
28
|
+
- Label each finding as true positive, false positive, duplicate, or inconclusive.
|
|
29
|
+
- Measure total added time and user intervention.
|
|
30
|
+
- Include cases where the agent succeeds cleanly; do not evaluate only failures.
|
|
31
|
+
- Do not claim statistical generality from a small pilot.
|
|
32
|
+
|
|
33
|
+
## Initial go/no-go
|
|
34
|
+
|
|
35
|
+
No numerical threshold is asserted before baseline data exists. The pilot should agree thresholds in advance for:
|
|
36
|
+
- minimum confirmed useful catches;
|
|
37
|
+
- maximum false-positive block rate;
|
|
38
|
+
- maximum added latency;
|
|
39
|
+
- maximum user review burden;
|
|
40
|
+
- no tolerance for a false `verified` status in the mandatory negative fixtures.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# ADR-0001: Product boundary and independence
|
|
2
|
+
|
|
3
|
+
- Status: Accepted
|
|
4
|
+
- Date: 2026-10-14
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
Agent frameworks already expose hooks and lifecycle events. A thin hook wrapper would be easy to copy and would not solve the central problem: whether the requested work was actually completed and independently verified.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
Build the product around an accepted task contract, evidence ledger, independent verification, reconciliation, and delivery decision. Hooks/adapters are replaceable integration layers. HiveVault is optional.
|
|
13
|
+
|
|
14
|
+
## Consequences
|
|
15
|
+
|
|
16
|
+
Positive:
|
|
17
|
+
- Core value is not tied to a single agent.
|
|
18
|
+
- Evidence can be exported and audited independently.
|
|
19
|
+
- Standalone use is possible.
|
|
20
|
+
|
|
21
|
+
Trade-offs:
|
|
22
|
+
- More upfront design and conformance work than a prompt-only tool.
|
|
23
|
+
- Verification quality is limited by the quality of the contract and available oracles.
|
|
24
|
+
- The first adapter must be selected carefully to avoid premature generalization.
|
|
25
|
+
|
|
26
|
+
## Revisit when
|
|
27
|
+
|
|
28
|
+
Pilot evidence shows that the core abstraction is too heavy, or a specific host imposes lifecycle constraints that materially change the architecture.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# ADR-0002: Evidence-based delivery states
|
|
2
|
+
|
|
3
|
+
- Status: Accepted
|
|
4
|
+
- Date: 2026-10-14
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
Agent completion messages and process exit codes are insufficient to establish that all requirements were met. Skipped checks and stale evidence must not be confused with passing checks.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
Use explicit check states (`passed`, `failed`, `skipped`, `not_run`, `inconclusive`, `stale`, `waived`) and terminal delivery states (`verified`, `partial`, `blocked`, `unverified`). Every gate decision is derived from the contract, policy, evidence freshness, and unresolved findings.
|
|
13
|
+
|
|
14
|
+
## Consequences
|
|
15
|
+
|
|
16
|
+
- Reports are more nuanced than a single green/red result.
|
|
17
|
+
- The gate engine and state machine require thorough negative tests.
|
|
18
|
+
- Human-approved exceptions remain visible and do not rewrite the evidence history.
|
|
19
|
+
|
|
20
|
+
## Revisit when
|
|
21
|
+
|
|
22
|
+
Pilot workflows demonstrate a required terminal state not covered by the model.
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# ADR-0003: Standalone-first, context connectors optional
|
|
2
|
+
|
|
3
|
+
- Status: Accepted
|
|
4
|
+
- Date: 2026-10-14
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
HiveVault can provide persistent project context, but coupling Proofline to it would limit adoption and blur the difference between remembered claims and current evidence.
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
The MVP works locally without HiveVault. Any connector is optional and initially supplies clearly labelled historical context only.
|
|
13
|
+
|
|
14
|
+
## Consequences
|
|
15
|
+
|
|
16
|
+
- Standalone use remains possible.
|
|
17
|
+
- Cross-run knowledge can be added later without making the core dependent on it.
|
|
18
|
+
- Imported context must never satisfy current-run verification gates by itself.
|
package/docs/receipt.md
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Receipt
|
|
2
|
+
|
|
3
|
+
Without a contract, `receipt` infers the suite from the manifest at the base revision. A change to that command is reported and is not what runs.
|
|
4
|
+
|
|
5
|
+
| Manifest | Command |
|
|
6
|
+
| --- | --- |
|
|
7
|
+
| package.json `scripts.test` | that script, else `vitest run` or `jest` when that dev dependency is present |
|
|
8
|
+
| Cargo.toml | `cargo test` |
|
|
9
|
+
| pyproject.toml `[tool.pytest]` or pytest.ini | `pytest` |
|
|
10
|
+
| go.mod | `go test ./...` |
|
|
11
|
+
| build.gradle, `*.sln` | no default command, status blocked |
|
|
12
|
+
|
|
13
|
+
Changed tests are paths under `test/` or `tests/`, or names matching `*_test.*`, `*.test.*`, `*.spec.*` or `test_*.py`. Nothing inferable yields an L0 receipt and exit 2. A missing toolchain is blocked, not a pass.
|
|
14
|
+
|
|
15
|
+
`verify-receipt` checks the DSSE envelope offline against PEM files in `--keys` (or `$AGENTKEY_HOME/keys`), named `<keyid>.pem`. Exit 0 is verified, 1 is a bad signature, a schema problem or a `--recompute` difference, 2 is an unknown key, and 3 is usage. `--sigstore` is the only path that loads that verifier.
|
|
16
|
+
|
|
17
|
+
The MCP tools `receipt` and `verify_receipt` return that same JSON. `tools/list` shows nine tools.
|
|
18
|
+
|
|
19
|
+
`proofline hook stop` reads a Claude Code Stop payload and writes one receipt for the commits that session made. A second run on the same range writes none. `proofline connect claude-code` adds that hook after any Stop hook already in `~/.claude/settings.json`.
|
|
20
|
+
|
|
21
|
+
When `GET /v1/state/digest` answers within 2 seconds, an L1 receipt gains `memory.vault_digest` and `memory.frontier`. An L2 receipt in ci mode gains only `memory_commitment`, the sha256 of the vault id, the digest and the frontier JSON, so a replica id is not in the receipt. No answer leaves the receipt unchanged. `--vault` writes `work/receipts/<date>-<role>-<sha7>.md` with origin `proofline`.
|
|
22
|
+
|
|
23
|
+
The weakening check is a warning beside the owner signature. It names a loosened threshold, a removed criterion, or a changed command. Once `guard/owner.pub` is pinned, a missing, altered or wrongly-signed contract is refused with `contract not owner-signed`.
|
|
24
|
+
|
|
25
|
+
grep, wiring and the forbidden scan are marked `heuristic` in the receipt. They are one layer, not a boundary. An isolated run records `isolated` and `env_digest`. A local run records `isolated` false.
|