@cassiomc1/forgeloop 1.6.4 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT_COMPATIBILITY.md +11 -0
- package/DOCS_INDEX.md +1 -0
- package/GUIDE_ROUTER.md +26 -1
- package/LOOP_ENGINEERING.md +48 -0
- package/ORCHESTRATOR_INTEGRATION.md +38 -0
- package/PROTOCOL_INTEGRATION.md +62 -0
- package/README.md +25 -0
- package/benchmarks/execution-profiles/README.md +44 -0
- package/benchmarks/execution-profiles/api-feature.json +18 -0
- package/benchmarks/execution-profiles/authentication-change.json +18 -0
- package/benchmarks/execution-profiles/documentation-correction.json +18 -0
- package/benchmarks/execution-profiles/infrastructure-release.json +18 -0
- package/benchmarks/execution-profiles/novatask-saas-landing-page.json +36 -0
- package/benchmarks/execution-profiles/small-bug-fix.json +18 -0
- package/benchmarks/execution-profiles/static-landing-page.json +18 -0
- package/completions/_forgeloop +6 -4
- package/completions/forgeloop.bash +8 -4
- package/completions/forgeloop.fish +24 -1
- package/docs/AGENT_PROTOCOL_SUMMARY.md +45 -1
- package/docs/ARTIFACT_REFERENCE.md +53 -7
- package/docs/CLI_REFERENCE.md +83 -1
- package/docs/EXECUTION_PROFILE_BENCHMARKS.md +208 -0
- package/docs/GETTING_STARTED.md +28 -0
- package/docs/MCP.md +6 -0
- package/docs/RELEASE_CHECKLIST.md +4 -0
- package/docs/TROUBLESHOOTING.md +6 -0
- package/docs/UNIVERSAL_INTEGRATION.md +58 -0
- package/package.json +14 -2
- package/schemas/config.schema.json +1 -0
- package/schemas/execution-profile-benchmark-aggregate.schema.json +43 -0
- package/schemas/execution-profile-benchmark-run.schema.json +106 -0
- package/schemas/execution-profile-benchmark-scenario.schema.json +66 -0
- package/schemas/routing-result.schema.json +12 -0
- package/schemas/usage.schema.json +30 -0
- package/scripts/check-efficiency-regression.mjs +99 -0
- package/scripts/generate-agent-protocol-summary.mjs +35 -0
- package/scripts/lib/execution-profile-benchmark-io.mjs +67 -0
- package/scripts/run-execution-profile-benchmarks.mjs +265 -0
- package/scripts/summarize-execution-profile-benchmarks.mjs +84 -0
- package/scripts/validate-execution-profile-benchmarks.mjs +120 -0
- package/src/cli.js +16 -4
- package/src/commands/efficiency.js +12 -0
- package/src/commands/eval.js +8 -2
- package/src/commands/metrics.js +2 -2
- package/src/commands/next.js +26 -2
- package/src/commands/route.js +20 -2
- package/src/commands/task-show.js +31 -2
- package/src/commands/usage-record.js +61 -0
- package/src/core/artifact-registry.js +12 -0
- package/src/core/cli-command-definitions.js +27 -0
- package/src/core/command-executors.js +29 -5
- package/src/core/command-input.js +34 -0
- package/src/core/config.js +9 -0
- package/src/core/efficiency.js +197 -0
- package/src/core/error-codes.js +18 -0
- package/src/core/execution-profile-benchmarks.js +674 -0
- package/src/core/execution-profile-context.js +177 -0
- package/src/core/execution-profile.js +248 -0
- package/src/core/integration-invocation-policy.js +43 -0
- package/src/core/integration-resources.js +22 -1
- package/src/core/protocol-info.js +42 -0
- package/src/core/resumability.js +27 -1
- package/src/core/router.js +23 -1
- package/src/core/runtime-context.js +11 -0
- package/src/core/schema-validation.js +4 -0
- package/src/core/task-paths.js +2 -0
- package/src/core/templates.js +4 -0
- package/src/core/trace.js +1 -0
- package/src/core/trajectory-evaluation.js +2 -2
- package/src/core/trajectory-metrics.js +18 -2
- package/src/core/usage.js +137 -0
- package/src/integration.d.ts +84 -0
- package/src/integration.js +14 -0
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Generated by scripts/generate-shell-completions.mjs. Do not edit.
|
|
2
|
-
complete -c forgeloop -f -n '__fish_use_subcommand' -a 'action-authorize action-propose action-reconcile action-record action-show action-verify activate advance approval-request approval-resolve attestation-create attestation-status attestation-verify attestation-verify-range audit baseline bundle clear-continuity clear-state complete continuity doctor eval handoff-create handoff-list handoff-show history init inspect metrics migrate-protocol next policy policy-diff policy-discover policy-status preflight prepare-completion profile-interview progress protocol-info reconcile-closure reconcile-continuity record-check record-continuity record-decision-criterion record-diagnosis record-hypothesis-disposition record-intervention record-terminal-result reflect report responsibility-set responsibility-status route rule-verify run-action run-check status task-create task-list task-lock-status task-migrate task-recover task-repair-legacy-recovery task-resume task-scope task-show task-unlock trace update validate-protocol validate-receipt validate-state verify-scope workspace-bind workspace-status'
|
|
2
|
+
complete -c forgeloop -f -n '__fish_use_subcommand' -a 'action-authorize action-propose action-reconcile action-record action-show action-verify activate advance approval-request approval-resolve attestation-create attestation-status attestation-verify attestation-verify-range audit baseline bundle clear-continuity clear-state complete continuity doctor efficiency eval handoff-create handoff-list handoff-show history init inspect metrics migrate-protocol next policy policy-diff policy-discover policy-status preflight prepare-completion profile-interview progress protocol-info reconcile-closure reconcile-continuity record-check record-continuity record-decision-criterion record-diagnosis record-hypothesis-disposition record-intervention record-terminal-result reflect report responsibility-set responsibility-status route rule-verify run-action run-check status task-create task-list task-lock-status task-migrate task-recover task-repair-legacy-recovery task-resume task-scope task-show task-unlock trace update usage-record validate-protocol validate-receipt validate-state verify-scope workspace-bind workspace-status'
|
|
3
3
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from action-authorize' -l 'action' -d 'durable action ID'
|
|
4
4
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from action-authorize' -l 'approval' -d 'current fingerprint-bound approval'
|
|
5
5
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from action-authorize' -l 'help' -d 'show this help'
|
|
@@ -161,6 +161,12 @@ complete -c forgeloop -f -n '__fish_seen_subcommand_from doctor' -l 'json' -d 'e
|
|
|
161
161
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from doctor' -l 'path' -d 'target project directory (default: current directory)'
|
|
162
162
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from doctor' -l 'strict' -d 'treat warnings as unhealthy'
|
|
163
163
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from doctor' -l 'version' -d 'show the installed package version'
|
|
164
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from efficiency' -l 'baseline' -d 'optional comparable efficiency baseline JSON'
|
|
165
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from efficiency' -l 'help' -d 'show this help'
|
|
166
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from efficiency' -l 'json' -d 'emit efficiency metrics as JSON'
|
|
167
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from efficiency' -l 'path' -d 'target project directory (default: current directory)'
|
|
168
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from efficiency' -l 'task' -d 'task ID to operate on (when omitted, resolved from context or single active task)'
|
|
169
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from efficiency' -l 'version' -d 'show the installed package version'
|
|
164
170
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from eval' -l 'help' -d 'show this help'
|
|
165
171
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from eval' -l 'json' -d 'emit evaluation as JSON'
|
|
166
172
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from eval' -l 'path' -d 'target project directory (default: current directory)'
|
|
@@ -220,6 +226,7 @@ complete -c forgeloop -f -n '__fish_seen_subcommand_from migrate-protocol' -l 'j
|
|
|
220
226
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from migrate-protocol' -l 'path' -d 'target project directory (default: current directory)'
|
|
221
227
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from migrate-protocol' -l 'to' -d 'target supported protocol version'
|
|
222
228
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from migrate-protocol' -l 'version' -d 'show the installed package version'
|
|
229
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from next' -l 'compact' -d 'emit a bounded next-action projection'
|
|
223
230
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from next' -l 'help' -d 'show this help'
|
|
224
231
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from next' -l 'json' -d 'emit structured output as JSON'
|
|
225
232
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from next' -l 'path' -d 'target project directory (default: current directory)'
|
|
@@ -388,6 +395,7 @@ complete -c forgeloop -f -n '__fish_seen_subcommand_from responsibility-status'
|
|
|
388
395
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from responsibility-status' -l 'version' -d 'show the installed package version'
|
|
389
396
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from route' -l 'behavior-change' -d 'declare behavior change'
|
|
390
397
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from route' -l 'executable-change' -d 'declare executable/configuration change'
|
|
398
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from route' -l 'execution-profile' -d 'requested execution profile; safety floors always win'
|
|
391
399
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from route' -l 'help' -d 'show this help'
|
|
392
400
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from route' -l 'json' -d 'emit route result as JSON'
|
|
393
401
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from route' -l 'path' -d 'target project directory (default: current directory)'
|
|
@@ -480,6 +488,7 @@ complete -c forgeloop -f -n '__fish_seen_subcommand_from task-scope' -l 'json' -
|
|
|
480
488
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from task-scope' -l 'path' -d 'target project directory (default: current directory)'
|
|
481
489
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from task-scope' -l 'task' -d 'task ID to operate on (when omitted, resolved from context or single active task)'
|
|
482
490
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from task-scope' -l 'version' -d 'show the installed package version'
|
|
491
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from task-show' -l 'compact' -d 'emit a bounded task-status projection'
|
|
483
492
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from task-show' -l 'help' -d 'show this help'
|
|
484
493
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from task-show' -l 'json' -d 'emit structured output as JSON'
|
|
485
494
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from task-show' -l 'path' -d 'target project directory (default: current directory)'
|
|
@@ -501,6 +510,20 @@ complete -c forgeloop -f -n '__fish_seen_subcommand_from update' -l 'dry-run' -d
|
|
|
501
510
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from update' -l 'help' -d 'show this help'
|
|
502
511
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from update' -l 'path' -d 'target project directory (default: current directory)'
|
|
503
512
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from update' -l 'version' -d 'show the installed package version'
|
|
513
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'cache-read-tokens' -d 'provider-reported cache-read token count'
|
|
514
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'cache-write-tokens' -d 'provider-reported cache-write token count'
|
|
515
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'cost-usd' -d 'provider-reported cost in USD; never estimated'
|
|
516
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'help' -d 'show this help'
|
|
517
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'input-tokens' -d 'provider-reported input token count'
|
|
518
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'json' -d 'emit usage telemetry as JSON'
|
|
519
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'model' -d 'model name reported by the actor'
|
|
520
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'output-tokens' -d 'provider-reported output token count'
|
|
521
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'path' -d 'target project directory (default: current directory)'
|
|
522
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'provider' -d 'provider name reported by the actor'
|
|
523
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'source' -d 'usage source; CLI fallback accepts only ACTOR_REPORTED'
|
|
524
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'task' -d 'task ID to operate on (when omitted, resolved from context or single active task)'
|
|
525
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'total-tokens' -d 'provider-reported total token count; never estimated'
|
|
526
|
+
complete -c forgeloop -f -n '__fish_seen_subcommand_from usage-record' -l 'version' -d 'show the installed package version'
|
|
504
527
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from validate-protocol' -l 'continuity-file' -d 'optional execution-continuity JSON relative to target'
|
|
505
528
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from validate-protocol' -l 'contract-file' -d 'current JSON contract used for freshness comparison'
|
|
506
529
|
complete -c forgeloop -f -n '__fish_seen_subcommand_from validate-protocol' -l 'delegated-result-file' -d 'delegated result JSON file'
|
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
ForgeLoop is a portable protocol and support CLI for verifiable engineering workflows. It records and validates task state, contracts, routing, checks, evidence, continuity, and optional code attestations. It does not become an agent scheduler, delegation service, source-control authority, or secret manager.
|
|
8
8
|
|
|
9
9
|
Protocol version: 1
|
|
10
|
-
Package version: 1.
|
|
10
|
+
Package version: 1.7.0
|
|
11
11
|
|
|
12
12
|
## Canonical loop
|
|
13
13
|
|
|
@@ -19,6 +19,41 @@ Package version: 1.6.4
|
|
|
19
19
|
6. Run forgeloop complete; accept completion only when the validator returns VALID.
|
|
20
20
|
7. Run forgeloop next again and follow the returned lifecycle action to a terminal state or an explicit blocker.
|
|
21
21
|
|
|
22
|
+
## Adaptive execution profiles
|
|
23
|
+
|
|
24
|
+
`complianceMode` controls how strongly project policy is enforced. The
|
|
25
|
+
orthogonal `executionProfile` controls process and context depth: `auto`
|
|
26
|
+
resolves deterministically to `light`, `balanced`, or `full` from route,
|
|
27
|
+
contract, and task metadata. CLI requests take precedence over project
|
|
28
|
+
configuration, but a safety floor always wins. Profiles never remove required
|
|
29
|
+
contracts, gates, verification, provenance, lifecycle phases, or validated
|
|
30
|
+
completion. Protocol-v1 routes without the field project to `balanced` for
|
|
31
|
+
compatibility without rewriting historical artifacts.
|
|
32
|
+
|
|
33
|
+
For `light` tasks, hosts should use `next --compact` or `task-show --compact`,
|
|
34
|
+
load only relevant guide sections, keep plans concise, and avoid optional
|
|
35
|
+
reflection, trajectory evaluation, handoff, attestation, and continuity
|
|
36
|
+
artifacts unless requested, required, or needed for recovery. The lifecycle
|
|
37
|
+
chronology remains unchanged.
|
|
38
|
+
|
|
39
|
+
The read-only `task/context` integration resource provides the canonical
|
|
40
|
+
profile-aware projection: objective, deliverables, constraints, selected guide
|
|
41
|
+
IDs, phase, next action, verification requirements, and context policy. A
|
|
42
|
+
profile changes presentation and optional context only; it never permits a
|
|
43
|
+
phase or required gate to be skipped.
|
|
44
|
+
|
|
45
|
+
Usage telemetry is provider or host reported when available, actor-reported
|
|
46
|
+
only through the explicit `usage-record` fallback, and `UNKNOWN` otherwise.
|
|
47
|
+
ForgeLoop never estimates tokens or treats usage as verification evidence.
|
|
48
|
+
`efficiency --task` is read-only and returns `NOT_COMPARABLE` unless a
|
|
49
|
+
project-local baseline has matching metadata.
|
|
50
|
+
|
|
51
|
+
Measured execution-profile benchmarks are observational. The reproducible
|
|
52
|
+
runner accepts provider or host usage, records actual timing, requires PASS
|
|
53
|
+
verification and matching metadata for comparisons, and reports `NOT_MEASURED`
|
|
54
|
+
or `NOT_COMPARABLE` when evidence is absent or incompatible. See
|
|
55
|
+
`docs/EXECUTION_PROFILE_BENCHMARKS.md` for the runner and schemas.
|
|
56
|
+
|
|
22
57
|
## Authority boundaries
|
|
23
58
|
|
|
24
59
|
- Protocol-derived facts outrank actor-provided labels, free-form summaries, and guessed identities.
|
|
@@ -38,14 +73,19 @@ Phases: RECEIVED, DISCOVERING, CONTRACT_READY, ROUTED, DESIGNING, PLANNED, EXECU
|
|
|
38
73
|
|
|
39
74
|
| Feature | Version | Supported |
|
|
40
75
|
| --- | --- | --- |
|
|
76
|
+
| adaptiveExecutionProfiles | 1 | yes |
|
|
41
77
|
| canonicalHandoffs | 1 | yes |
|
|
42
78
|
| capabilityPolicy | 1 | yes |
|
|
43
79
|
| codeAttestation | 1 | yes |
|
|
80
|
+
| compactLifecycleOutput | 1 | yes |
|
|
81
|
+
| contextUsageObservability | 1 | yes |
|
|
44
82
|
| diagnostics | n/a | yes |
|
|
45
83
|
| differentialVerificationScope | 1 | yes |
|
|
46
84
|
| durableActions | 1 | yes |
|
|
47
85
|
| durableApprovals | 1 | yes |
|
|
86
|
+
| efficiencyMetrics | 1 | yes |
|
|
48
87
|
| executionHistory | 1 | yes |
|
|
88
|
+
| executionProfileContext | 1 | yes |
|
|
49
89
|
| integrationApi | 1 | yes |
|
|
50
90
|
| observabilityStability | n/a | yes |
|
|
51
91
|
| reflection | 1 | yes |
|
|
@@ -55,6 +95,7 @@ Phases: RECEIVED, DISCOVERING, CONTRACT_READY, ROUTED, DESIGNING, PLANNED, EXECU
|
|
|
55
95
|
| taskInspection | 1 | yes |
|
|
56
96
|
| trajectoryEvaluation | 1 | yes |
|
|
57
97
|
| trajectoryMetrics | 1 | yes |
|
|
98
|
+
| usageTelemetry | 1 | yes |
|
|
58
99
|
| verificationExecutionIsolation | 1 | yes |
|
|
59
100
|
| workspaceBinding | 1 | yes |
|
|
60
101
|
|
|
@@ -90,6 +131,7 @@ Phases: RECEIVED, DISCOVERING, CONTRACT_READY, ROUTED, DESIGNING, PLANNED, EXECU
|
|
|
90
131
|
| session | SESSION | .forgeloop/sessions/<session-id>.json | activation | SESSION_MARKER |
|
|
91
132
|
| sources | PROJECT | .forgeloop/sources.json | source-registry | SOURCE_ATTESTATION |
|
|
92
133
|
| state | TASK | .forgeloop/task-state/<task-key>/work-state.json | work-state | CANONICAL_LIFECYCLE_STATE |
|
|
134
|
+
| usage | TASK | .forgeloop/task-state/<task-key>/usage.json | usage | INFORMATIONAL_USAGE_TELEMETRY |
|
|
93
135
|
| verificationScope | TASK | .forgeloop/task-state/<task-key>/verification-scope.json | verification-scope | VERIFICATION_SCOPE_PLAN |
|
|
94
136
|
| workspaceBinding | TASK | .forgeloop/task-state/<task-key>/workspace-binding.json | workspace-binding | WORKSPACE_IDENTITY_BINDING |
|
|
95
137
|
|
|
@@ -135,6 +177,7 @@ Phases: RECEIVED, DISCOVERING, CONTRACT_READY, ROUTED, DESIGNING, PLANNED, EXECU
|
|
|
135
177
|
| Command | Mutation | Purpose |
|
|
136
178
|
| --- | --- | --- |
|
|
137
179
|
| doctor | MUTATING | Diagnoses project health, discovers adapters, and optionally repairs missing template files. |
|
|
180
|
+
| efficiency | READ_ONLY | Projects usage and timing efficiency, comparing only against a metadata-compatible local baseline. |
|
|
138
181
|
| eval | MUTATING | Evaluates the current trajectory against a validated project-local reference scenario. |
|
|
139
182
|
| history | READ_ONLY | Shows chronological protocol history reconstructed from canonical ForgeLoop state. |
|
|
140
183
|
| inspect | READ_ONLY | Inspects target repository health, dirty files, active branch, and artifact freshness. |
|
|
@@ -145,6 +188,7 @@ Phases: RECEIVED, DISCOVERING, CONTRACT_READY, ROUTED, DESIGNING, PLANNED, EXECU
|
|
|
145
188
|
| reflect | READ_ONLY | Analyzes diagnostic and correction history deterministically for information gain, repeated failures, ineffective interventions, and oscillation. |
|
|
146
189
|
| status | READ_ONLY | Displays current lifecycle phase, active checks, blockers, and artifact freshness bindings. |
|
|
147
190
|
| trace | READ_ONLY | Emits detailed structured task trace with provenance and artifact relationships. |
|
|
191
|
+
| usage-record | MUTATING | Records actor-reported usage telemetry without treating it as verification evidence. |
|
|
148
192
|
| validate-protocol | READ_ONLY | Validates end-to-end cryptographic freshness, fingerprint bindings, and ledger integrity. |
|
|
149
193
|
| validate-state | READ_ONLY | Validates schema adherence and internal consistency of work-state.json. |
|
|
150
194
|
|
|
@@ -35,6 +35,7 @@ All artifact schemas are defined in `schemas/*.schema.json`. Persisted artifact
|
|
|
35
35
|
| `task-state/<task-key>/approvals/approval-<id>.json` | `approval` | Protocol Managed | Append Decision Once | Action Approval Attestation |
|
|
36
36
|
| `policy/capabilities.json` | `capability-policy` | Operator Or Agent | Mutable Configuration | Capability Policy Specification |
|
|
37
37
|
| `task-state/<task-key>/evaluations/eval-<id>.json` | `trajectory-evaluation` | Protocol Compiled | Immutable Once Written | Trajectory Evaluation |
|
|
38
|
+
| `task-state/<task-key>/usage.json` | `usage` | Actor Or Trusted Host | Overwritten On Usage Record | Informational Usage Telemetry |
|
|
38
39
|
| `task-state/<task-key>/workspace-binding.json` | `workspace-binding` | Protocol Generated | Immutable After Bind | Workspace Identity Binding |
|
|
39
40
|
| `task-state/<task-key>/handoffs/handoff-<id>.json` | `handoff-envelope` | Protocol Compiled | Immutable Once Written | Canonical Handoff Snapshot |
|
|
40
41
|
| `task-state/<task-key>/responsibility.json` | `responsibility` | Agent Or Harness Constrained | Immutable During Pass | Responsibility Constraint |
|
|
@@ -110,6 +111,14 @@ The operational task specification authored before execution begins.
|
|
|
110
111
|
|
|
111
112
|
The deterministic result of routing task signals against the engineering guide router.
|
|
112
113
|
|
|
114
|
+
New routes include a deterministic `executionProfile`. It is separate from
|
|
115
|
+
the project `complianceMode`: the profile controls context/process depth while
|
|
116
|
+
compliance controls enforcement. The persisted route is authoritative for the
|
|
117
|
+
resolved profile and is included in route fingerprints. Historical protocol-v1
|
|
118
|
+
routes may omit this optional field and remain readable; consumers project
|
|
119
|
+
missing profile metadata to `balanced` compatibility behavior without rewriting
|
|
120
|
+
the historical artifact.
|
|
121
|
+
|
|
113
122
|
#### Canonical Fields
|
|
114
123
|
|
|
115
124
|
<!-- BEGIN FORGELOOP GENERATED: schema:routing-result -->
|
|
@@ -117,6 +126,12 @@ The deterministic result of routing task signals against the engineering guide r
|
|
|
117
126
|
- `schemaVersion` *(number, required, const: 1)*
|
|
118
127
|
- `protocolVersion` *(number, required, const: 1)*
|
|
119
128
|
- `contractFingerprint` *(string, optional, pattern: `^[a-f0-9]{64}$`)*
|
|
129
|
+
- `executionProfile` *(object, optional)*
|
|
130
|
+
- `requested` *(string, required, enum: `auto`, `light`, `balanced`, `full`)*
|
|
131
|
+
- `floor` *(string, required, enum: `light`, `balanced`, `full`)*
|
|
132
|
+
- `resolved` *(string, required, enum: `light`, `balanced`, `full`)*
|
|
133
|
+
- `reasons` *(array<string>, required, minItems: 1)*
|
|
134
|
+
- `escalated` *(boolean, required)*
|
|
120
135
|
- `input` *(object, required)*
|
|
121
136
|
- `primary` *(string or null, required)*
|
|
122
137
|
- `guides` *(array<string>, required)*
|
|
@@ -231,6 +246,7 @@ Local ForgeLoop configuration settings and policy bindings.
|
|
|
231
246
|
- `schemaVersion` *(number, required, const: 1)*
|
|
232
247
|
- `protocolVersion` *(number, required, const: 1)*
|
|
233
248
|
- `complianceMode` *(string, required, enum: `advisory`, `standard`, `strict`)*
|
|
249
|
+
- `executionProfile` *(string, optional, enum: `auto`, `light`, `balanced`, `full`)*
|
|
234
250
|
- `policy` *(string, optional, minLength: 1)*
|
|
235
251
|
- `requiredGates` *(array<string>, optional)*
|
|
236
252
|
- `requiredEvidence` *(array<string>, optional)*
|
|
@@ -830,7 +846,37 @@ evidence and never override lifecycle validation.
|
|
|
830
846
|
|
|
831
847
|
<!-- END FORGELOOP GENERATED: schema:trajectory-evaluation -->
|
|
832
848
|
|
|
833
|
-
### 2.24 `task-state/<taskKey>/
|
|
849
|
+
### 2.24 `task-state/<taskKey>/usage.json`
|
|
850
|
+
|
|
851
|
+
<!-- forgeloop-doc: schema=usage artifact=.forgeloop/task-state/<task-key>/usage.json -->
|
|
852
|
+
|
|
853
|
+
Optional task-scoped usage telemetry. Provider and host reports cross the
|
|
854
|
+
trusted runtime boundary; the CLI fallback remains explicitly
|
|
855
|
+
`ACTOR_REPORTED`. Usage is informational and never acts as verification
|
|
856
|
+
evidence.
|
|
857
|
+
|
|
858
|
+
#### Canonical Fields
|
|
859
|
+
|
|
860
|
+
<!-- BEGIN FORGELOOP GENERATED: schema:usage -->
|
|
861
|
+
|
|
862
|
+
- `schemaVersion` *(number, required, const: 1)*
|
|
863
|
+
- `protocolVersion` *(number, required, const: 1)*
|
|
864
|
+
- `taskId` *(string, required, minLength: 1)*
|
|
865
|
+
- `recordedAt` *(string, required, minLength: 1)*
|
|
866
|
+
- `usage` *(object, required)*
|
|
867
|
+
- `inputTokens` *(integer,null, required, minimum: 0)*
|
|
868
|
+
- `outputTokens` *(integer,null, required, minimum: 0)*
|
|
869
|
+
- `cacheReadTokens` *(integer,null, required, minimum: 0)*
|
|
870
|
+
- `cacheWriteTokens` *(integer,null, required, minimum: 0)*
|
|
871
|
+
- `totalTokens` *(integer,null, required, minimum: 0)*
|
|
872
|
+
- `costUsd` *(number,null, required, minimum: 0)*
|
|
873
|
+
- `model` *(string,null, required)*
|
|
874
|
+
- `provider` *(string,null, required)*
|
|
875
|
+
- `source` *(string, required, enum: `PROVIDER_REPORTED`, `HOST_REPORTED`, `ACTOR_REPORTED`, `UNKNOWN`)*
|
|
876
|
+
|
|
877
|
+
<!-- END FORGELOOP GENERATED: schema:usage -->
|
|
878
|
+
|
|
879
|
+
### 2.25 `task-state/<taskKey>/workspace-binding.json`
|
|
834
880
|
|
|
835
881
|
<!-- forgeloop-doc: schema=workspace-binding artifact=.forgeloop/task-state/<task-key>/workspace-binding.json -->
|
|
836
882
|
|
|
@@ -855,7 +901,7 @@ does not fabricate a binding when absent.
|
|
|
855
901
|
|
|
856
902
|
<!-- END FORGELOOP GENERATED: schema:workspace-binding -->
|
|
857
903
|
|
|
858
|
-
### 2.
|
|
904
|
+
### 2.26 `task-state/<taskKey>/handoffs/handoff-<id>.json`
|
|
859
905
|
|
|
860
906
|
<!-- forgeloop-doc: schema=handoff-envelope artifact=.forgeloop/task-state/<task-key>/handoffs/handoff-<id>.json -->
|
|
861
907
|
|
|
@@ -895,7 +941,7 @@ completion, or authority evidence.
|
|
|
895
941
|
|
|
896
942
|
<!-- END FORGELOOP GENERATED: schema:handoff-envelope -->
|
|
897
943
|
|
|
898
|
-
### 2.
|
|
944
|
+
### 2.27 `task-state/<taskKey>/responsibility.json`
|
|
899
945
|
|
|
900
946
|
<!-- forgeloop-doc: schema=responsibility artifact=.forgeloop/task-state/<task-key>/responsibility.json -->
|
|
901
947
|
|
|
@@ -926,7 +972,7 @@ and completion boundaries.
|
|
|
926
972
|
|
|
927
973
|
<!-- END FORGELOOP GENERATED: schema:responsibility -->
|
|
928
974
|
|
|
929
|
-
### 2.
|
|
975
|
+
### 2.28 `task-state/<taskKey>/verification-scope.json`
|
|
930
976
|
|
|
931
977
|
<!-- forgeloop-doc: schema=verification-scope artifact=.forgeloop/task-state/<task-key>/verification-scope.json -->
|
|
932
978
|
|
|
@@ -958,7 +1004,7 @@ otherwise it resolves to `FULL`. ForgeLoop does not publish a heuristic
|
|
|
958
1004
|
|
|
959
1005
|
<!-- END FORGELOOP GENERATED: schema:verification-scope -->
|
|
960
1006
|
|
|
961
|
-
### 2.
|
|
1007
|
+
### 2.29 `task-state/<taskKey>/attestations/code-manifest.json`
|
|
962
1008
|
|
|
963
1009
|
<!-- forgeloop-doc: schema=code-manifest artifact=.forgeloop/task-state/<task-key>/attestations/code-manifest.json -->
|
|
964
1010
|
|
|
@@ -999,7 +1045,7 @@ represented by provider identities, and `.forgeloop/**` is excluded.
|
|
|
999
1045
|
|
|
1000
1046
|
<!-- END FORGELOOP GENERATED: schema:code-manifest -->
|
|
1001
1047
|
|
|
1002
|
-
### 2.
|
|
1048
|
+
### 2.30 `task-state/<taskKey>/attestations/statement.json`
|
|
1003
1049
|
|
|
1004
1050
|
<!-- forgeloop-doc: schema=in-toto-statement artifact=.forgeloop/task-state/<task-key>/attestations/statement.json -->
|
|
1005
1051
|
|
|
@@ -1022,7 +1068,7 @@ back-reference itself from the execution receipt.
|
|
|
1022
1068
|
|
|
1023
1069
|
<!-- END FORGELOOP GENERATED: schema:in-toto-statement -->
|
|
1024
1070
|
|
|
1025
|
-
### 2.
|
|
1071
|
+
### 2.31 `task-state/<taskKey>/attestations/statement.sigstore.json`
|
|
1026
1072
|
|
|
1027
1073
|
<!-- forgeloop-doc: external-artifact=.forgeloop/task-state/<task-key>/attestations/statement.sigstore.json -->
|
|
1028
1074
|
|
package/docs/CLI_REFERENCE.md
CHANGED
|
@@ -31,6 +31,25 @@ ForgeLoop uses a definition-driven command-line parser:
|
|
|
31
31
|
- **Boolean Flags**: Never accept inline values (e.g. `--json=false` is rejected).
|
|
32
32
|
- **Argv Passthrough**: Everything after a command's `--` passthrough marker is preserved exactly and is not parsed as ForgeLoop syntax.
|
|
33
33
|
|
|
34
|
+
### Execution profile and compact context
|
|
35
|
+
|
|
36
|
+
`route` resolves the orthogonal execution profile from `auto`, `light`,
|
|
37
|
+
`balanced`, or `full`. CLI requests take precedence over project configuration,
|
|
38
|
+
but a deterministic safety floor always wins. The profile changes context and
|
|
39
|
+
process depth only; it never bypasses lifecycle phases, gates, verification,
|
|
40
|
+
authority, provenance, or completion validation.
|
|
41
|
+
|
|
42
|
+
For high-frequency host reads, request bounded projections:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
forgeloop next --task <id> --compact --json
|
|
46
|
+
forgeloop task-show --task <id> --compact --json
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Compact output always preserves task identity, phase, resolved profile,
|
|
50
|
+
required next action or status, terminal state where applicable, and blocker
|
|
51
|
+
error codes. Default output and default JSON remain unchanged.
|
|
52
|
+
|
|
34
53
|
---
|
|
35
54
|
|
|
36
55
|
## Command Index by Purpose
|
|
@@ -39,7 +58,7 @@ ForgeLoop uses a definition-driven command-line parser:
|
|
|
39
58
|
|
|
40
59
|
| Category | Commands |
|
|
41
60
|
| --- | --- |
|
|
42
|
-
| **Inspection & Diagnostics** | [`protocol-info`](#protocol-info), [`doctor`](#doctor), [`metrics`](#metrics), [`eval`](#eval), [`history`](#history), [`trace`](#trace), [`reflect`](#reflect), [`progress`](#progress), [`profile-interview`](#profile-interview), [`inspect`](#inspect), [`status`](#status), [`validate-state`](#validate-state), [`validate-protocol`](#validate-protocol) |
|
|
61
|
+
| **Inspection & Diagnostics** | [`protocol-info`](#protocol-info), [`doctor`](#doctor), [`metrics`](#metrics), [`usage-record`](#usage-record), [`efficiency`](#efficiency), [`eval`](#eval), [`history`](#history), [`trace`](#trace), [`reflect`](#reflect), [`progress`](#progress), [`profile-interview`](#profile-interview), [`inspect`](#inspect), [`status`](#status), [`validate-state`](#validate-state), [`validate-protocol`](#validate-protocol) |
|
|
43
62
|
| **Setup & Maintenance** | [`init`](#init), [`update`](#update), [`task-migrate`](#task-migrate), [`migrate-protocol`](#migrate-protocol), [`task-unlock`](#task-unlock), [`task-recover`](#task-recover), [`task-repair-legacy-recovery`](#task-repair-legacy-recovery), [`task-resume`](#task-resume) |
|
|
44
63
|
| **Lifecycle & State** | [`activate`](#activate), [`route`](#route), [`preflight`](#preflight), [`advance`](#advance), [`next`](#next), [`record-diagnosis`](#record-diagnosis), [`record-intervention`](#record-intervention), [`record-hypothesis-disposition`](#record-hypothesis-disposition), [`record-decision-criterion`](#record-decision-criterion), [`complete`](#complete), [`clear-state`](#clear-state), [`reconcile-closure`](#reconcile-closure), [`task-create`](#task-create), [`task-list`](#task-list), [`task-show`](#task-show), [`task-lock-status`](#task-lock-status), [`task-scope`](#task-scope) |
|
|
45
64
|
| **Cross-Harness Continuity** | [`continuity`](#continuity), [`record-continuity`](#record-continuity), [`reconcile-continuity`](#reconcile-continuity), [`clear-continuity`](#clear-continuity), [`handoff-create`](#handoff-create), [`handoff-list`](#handoff-list), [`handoff-show`](#handoff-show) |
|
|
@@ -476,6 +495,60 @@ only exact argv. Caller-reported and externally observed provenance are not host
|
|
|
476
495
|
|
|
477
496
|
<!-- END FORGELOOP GENERATED: cli:metrics:options -->
|
|
478
497
|
|
|
498
|
+
### `usage-record`
|
|
499
|
+
|
|
500
|
+
Records optional actor-reported usage telemetry for a task. This data is
|
|
501
|
+
informational and never satisfies verification or completion requirements.
|
|
502
|
+
|
|
503
|
+
<!-- BEGIN FORGELOOP GENERATED: cli:usage-record:options -->
|
|
504
|
+
|
|
505
|
+
- `--path <directory>`: target project directory (default: current directory)
|
|
506
|
+
- `--task <id>`: task ID to operate on (when omitted, resolved from context or single active task)
|
|
507
|
+
- `--provider <name>`: provider name reported by the actor
|
|
508
|
+
- `--model <name>`: model name reported by the actor
|
|
509
|
+
- `--input-tokens <number>`: provider-reported input token count
|
|
510
|
+
- `--output-tokens <number>`: provider-reported output token count
|
|
511
|
+
- `--cache-read-tokens <number>`: provider-reported cache-read token count
|
|
512
|
+
- `--cache-write-tokens <number>`: provider-reported cache-write token count
|
|
513
|
+
- `--total-tokens <number>`: provider-reported total token count; never estimated
|
|
514
|
+
- `--cost-usd <amount>`: provider-reported cost in USD; never estimated
|
|
515
|
+
- `--source <kind>`: usage source; CLI fallback accepts only ACTOR_REPORTED
|
|
516
|
+
- `--json`: emit usage telemetry as JSON
|
|
517
|
+
|
|
518
|
+
<!-- END FORGELOOP GENERATED: cli:usage-record:options -->
|
|
519
|
+
|
|
520
|
+
- **Example**:
|
|
521
|
+
|
|
522
|
+
```bash
|
|
523
|
+
forgeloop usage-record --task task-001 --provider provider --model provider/model --input-tokens 10000 --output-tokens 2500 --source ACTOR_REPORTED --json
|
|
524
|
+
```
|
|
525
|
+
|
|
526
|
+
The CLI fallback is always actor-reported. It never promotes the source to a
|
|
527
|
+
provider or host observation and never satisfies verification evidence.
|
|
528
|
+
|
|
529
|
+
### `efficiency`
|
|
530
|
+
|
|
531
|
+
Projects timing and usage efficiency. Comparisons are reported only when the
|
|
532
|
+
project-local baseline metadata is compatible with the current task.
|
|
533
|
+
|
|
534
|
+
<!-- BEGIN FORGELOOP GENERATED: cli:efficiency:options -->
|
|
535
|
+
|
|
536
|
+
- `--path <directory>`: target project directory (default: current directory)
|
|
537
|
+
- `--task <id>`: task ID to operate on (when omitted, resolved from context or single active task)
|
|
538
|
+
- `--baseline <path>`: optional comparable efficiency baseline JSON
|
|
539
|
+
- `--json`: emit efficiency metrics as JSON
|
|
540
|
+
|
|
541
|
+
<!-- END FORGELOOP GENERATED: cli:efficiency:options -->
|
|
542
|
+
|
|
543
|
+
- **Example**:
|
|
544
|
+
|
|
545
|
+
```bash
|
|
546
|
+
forgeloop efficiency --task task-001 --baseline benchmark-baseline.json --json
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
A missing or metadata-incompatible baseline produces `NOT_COMPARABLE` and
|
|
550
|
+
null comparison values; ForgeLoop does not estimate token or time overhead.
|
|
551
|
+
|
|
479
552
|
### `eval`
|
|
480
553
|
|
|
481
554
|
<!-- BEGIN FORGELOOP GENERATED: cli:eval:options -->
|
|
@@ -631,6 +704,7 @@ Calculates and persists deterministic engineering guide routing.
|
|
|
631
704
|
- `--platform <value>`: affected platform (repeatable)
|
|
632
705
|
- `--behavior-change`: declare behavior change
|
|
633
706
|
- `--executable-change`: declare executable/configuration change
|
|
707
|
+
- `--execution-profile <profile>`: requested execution profile; safety floors always win
|
|
634
708
|
- `--json`: emit route result as JSON
|
|
635
709
|
|
|
636
710
|
<!-- END FORGELOOP GENERATED: cli:route:options -->
|
|
@@ -641,6 +715,12 @@ Calculates and persists deterministic engineering guide routing.
|
|
|
641
715
|
forgeloop route --work complete-website --surface ui --risk untrusted-input --json
|
|
642
716
|
```
|
|
643
717
|
|
|
718
|
+
A requested profile is an override request, not a safety bypass:
|
|
719
|
+
|
|
720
|
+
```bash
|
|
721
|
+
forgeloop route --task landing-page --work complete-website --surface ui --execution-profile light --json
|
|
722
|
+
```
|
|
723
|
+
|
|
644
724
|
### `preflight`
|
|
645
725
|
|
|
646
726
|
Validates pre-implementation readiness and establishes protocol readiness state.
|
|
@@ -752,6 +832,7 @@ Computes the deterministic next action required by the protocol.
|
|
|
752
832
|
|
|
753
833
|
- `--path <directory>`: target project directory (default: current directory)
|
|
754
834
|
- `--task <id>`: task ID to operate on (when omitted, resolved from context or single active task)
|
|
835
|
+
- `--compact`: emit a bounded next-action projection
|
|
755
836
|
- `--json`: emit structured output as JSON
|
|
756
837
|
|
|
757
838
|
<!-- END FORGELOOP GENERATED: cli:next:options -->
|
|
@@ -1751,6 +1832,7 @@ Displays details of a specific task by ID or storage key.
|
|
|
1751
1832
|
|
|
1752
1833
|
- `--path <directory>`: target project directory (default: current directory)
|
|
1753
1834
|
- `--task <id>`: task ID to operate on (when omitted, resolved from context or single active task)
|
|
1835
|
+
- `--compact`: emit a bounded task-status projection
|
|
1754
1836
|
- `--json`: emit structured output as JSON
|
|
1755
1837
|
|
|
1756
1838
|
<!-- END FORGELOOP GENERATED: cli:task-show:options -->
|
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
# Execution-profile benchmark protocol
|
|
2
|
+
|
|
3
|
+
ForgeLoop keeps execution-profile efficiency evidence separate from lifecycle
|
|
4
|
+
trust. The benchmark scenarios under
|
|
5
|
+
[`benchmarks/execution-profiles/`](../benchmarks/execution-profiles/) are
|
|
6
|
+
deterministic routing specifications. They are not measurements and they do
|
|
7
|
+
not authorize a performance claim by themselves.
|
|
8
|
+
|
|
9
|
+
## What is measured
|
|
10
|
+
|
|
11
|
+
The runner executes every scenario in these modes:
|
|
12
|
+
|
|
13
|
+
| Mode | Meaning |
|
|
14
|
+
| --- | --- |
|
|
15
|
+
| `direct` | The direct baseline supplied by the host adapter. |
|
|
16
|
+
| `forgeloopBalanced` | The normal ForgeLoop context and process presentation. |
|
|
17
|
+
| `forgeloopAdaptive` | The deterministic profile resolved from the scenario, contract, and scope. |
|
|
18
|
+
|
|
19
|
+
Each run records provider or host usage, runner-measured wall-clock time,
|
|
20
|
+
verification outcome, verification cycles, comparable work steps, and the
|
|
21
|
+
metadata needed to determine whether a comparison is valid. The runner never
|
|
22
|
+
derives token counts, costs, cache values, verification results, or comparable
|
|
23
|
+
steps from prompt length, elapsed time, model names, or estimates.
|
|
24
|
+
|
|
25
|
+
The required reference set contains the six original scenarios plus
|
|
26
|
+
`novatask-saas-landing-page`: documentation correction, static landing page,
|
|
27
|
+
small bug fix, API feature, authentication change, infrastructure/release, and
|
|
28
|
+
the NovaTask static SaaS landing page. NovaTask is a local HTML5/CSS3/vanilla
|
|
29
|
+
JavaScript workload with no external services, authentication, secrets, or
|
|
30
|
+
publication, and must resolve to `light`.
|
|
31
|
+
|
|
32
|
+
Trusted efficiency comparisons require all of the following:
|
|
33
|
+
|
|
34
|
+
- `PROVIDER_REPORTED` or `HOST_REPORTED` usage;
|
|
35
|
+
- actual non-null token and timing values for the metric being compared;
|
|
36
|
+
- `PASS` verification for both paired runs;
|
|
37
|
+
- a positive comparable-step count; and
|
|
38
|
+
- matching model, provider, prompt-spec fingerprint, project revision,
|
|
39
|
+
benchmark version, and environment metadata.
|
|
40
|
+
|
|
41
|
+
If any condition is missing, the aggregate reports `NOT_COMPARABLE` and
|
|
42
|
+
`claimsAllowed: false` for that comparison. `UNKNOWN` is a valid telemetry
|
|
43
|
+
state, not a zero value. Benchmark outcomes are observational and never relax
|
|
44
|
+
the lifecycle, evidence, authority, provenance, safety-floor, or completion
|
|
45
|
+
requirements.
|
|
46
|
+
|
|
47
|
+
## Running a benchmark
|
|
48
|
+
|
|
49
|
+
The command requires a host adapter because ForgeLoop cannot create real model
|
|
50
|
+
or provider measurements on its own:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npm run benchmark:profiles -- \
|
|
54
|
+
--adapter ./path/to/actual-host-adapter.mjs \
|
|
55
|
+
--runs 5 \
|
|
56
|
+
--json
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
The adapter exports `runBenchmark(input)` and must execute the supplied
|
|
60
|
+
scenario. Its result must include actual usage and verification data:
|
|
61
|
+
|
|
62
|
+
```js
|
|
63
|
+
export async function runBenchmark({ scenario, mode, runIndex, target }) {
|
|
64
|
+
// Execute the host's real workload here.
|
|
65
|
+
return {
|
|
66
|
+
usage: {
|
|
67
|
+
inputTokens: 120,
|
|
68
|
+
outputTokens: 80,
|
|
69
|
+
cacheReadTokens: null,
|
|
70
|
+
cacheWriteTokens: null,
|
|
71
|
+
totalTokens: 200,
|
|
72
|
+
costUsd: null,
|
|
73
|
+
model: "host-reported-model",
|
|
74
|
+
provider: "host-reported-provider",
|
|
75
|
+
source: "HOST_REPORTED",
|
|
76
|
+
},
|
|
77
|
+
promptSpecFingerprint: "sha256-or-other-host-bound-fingerprint",
|
|
78
|
+
verification: "PASS",
|
|
79
|
+
verificationCycles: 1,
|
|
80
|
+
comparableSteps: 4,
|
|
81
|
+
};
|
|
82
|
+
}
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
The values above are shape-only examples. They must not be copied as evidence
|
|
86
|
+
unless the host actually reports them. The runner owns the elapsed-time
|
|
87
|
+
measurement and derives the Git revision from the target checkout. Results
|
|
88
|
+
are written under a unique run-set directory and existing history is never
|
|
89
|
+
overwritten.
|
|
90
|
+
|
|
91
|
+
An adapter may also return host-observed context usage. ForgeLoop does not
|
|
92
|
+
tokenize provider prompts. The optional shape is:
|
|
93
|
+
|
|
94
|
+
```json
|
|
95
|
+
{
|
|
96
|
+
"contextUsage": {
|
|
97
|
+
"source": "HOST_REPORTED",
|
|
98
|
+
"profile": "light",
|
|
99
|
+
"items": {
|
|
100
|
+
"taskContext": null,
|
|
101
|
+
"guides": null,
|
|
102
|
+
"history": null,
|
|
103
|
+
"protocolInstructions": null,
|
|
104
|
+
"repositoryContext": null,
|
|
105
|
+
"other": null
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Every item is nullable and `UNKNOWN` requires all items to remain `null`.
|
|
112
|
+
Missing values are never inferred. When adaptive resolves to `light`, the
|
|
113
|
+
aggregate compares complete host-reported context items with the matching
|
|
114
|
+
balanced run and reports `CONTEXT_INFLATION` when light context is larger.
|
|
115
|
+
This is an observational diagnostic and never blocks lifecycle completion.
|
|
116
|
+
|
|
117
|
+
UI adapters may additionally return independently evaluated quality scores:
|
|
118
|
+
|
|
119
|
+
```json
|
|
120
|
+
{
|
|
121
|
+
"quality": {
|
|
122
|
+
"source": "EXTERNAL_REPORTED",
|
|
123
|
+
"scores": {
|
|
124
|
+
"visualQuality": null,
|
|
125
|
+
"responsiveQuality": null,
|
|
126
|
+
"accessibility": null,
|
|
127
|
+
"interactionPolish": null,
|
|
128
|
+
"requirementsCompleteness": null
|
|
129
|
+
}
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Scores are on a 0–5 scale and remain `UNKNOWN` when no external evaluator or
|
|
135
|
+
host observation is available. Implementer self-ratings should not be used as
|
|
136
|
+
independent quality evidence; for NovaTask, review direct and ForgeLoop
|
|
137
|
+
outputs blind to execution mode before revealing the labels.
|
|
138
|
+
|
|
139
|
+
## Inspecting and validating results
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
npm run benchmark:profiles:summary -- --json
|
|
143
|
+
npm run benchmark:profiles:check -- --json
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
Before a host run exists, the summary is intentionally:
|
|
147
|
+
|
|
148
|
+
```json
|
|
149
|
+
{
|
|
150
|
+
"status": "NOT_MEASURED",
|
|
151
|
+
"claimsAllowed": false
|
|
152
|
+
}
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Raw runs are stored below `benchmarks/execution-profiles/results/raw/` and
|
|
156
|
+
recomputed aggregates below `benchmarks/execution-profiles/results/aggregate/`.
|
|
157
|
+
The validator checks all scenario, raw-run, and aggregate schemas and
|
|
158
|
+
recomputes aggregates from raw measurements. A changed overhead target is
|
|
159
|
+
observable in the report; it is not a blocking lifecycle gate.
|
|
160
|
+
|
|
161
|
+
CI can run `npm run benchmark:profiles:regression -- --json`. It reports
|
|
162
|
+
`OK`, `EFFICIENCY_REGRESSION`, `NOT_MEASURED`, or `NOT_COMPARABLE`; a regression
|
|
163
|
+
is a release-quality warning, not a ForgeLoop lifecycle failure.
|
|
164
|
+
|
|
165
|
+
For LIGHT scenarios, the initial non-blocking objectives are P50 token
|
|
166
|
+
overhead no greater than +35% and P95 token overhead no greater than +60%
|
|
167
|
+
against a comparable direct baseline. These objectives never trade away
|
|
168
|
+
verification quality, requirement coverage, or protocol safety.
|
|
169
|
+
|
|
170
|
+
## Host context contract
|
|
171
|
+
|
|
172
|
+
The universal integration API exposes `task/context`, and the MCP adapter
|
|
173
|
+
registers it as `forgeloop://task/{taskId}/context`. The projection includes
|
|
174
|
+
the resolved profile, objective, deliverables, constraints, selected guide
|
|
175
|
+
IDs, next action, verification requirements, and an explicit context policy:
|
|
176
|
+
|
|
177
|
+
- `light`: targeted context, compact output, short planning, focused checks,
|
|
178
|
+
and lazy optional artifacts;
|
|
179
|
+
- `balanced`: relevant context, standard output and planning, normal checks;
|
|
180
|
+
- `full`: expanded context, deep planning, broad risk context, and expanded
|
|
181
|
+
verification when justified.
|
|
182
|
+
|
|
183
|
+
Every projection also states that lifecycle phases, required gates, evidence,
|
|
184
|
+
verification truth, authority, provenance, the safety floor, and validated
|
|
185
|
+
completion remain unchanged. A host may omit optional presentation context,
|
|
186
|
+
but it may not use `light` to skip a required lifecycle phase or gate.
|
|
187
|
+
|
|
188
|
+
Historical protocol-v1 routes without profile metadata remain readable and are
|
|
189
|
+
projected to the balanced compatibility profile.
|
|
190
|
+
|
|
191
|
+
For UI scenarios, hosts should report independent visual, responsive,
|
|
192
|
+
accessibility, interaction-polish, and requirements-completeness scores when
|
|
193
|
+
an external evaluator or blind human review is available. Implementer
|
|
194
|
+
self-ratings are not trusted as a substitute. NovaTask comparisons should
|
|
195
|
+
hide the execution mode until this review is complete.
|
|
196
|
+
|
|
197
|
+
## Schemas and source policy
|
|
198
|
+
|
|
199
|
+
The machine-readable contracts are:
|
|
200
|
+
|
|
201
|
+
- `schemas/execution-profile-benchmark-scenario.schema.json`;
|
|
202
|
+
- `schemas/execution-profile-benchmark-run.schema.json`; and
|
|
203
|
+
- `schemas/execution-profile-benchmark-aggregate.schema.json`.
|
|
204
|
+
|
|
205
|
+
The source policy is intentionally narrow: provider or host observations are
|
|
206
|
+
accepted for benchmark claims, while actor-reported or absent telemetry stays
|
|
207
|
+
non-comparable. This is an efficiency observation boundary, not verification
|
|
208
|
+
evidence and not an external publication result.
|