@cassiomc1/forgeloop 1.6.4 → 1.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (73) hide show
  1. package/AGENT_COMPATIBILITY.md +11 -0
  2. package/DOCS_INDEX.md +1 -0
  3. package/GUIDE_ROUTER.md +26 -1
  4. package/LOOP_ENGINEERING.md +48 -0
  5. package/ORCHESTRATOR_INTEGRATION.md +38 -0
  6. package/PROTOCOL_INTEGRATION.md +62 -0
  7. package/README.md +25 -0
  8. package/benchmarks/execution-profiles/README.md +44 -0
  9. package/benchmarks/execution-profiles/api-feature.json +18 -0
  10. package/benchmarks/execution-profiles/authentication-change.json +18 -0
  11. package/benchmarks/execution-profiles/documentation-correction.json +18 -0
  12. package/benchmarks/execution-profiles/infrastructure-release.json +18 -0
  13. package/benchmarks/execution-profiles/novatask-saas-landing-page.json +36 -0
  14. package/benchmarks/execution-profiles/small-bug-fix.json +18 -0
  15. package/benchmarks/execution-profiles/static-landing-page.json +18 -0
  16. package/completions/_forgeloop +6 -4
  17. package/completions/forgeloop.bash +8 -4
  18. package/completions/forgeloop.fish +24 -1
  19. package/docs/AGENT_PROTOCOL_SUMMARY.md +45 -1
  20. package/docs/ARTIFACT_REFERENCE.md +53 -7
  21. package/docs/CLI_REFERENCE.md +83 -1
  22. package/docs/EXECUTION_PROFILE_BENCHMARKS.md +208 -0
  23. package/docs/GETTING_STARTED.md +28 -0
  24. package/docs/MCP.md +6 -0
  25. package/docs/RELEASE_CHECKLIST.md +4 -0
  26. package/docs/TROUBLESHOOTING.md +6 -0
  27. package/docs/UNIVERSAL_INTEGRATION.md +58 -0
  28. package/package.json +14 -2
  29. package/schemas/config.schema.json +1 -0
  30. package/schemas/execution-profile-benchmark-aggregate.schema.json +43 -0
  31. package/schemas/execution-profile-benchmark-run.schema.json +106 -0
  32. package/schemas/execution-profile-benchmark-scenario.schema.json +66 -0
  33. package/schemas/routing-result.schema.json +12 -0
  34. package/schemas/usage.schema.json +30 -0
  35. package/scripts/check-efficiency-regression.mjs +99 -0
  36. package/scripts/generate-agent-protocol-summary.mjs +35 -0
  37. package/scripts/lib/execution-profile-benchmark-io.mjs +67 -0
  38. package/scripts/run-execution-profile-benchmarks.mjs +265 -0
  39. package/scripts/summarize-execution-profile-benchmarks.mjs +84 -0
  40. package/scripts/validate-execution-profile-benchmarks.mjs +120 -0
  41. package/src/cli.js +16 -4
  42. package/src/commands/efficiency.js +12 -0
  43. package/src/commands/eval.js +8 -2
  44. package/src/commands/metrics.js +2 -2
  45. package/src/commands/next.js +26 -2
  46. package/src/commands/route.js +20 -2
  47. package/src/commands/task-show.js +31 -2
  48. package/src/commands/usage-record.js +61 -0
  49. package/src/core/artifact-registry.js +12 -0
  50. package/src/core/cli-command-definitions.js +27 -0
  51. package/src/core/command-executors.js +29 -5
  52. package/src/core/command-input.js +34 -0
  53. package/src/core/config.js +9 -0
  54. package/src/core/efficiency.js +197 -0
  55. package/src/core/error-codes.js +18 -0
  56. package/src/core/execution-profile-benchmarks.js +674 -0
  57. package/src/core/execution-profile-context.js +177 -0
  58. package/src/core/execution-profile.js +248 -0
  59. package/src/core/integration-invocation-policy.js +43 -0
  60. package/src/core/integration-resources.js +22 -1
  61. package/src/core/protocol-info.js +42 -0
  62. package/src/core/resumability.js +27 -1
  63. package/src/core/router.js +23 -1
  64. package/src/core/runtime-context.js +11 -0
  65. package/src/core/schema-validation.js +4 -0
  66. package/src/core/task-paths.js +2 -0
  67. package/src/core/templates.js +4 -0
  68. package/src/core/trace.js +1 -0
  69. package/src/core/trajectory-evaluation.js +2 -2
  70. package/src/core/trajectory-metrics.js +18 -2
  71. package/src/core/usage.js +137 -0
  72. package/src/integration.d.ts +84 -0
  73. package/src/integration.js +14 -0
@@ -26,6 +26,34 @@ Core mental model:
26
26
  - `checks = verification truth`
27
27
  - `receipt = completion/publication record`
28
28
 
29
+ ### Adaptive execution profiles
30
+
31
+ ForgeLoop uses `executionProfile` to choose the appropriate process and context
32
+ depth without changing its assurance guarantees. It is separate from
33
+ `complianceMode`, which controls policy enforcement.
34
+
35
+ ```text
36
+ auto → light | balanced | full
37
+ ```
38
+
39
+ Small, local, reversible documentation and UI work normally resolves to
40
+ `light`; behavior changes and ordinary application work resolve to
41
+ `balanced`; authentication, publication, infrastructure, secrets, personal
42
+ data, critical paths, and irreversible work resolve to `full`. A CLI or project
43
+ request may raise the profile, but never lower the deterministic safety floor.
44
+
45
+ For compact host context, use:
46
+
47
+ ```bash
48
+ forgeloop next --task task-contact-form-001 --compact --json
49
+ forgeloop task-show --task task-contact-form-001 --compact --json
50
+ ```
51
+
52
+ These commands do not bypass contracts, gates, verification, provenance,
53
+ lifecycle phases, or validator-backed completion. Usage telemetry is optional,
54
+ never estimated, and never verification evidence; `efficiency --task` compares
55
+ only against a metadata-compatible local baseline.
56
+
29
57
  ---
30
58
 
31
59
  ## 2. Prerequisites
package/docs/MCP.md CHANGED
@@ -50,6 +50,7 @@ Capability flags (process-scoped, immutable after launch):
50
50
  - `forgeloop://task/{taskId}/action/{actionId}`
51
51
  - `forgeloop://task/{taskId}/approvals`
52
52
  - `forgeloop://task/{taskId}/metrics`
53
+ - `forgeloop://task/{taskId}/context` — bounded profile-aware host context
53
54
  - `forgeloop://task/{taskId}/evaluations`
54
55
  - `forgeloop://project/capability-policy`
55
56
 
@@ -83,6 +84,11 @@ content.
83
84
  Raw recovery artifacts, transaction journals, lock files, and unbounded event
84
85
  ledgers are intentionally not exposed.
85
86
 
87
+ The context resource is read-only. It lets an MCP host adapt presentation depth
88
+ from the canonical resolved execution profile while preserving lifecycle
89
+ phases, required gates, verification truth, authority, provenance, and
90
+ validator-backed completion.
91
+
86
92
  Optional workspace binding, handoff, responsibility, verification-scope,
87
93
  RevisionProvider, SigningProvider, and attestation operations remain governed
88
94
  by the same canonical command executors when exposed by a compatible host.
@@ -16,6 +16,10 @@ preparation and verification checklist; it does not authorize publication.
16
16
  - [ ] `npm run dependency:policy` passes without adding runtime dependencies.
17
17
  - [ ] `npm run lint` passes.
18
18
  - [ ] `npm test` passes.
19
+ - [ ] `npm run benchmark:profiles:check` passes; absent provider/host history
20
+ is reported as `NOT_MEASURED`, never as zero or a passing efficiency claim.
21
+ - [ ] `npm run benchmark:profiles:regression` reports the observed status;
22
+ `EFFICIENCY_REGRESSION` remains a non-blocking warning.
19
23
  - [ ] `npm run coverage` passes the configured global and critical-module gates.
20
24
  - [ ] `npm run docs:check`, `npm run docs:generated:check`,
21
25
  `npm run docs:conformance`, and `npm run docs:examples:check` pass.
@@ -986,6 +986,7 @@ forgeloop next --task <id> --json
986
986
  | `E_DIAGNOSTIC_CASE_CYCLE_MISMATCH` | Diagnostic case verification cycle does not match the active work state verification cycle. | Record the diagnostic case for the current active verification cycle. |
987
987
  | `E_DIAGNOSTIC_CASE_EVIDENCE_INVALID` | A diagnostic case evidence reference does not match any check from the active verification cycle. | Reference check IDs recorded during the active verification cycle. |
988
988
  | `E_DIAGNOSTIC_CASE_INVALID` | Structured diagnostic case details or parameters are malformed. | Provide valid observations, contributors, hypotheses with settlement criteria, and nextSafeAction. |
989
+ | `E_EFFICIENCY_BASELINE_INVALID` | A ForgeLoop boundary, artifact, provider, or attestation validation condition was not satisfied. | Inspect the structured command result, correct the named boundary or artifact, then retry the canonical command. |
989
990
  | `E_EVIDENCE_COVERAGE_INVALID` | A ForgeLoop protocol validation or lifecycle condition was not satisfied. | Inspect the structured command result, correct the named artifact or prerequisite, then run forgeloop next --json. |
990
991
  | `E_EVIDENCE_COVERAGE_PARTIAL` | A ForgeLoop protocol validation or lifecycle condition was not satisfied. | Inspect the structured command result, correct the named artifact or prerequisite, then run forgeloop next --json. |
991
992
  | `E_EVIDENCE_INVALID` | A ForgeLoop protocol validation or lifecycle condition was not satisfied. | Inspect the structured command result, correct the named artifact or prerequisite, then run forgeloop next --json. |
@@ -993,6 +994,9 @@ forgeloop next --task <id> --json
993
994
  | `E_EVIDENCE_PARTIAL` | A ForgeLoop protocol validation or lifecycle condition was not satisfied. | Inspect the structured command result, correct the named artifact or prerequisite, then run forgeloop next --json. |
994
995
  | `E_EVIDENCE_REQUIRED` | A ForgeLoop protocol validation or lifecycle condition was not satisfied. | Inspect the structured command result, correct the named artifact or prerequisite, then run forgeloop next --json. |
995
996
  | `E_EVIDENCE_STALE` | A ForgeLoop protocol validation or lifecycle condition was not satisfied. | Inspect the structured command result, correct the named artifact or prerequisite, then run forgeloop next --json. |
997
+ | `E_EXECUTION_PROFILE_INCONSISTENT` | A ForgeLoop boundary, artifact, provider, or attestation validation condition was not satisfied. | Inspect the structured command result, correct the named boundary or artifact, then retry the canonical command. |
998
+ | `E_EXECUTION_PROFILE_INVALID` | A ForgeLoop boundary, artifact, provider, or attestation validation condition was not satisfied. | Inspect the structured command result, correct the named boundary or artifact, then retry the canonical command. |
999
+ | `E_EXECUTION_PROFILE_SAFETY_FLOOR_INVALID` | A ForgeLoop boundary, artifact, provider, or attestation validation condition was not satisfied. | Inspect the structured command result, correct the named boundary or artifact, then retry the canonical command. |
996
1000
  | `E_EXECUTION_REF_INVALID` | Referenced execution ID does not exist. | Re-run check via forgeloop run-check. |
997
1001
  | `E_FAILURE_SIGNATURE_INVALID` | A ForgeLoop protocol validation or lifecycle condition was not satisfied. | Inspect the structured command result, correct the named artifact or prerequisite, then run forgeloop next --json. |
998
1002
  | `E_FUTURE_LIFECYCLE_EVIDENCE` | A ForgeLoop protocol validation or lifecycle condition was not satisfied. | Inspect the structured command result, correct the named artifact or prerequisite, then run forgeloop next --json. |
@@ -1123,6 +1127,8 @@ forgeloop next --task <id> --json
1123
1127
  | `E_TRACE_SNAPSHOT_INCONSISTENT` | Task artifacts changed while the execution trace was being read. | Rerun the read-only projection to obtain a consistent view. |
1124
1128
  | `E_TRAJECTORY_REFERENCE_REQUIRED` | Comparative efficiency requires a reference scenario with positive comparableSteps. | Provide --scenario with reference.comparableSteps, or omit efficiency from the result. |
1125
1129
  | `E_TRAJECTORY_SCENARIO_INVALID` | Trajectory scenario file is missing required fields or schema-invalid. | Correct the scenario JSON against schemas/trajectory-scenario.schema.json. |
1130
+ | `E_USAGE_INVALID` | A ForgeLoop boundary, artifact, provider, or attestation validation condition was not satisfied. | Inspect the structured command result, correct the named boundary or artifact, then retry the canonical command. |
1131
+ | `E_USAGE_SOURCE_INVALID` | A ForgeLoop boundary, artifact, provider, or attestation validation condition was not satisfied. | Inspect the structured command result, correct the named boundary or artifact, then retry the canonical command. |
1126
1132
  | `E_VERIFICATION_EXECUTION_INVALID` | The trusted verification execution adapter returned incomplete or invalid execution metadata. | Repair the adapter contract and rerun verification; do not promote incomplete execution evidence. |
1127
1133
  | `E_VERIFICATION_ISOLATION_UNAVAILABLE` | Verification cannot run because the required disposable or system isolation boundary is unavailable. | Use a trusted ForgeLoop execution adapter with the required isolation mode; never run the check in the live project. |
1128
1134
  | `E_VERIFICATION_SCOPE_INVALID` | A ForgeLoop boundary, artifact, provider, or attestation validation condition was not satisfied. | Inspect the structured command result, correct the named boundary or artifact, then retry the canonical command. |
@@ -35,6 +35,64 @@ import {
35
35
  FORCE_DESTRUCTIVE) describe what an invocation would do; launch policy
36
36
  decides what is allowed.
37
37
 
38
+ ### Profile-aware context
39
+
40
+ The resolved execution profile is available from the persisted route and from
41
+ compact `next`/`task-show` projections. `complianceMode` remains the policy
42
+ enforcement dimension; `executionProfile` is the independent process/context
43
+ depth dimension.
44
+
45
+ The API's read-only `task/context` resource provides the canonical host
46
+ projection for this decision. It includes the objective, deliverables,
47
+ constraints, selected guide IDs, current phase, next action, verification
48
+ requirements, resolved profile, and bounded optional-context policy. The MCP
49
+ adapter exposes the same projection at
50
+ `forgeloop://task/{taskId}/context`.
51
+
52
+ For `light`, an adapter should send only the current objective, deliverables,
53
+ hard constraints, resolved profile, selected guide IDs or relevant guide
54
+ sections, current phase, exact next action, and verification requirements. It
55
+ should reuse unchanged state locally instead of repeatedly sending full
56
+ ledgers, schemas, receipts, or protocol documents. This is presentation
57
+ optimization only: required gates, verification truth, authority, provenance,
58
+ and validated completion remain unchanged. A lifecycle fast path is not part
59
+ of protocol v1.
60
+
61
+ Optional reflection, trajectory evaluation, handoff, responsibility,
62
+ attestation, benchmark, and continuity artifacts are lazy and should be
63
+ created only when policy, an explicit request, or recovery requires them.
64
+
65
+ The profile never authorizes a lifecycle-phase or required-gate skip. A phase
66
+ may be absent only when the canonical protocol marks it not applicable;
67
+ presentation depth cannot change evidence, verification truth, authority,
68
+ provenance, safety-floor, or validator-backed completion requirements.
69
+
70
+ ### Usage and efficiency boundary
71
+
72
+ Create a context with an optional trusted provider:
73
+
74
+ ```js
75
+ const context = createForgeLoopContext({
76
+ usageProvider: {
77
+ async getTaskUsage({ projectPath, taskId }) {
78
+ return hostUsageStore.lookup({ projectPath, taskId });
79
+ },
80
+ },
81
+ });
82
+ ```
83
+
84
+ The provider returns `PROVIDER_REPORTED` or `HOST_REPORTED` snapshots. The CLI
85
+ fallback is explicitly `ACTOR_REPORTED`; missing fields remain `null` and no
86
+ token or cost estimate is produced. Usage never satisfies a check or
87
+ completion requirement. Use `efficiency --task --baseline <project-local-json>`
88
+ only for metadata-compatible comparisons; otherwise the result is
89
+ `NOT_COMPARABLE`.
90
+
91
+ For measured execution-profile comparisons, use the reproducible benchmark
92
+ commands in [`EXECUTION_PROFILE_BENCHMARKS.md`](./EXECUTION_PROFILE_BENCHMARKS.md).
93
+ The benchmark source policy accepts provider or host observations only and
94
+ keeps unavailable or non-comparable measurements out of efficiency claims.
95
+
38
96
  ## Consumers
39
97
 
40
98
  | Surface | Entry |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@cassiomc1/forgeloop",
3
- "version": "1.6.4",
3
+ "version": "1.7.0",
4
4
  "description": "Portable, verifiable engineering protocol for AI coding environments and developer workflows",
5
5
  "repository": {
6
6
  "type": "git",
@@ -22,12 +22,14 @@
22
22
  "src",
23
23
  "ENG",
24
24
  "schemas",
25
+ "benchmarks/execution-profiles",
25
26
  ".forgeloop/forgeloop.gitignore",
26
27
  "AGENTS.md",
27
28
  "CLAUDE.md",
28
29
  "GUIDE_ROUTER.md",
29
30
  "LOOP_ENGINEERING.md",
30
31
  "PROTOCOL_INTEGRATION.md",
32
+ "AGENT_COMPATIBILITY.md",
31
33
  "LOOP_SYSTEM_DESIGN.md",
32
34
  "QUALITY_SCORECARD.md",
33
35
  "TERMINOLOGY.md",
@@ -64,8 +66,14 @@
64
66
  "docs/SIGNING_PROVIDERS.md",
65
67
  "docs/PLATFORM_ADAPTERS.md",
66
68
  "docs/AGENT_PROTOCOL_SUMMARY.md",
69
+ "docs/EXECUTION_PROFILE_BENCHMARKS.md",
67
70
  "scripts/generate-shell-completions.mjs",
68
71
  "scripts/generate-agent-protocol-summary.mjs",
72
+ "scripts/run-execution-profile-benchmarks.mjs",
73
+ "scripts/summarize-execution-profile-benchmarks.mjs",
74
+ "scripts/validate-execution-profile-benchmarks.mjs",
75
+ "scripts/check-efficiency-regression.mjs",
76
+ "scripts/lib/execution-profile-benchmark-io.mjs",
69
77
  "scripts/benchmark-cli-startup.mjs",
70
78
  "scripts/check-critical-coverage.mjs",
71
79
  "scripts/check-changelog-freshness.mjs",
@@ -103,7 +111,11 @@
103
111
  "mcp:setup": "node scripts/mcp-setup.mjs",
104
112
  "mcp:pack:check": "node scripts/mcp-package-smoke.mjs",
105
113
  "poc:evidence:verify": "node scripts/verify_poc_evidence.mjs",
106
- "poc:evidence:test": "node --test poc/test/poc-evidence-publication.test.js"
114
+ "poc:evidence:test": "node --test poc/test/poc-evidence-publication.test.js",
115
+ "benchmark:profiles": "node scripts/run-execution-profile-benchmarks.mjs",
116
+ "benchmark:profiles:summary": "node scripts/summarize-execution-profile-benchmarks.mjs",
117
+ "benchmark:profiles:check": "node scripts/validate-execution-profile-benchmarks.mjs",
118
+ "benchmark:profiles:regression": "node scripts/check-efficiency-regression.mjs"
107
119
  },
108
120
  "devDependencies": {
109
121
  "c8": "^12.0.0",
@@ -8,6 +8,7 @@
8
8
  "schemaVersion": { "const": 1 },
9
9
  "protocolVersion": { "const": 1 },
10
10
  "complianceMode": { "enum": ["advisory", "standard", "strict"] },
11
+ "executionProfile": { "enum": ["auto", "light", "balanced", "full"] },
11
12
  "policy": { "type": "string", "minLength": 1 },
12
13
  "requiredGates": { "type": "array", "items": { "type": "string", "minLength": 1 } },
13
14
  "requiredEvidence": { "type": "array", "items": { "type": "string", "minLength": 1 } },
@@ -0,0 +1,43 @@
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "$id": "forgeloop://schemas/execution-profile-benchmark-aggregate.schema.json",
4
+ "title": "ForgeLoop execution-profile benchmark aggregate",
5
+ "type": "object",
6
+ "required": ["schemaVersion", "benchmarkVersion", "runSetId", "scenarioId", "expectedProfile", "modeAggregates", "comparisons", "lightObjectives", "sourcePolicy", "claimsAllowed", "generatedFromRunCount"],
7
+ "properties": {
8
+ "schemaVersion": { "const": 1 },
9
+ "benchmarkVersion": { "const": "1" },
10
+ "runSetId": { "type": "string", "minLength": 1 },
11
+ "scenarioId": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{0,127}$" },
12
+ "expectedProfile": { "enum": ["light", "balanced", "full"] },
13
+ "modeAggregates": {
14
+ "type": "object",
15
+ "required": ["direct", "forgeloopBalanced", "forgeloopAdaptive"],
16
+ "additionalProperties": { "type": "object" }
17
+ },
18
+ "comparisons": {
19
+ "type": "object",
20
+ "required": ["direct", "forgeloopBalanced", "forgeloopAdaptive"],
21
+ "additionalProperties": true
22
+ },
23
+ "lightObjectives": { "type": ["object", "null"] },
24
+ "contextInflation": {
25
+ "type": "object",
26
+ "required": ["status", "comparablePairs", "inflatedPairs", "lightContextTokens", "balancedContextTokens", "blocking", "reason"],
27
+ "properties": {
28
+ "status": { "enum": ["CONTEXT_INFLATION", "NOT_DETECTED", "NOT_COMPARABLE"] },
29
+ "comparablePairs": { "type": "integer", "minimum": 0 },
30
+ "inflatedPairs": { "type": "integer", "minimum": 0 },
31
+ "lightContextTokens": { "type": "object" },
32
+ "balancedContextTokens": { "type": "object" },
33
+ "blocking": { "const": false },
34
+ "reason": { "type": "string", "minLength": 1 }
35
+ },
36
+ "additionalProperties": false
37
+ },
38
+ "sourcePolicy": { "const": "PROVIDER_REPORTED_OR_HOST_REPORTED_ONLY" },
39
+ "claimsAllowed": { "type": "boolean" },
40
+ "generatedFromRunCount": { "type": "integer", "minimum": 1 }
41
+ },
42
+ "additionalProperties": false
43
+ }
@@ -0,0 +1,106 @@
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "$id": "forgeloop://schemas/execution-profile-benchmark-run.schema.json",
4
+ "title": "ForgeLoop execution-profile benchmark run",
5
+ "type": "object",
6
+ "required": ["schemaVersion", "benchmarkVersion", "runSetId", "runId", "scenarioId", "mode", "runIndex", "recordedAt", "usage", "wallClockMs", "verification", "verificationCycles", "comparableSteps", "metadata"],
7
+ "properties": {
8
+ "schemaVersion": { "const": 1 },
9
+ "benchmarkVersion": { "const": "1" },
10
+ "runSetId": { "type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$" },
11
+ "runId": { "type": "string", "pattern": "^run-[A-Za-z0-9][A-Za-z0-9_-]{0,127}$" },
12
+ "scenarioId": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{0,127}$" },
13
+ "mode": { "enum": ["direct", "forgeloopBalanced", "forgeloopAdaptive"] },
14
+ "runIndex": { "type": "integer", "minimum": 1 },
15
+ "recordedAt": { "type": "string", "minLength": 1 },
16
+ "usage": {
17
+ "type": "object",
18
+ "required": ["inputTokens", "outputTokens", "cacheReadTokens", "cacheWriteTokens", "totalTokens", "costUsd", "model", "provider", "source"],
19
+ "properties": {
20
+ "inputTokens": { "type": ["integer", "null"], "minimum": 0 },
21
+ "outputTokens": { "type": ["integer", "null"], "minimum": 0 },
22
+ "cacheReadTokens": { "type": ["integer", "null"], "minimum": 0 },
23
+ "cacheWriteTokens": { "type": ["integer", "null"], "minimum": 0 },
24
+ "totalTokens": { "type": ["integer", "null"], "minimum": 0 },
25
+ "costUsd": { "type": ["number", "null"], "minimum": 0 },
26
+ "model": { "type": ["string", "null"] },
27
+ "provider": { "type": ["string", "null"] },
28
+ "source": { "enum": ["PROVIDER_REPORTED", "HOST_REPORTED", "UNKNOWN"] }
29
+ },
30
+ "additionalProperties": false
31
+ },
32
+ "wallClockMs": { "type": ["number", "null"], "minimum": 0 },
33
+ "verification": { "enum": ["PASS", "FAIL", "NOT_AVAILABLE"] },
34
+ "verificationCycles": { "type": ["integer", "null"], "minimum": 0 },
35
+ "comparableSteps": { "type": ["integer", "null"], "minimum": 0 },
36
+ "contextUsage": { "$ref": "#/$defs/contextUsage" },
37
+ "quality": { "$ref": "#/$defs/quality" },
38
+ "metadata": {
39
+ "type": "object",
40
+ "required": ["scenarioId", "mode", "model", "provider", "promptSpecFingerprint", "projectRevision", "benchmarkVersion", "environmentClass", "requestedProfile", "resolvedProfile", "verificationCycles", "comparableSteps"],
41
+ "properties": {
42
+ "scenarioId": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{0,127}$" },
43
+ "mode": { "enum": ["direct", "forgeloopBalanced", "forgeloopAdaptive"] },
44
+ "model": { "type": ["string", "null"] },
45
+ "provider": { "type": ["string", "null"] },
46
+ "promptSpecFingerprint": { "type": "string", "minLength": 1 },
47
+ "projectRevision": { "type": ["string", "null"] },
48
+ "benchmarkVersion": { "const": "1" },
49
+ "environmentClass": { "type": "string", "minLength": 1 },
50
+ "requestedProfile": { "type": ["string", "null"] },
51
+ "resolvedProfile": { "type": ["string", "null"] },
52
+ "verificationCycles": { "type": ["integer", "null"], "minimum": 0 },
53
+ "comparableSteps": { "type": ["integer", "null"], "minimum": 0 },
54
+ "nodeVersion": { "type": "string", "minLength": 1 },
55
+ "os": { "type": "string", "minLength": 1 },
56
+ "arch": { "type": "string", "minLength": 1 }
57
+ },
58
+ "additionalProperties": false
59
+ }
60
+ },
61
+ "$defs": {
62
+ "contextUsage": {
63
+ "type": "object",
64
+ "required": ["source", "profile", "items"],
65
+ "properties": {
66
+ "source": { "enum": ["HOST_REPORTED", "UNKNOWN"] },
67
+ "profile": { "enum": ["light", "balanced", "full", null] },
68
+ "items": {
69
+ "type": "object",
70
+ "required": ["taskContext", "guides", "history", "protocolInstructions", "repositoryContext", "other"],
71
+ "properties": {
72
+ "taskContext": { "type": ["integer", "null"], "minimum": 0 },
73
+ "guides": { "type": ["integer", "null"], "minimum": 0 },
74
+ "history": { "type": ["integer", "null"], "minimum": 0 },
75
+ "protocolInstructions": { "type": ["integer", "null"], "minimum": 0 },
76
+ "repositoryContext": { "type": ["integer", "null"], "minimum": 0 },
77
+ "other": { "type": ["integer", "null"], "minimum": 0 }
78
+ },
79
+ "additionalProperties": false
80
+ }
81
+ },
82
+ "additionalProperties": false
83
+ },
84
+ "quality": {
85
+ "type": "object",
86
+ "required": ["source", "scores"],
87
+ "properties": {
88
+ "source": { "enum": ["EXTERNAL_REPORTED", "HOST_REPORTED", "UNKNOWN"] },
89
+ "scores": {
90
+ "type": "object",
91
+ "required": ["visualQuality", "responsiveQuality", "accessibility", "interactionPolish", "requirementsCompleteness"],
92
+ "properties": {
93
+ "visualQuality": { "type": ["number", "null"], "minimum": 0, "maximum": 5 },
94
+ "responsiveQuality": { "type": ["number", "null"], "minimum": 0, "maximum": 5 },
95
+ "accessibility": { "type": ["number", "null"], "minimum": 0, "maximum": 5 },
96
+ "interactionPolish": { "type": ["number", "null"], "minimum": 0, "maximum": 5 },
97
+ "requirementsCompleteness": { "type": ["number", "null"], "minimum": 0, "maximum": 5 }
98
+ },
99
+ "additionalProperties": false
100
+ }
101
+ },
102
+ "additionalProperties": false
103
+ }
104
+ },
105
+ "additionalProperties": false
106
+ }
@@ -0,0 +1,66 @@
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "$id": "forgeloop://schemas/execution-profile-benchmark-scenario.schema.json",
4
+ "title": "ForgeLoop execution-profile benchmark scenario",
5
+ "type": "object",
6
+ "required": ["schemaVersion", "benchmarkVersion", "scenarioId", "description", "input", "expectedProfile", "measurements"],
7
+ "properties": {
8
+ "schemaVersion": { "const": 1 },
9
+ "benchmarkVersion": { "type": "string", "minLength": 1 },
10
+ "scenarioId": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{0,127}$" },
11
+ "description": { "type": "string", "minLength": 1 },
12
+ "input": {
13
+ "type": "object",
14
+ "required": ["workType", "surfaces", "risks", "platforms"],
15
+ "properties": {
16
+ "workType": { "type": "string", "minLength": 1 },
17
+ "surfaces": { "type": "array", "items": { "type": "string", "minLength": 1 } },
18
+ "risks": { "type": "array", "items": { "type": "string", "minLength": 1 } },
19
+ "platforms": { "type": "array", "items": { "type": "string", "minLength": 1 } },
20
+ "behaviorChange": { "type": "boolean" },
21
+ "executableChange": { "type": "boolean" }
22
+ },
23
+ "additionalProperties": false
24
+ },
25
+ "expectedProfile": { "enum": ["light", "balanced", "full"] },
26
+ "referenceTask": { "$ref": "#/$defs/referenceTask" },
27
+ "measurements": {
28
+ "type": "object",
29
+ "required": ["direct", "forgeloopBalanced", "forgeloopAdaptive"],
30
+ "properties": {
31
+ "direct": { "$ref": "#/$defs/measurement" },
32
+ "forgeloopBalanced": { "$ref": "#/$defs/measurement" },
33
+ "forgeloopAdaptive": { "$ref": "#/$defs/measurement" }
34
+ },
35
+ "additionalProperties": false
36
+ }
37
+ },
38
+ "$defs": {
39
+ "referenceTask": {
40
+ "type": "object",
41
+ "required": ["name", "requirements", "exclusions", "artifactMode"],
42
+ "properties": {
43
+ "name": { "type": "string", "minLength": 1 },
44
+ "requirements": { "type": "array", "items": { "type": "string", "minLength": 1 }, "minItems": 1 },
45
+ "exclusions": { "type": "array", "items": { "type": "string", "minLength": 1 }, "minItems": 1 },
46
+ "artifactMode": { "type": "string", "minLength": 1 }
47
+ },
48
+ "additionalProperties": false
49
+ },
50
+ "measurement": {
51
+ "type": "object",
52
+ "required": ["inputTokens", "outputTokens", "cacheReadTokens", "cacheWriteTokens", "totalTokens", "wallClockMs", "verification"],
53
+ "properties": {
54
+ "inputTokens": { "type": ["integer", "null"], "minimum": 0 },
55
+ "outputTokens": { "type": ["integer", "null"], "minimum": 0 },
56
+ "cacheReadTokens": { "type": ["integer", "null"], "minimum": 0 },
57
+ "cacheWriteTokens": { "type": ["integer", "null"], "minimum": 0 },
58
+ "totalTokens": { "type": ["integer", "null"], "minimum": 0 },
59
+ "wallClockMs": { "type": ["number", "null"], "minimum": 0 },
60
+ "verification": { "enum": ["PASS", "FAIL", "NOT_AVAILABLE"] }
61
+ },
62
+ "additionalProperties": false
63
+ }
64
+ },
65
+ "additionalProperties": false
66
+ }
@@ -8,6 +8,18 @@
8
8
  "schemaVersion": { "const": 1 },
9
9
  "protocolVersion": { "const": 1 },
10
10
  "contractFingerprint": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
11
+ "executionProfile": {
12
+ "type": "object",
13
+ "required": ["requested", "floor", "resolved", "reasons", "escalated"],
14
+ "additionalProperties": false,
15
+ "properties": {
16
+ "requested": { "enum": ["auto", "light", "balanced", "full"] },
17
+ "floor": { "enum": ["light", "balanced", "full"] },
18
+ "resolved": { "enum": ["light", "balanced", "full"] },
19
+ "reasons": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } },
20
+ "escalated": { "type": "boolean" }
21
+ }
22
+ },
11
23
  "input": { "type": "object" },
12
24
  "primary": { "oneOf": [{ "type": "string" }, { "type": "null" }] },
13
25
  "guides": { "type": "array", "items": { "type": "string" } },
@@ -0,0 +1,30 @@
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "$id": "forgeloop://schemas/usage.schema.json",
4
+ "title": "ForgeLoop task usage telemetry",
5
+ "type": "object",
6
+ "required": ["schemaVersion", "protocolVersion", "taskId", "recordedAt", "usage"],
7
+ "properties": {
8
+ "schemaVersion": { "const": 1 },
9
+ "protocolVersion": { "const": 1 },
10
+ "taskId": { "type": "string", "minLength": 1 },
11
+ "recordedAt": { "type": "string", "minLength": 1 },
12
+ "usage": {
13
+ "type": "object",
14
+ "required": ["inputTokens", "outputTokens", "cacheReadTokens", "cacheWriteTokens", "totalTokens", "costUsd", "model", "provider", "source"],
15
+ "additionalProperties": false,
16
+ "properties": {
17
+ "inputTokens": { "type": ["integer", "null"], "minimum": 0 },
18
+ "outputTokens": { "type": ["integer", "null"], "minimum": 0 },
19
+ "cacheReadTokens": { "type": ["integer", "null"], "minimum": 0 },
20
+ "cacheWriteTokens": { "type": ["integer", "null"], "minimum": 0 },
21
+ "totalTokens": { "type": ["integer", "null"], "minimum": 0 },
22
+ "costUsd": { "type": ["number", "null"], "minimum": 0 },
23
+ "model": { "type": ["string", "null"] },
24
+ "provider": { "type": ["string", "null"] },
25
+ "source": { "enum": ["PROVIDER_REPORTED", "HOST_REPORTED", "ACTOR_REPORTED", "UNKNOWN"] }
26
+ }
27
+ }
28
+ },
29
+ "additionalProperties": false
30
+ }
@@ -0,0 +1,99 @@
1
+ #!/usr/bin/env node
2
+
3
+ import path from "node:path";
4
+ import { fileURLToPath } from "node:url";
5
+
6
+ import { aggregateBenchmarkRuns } from "../src/core/execution-profile-benchmarks.js";
7
+ import { readBenchmarkRunSets, readBenchmarkScenarios } from "./lib/execution-profile-benchmark-io.mjs";
8
+
9
+ const repositoryRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
10
+ const defaultResultsDirectory = path.join(repositoryRoot, "benchmarks", "execution-profiles", "results");
11
+
12
+ function parseArgs(argv) {
13
+ const options = { results: defaultResultsDirectory, json: false };
14
+ for (let index = 0; index < argv.length; index += 1) {
15
+ const argument = argv[index];
16
+ if (argument === "--json") options.json = true;
17
+ else if (argument === "--results") {
18
+ const value = argv[++index];
19
+ if (!value || value.startsWith("--")) throw new Error("--results requires a value");
20
+ options.results = value;
21
+ } else if (argument === "--help" || argument === "-h") options.help = true;
22
+ else throw new Error(`unknown option: ${argument}`);
23
+ }
24
+ return options;
25
+ }
26
+
27
+ function statusForAggregate(aggregate, runs) {
28
+ const hasMeasuredUsage = runs.some((run) => (
29
+ ["PROVIDER_REPORTED", "HOST_REPORTED"].includes(run.usage.source)
30
+ && run.usage.totalTokens !== null
31
+ ));
32
+ if (!hasMeasuredUsage) return "NOT_MEASURED";
33
+ if (!aggregate.claimsAllowed) return "NOT_COMPARABLE";
34
+ if (aggregate.lightObjectives
35
+ && (aggregate.lightObjectives.p50Pass === false || aggregate.lightObjectives.p95Pass === false)) {
36
+ return "EFFICIENCY_REGRESSION";
37
+ }
38
+ return "OK";
39
+ }
40
+
41
+ function overallStatus(statuses) {
42
+ if (statuses.includes("EFFICIENCY_REGRESSION")) return "EFFICIENCY_REGRESSION";
43
+ if (statuses.includes("NOT_COMPARABLE")) return "NOT_COMPARABLE";
44
+ if (statuses.length === 0 || statuses.every((status) => status === "NOT_MEASURED")) return "NOT_MEASURED";
45
+ return "OK";
46
+ }
47
+
48
+ function helpText() {
49
+ return "Usage: npm run benchmark:profiles:regression -- [--results <directory>] [--json]";
50
+ }
51
+
52
+ async function main() {
53
+ const options = parseArgs(process.argv.slice(2));
54
+ if (options.help) {
55
+ console.log(helpText());
56
+ return;
57
+ }
58
+ const scenarios = await readBenchmarkScenarios(repositoryRoot);
59
+ const runSets = await readBenchmarkRunSets(path.resolve(options.results));
60
+ const reports = [];
61
+ for (const runSet of runSets) {
62
+ for (const scenario of scenarios) {
63
+ const runs = runSet.runs.filter((run) => run.scenarioId === scenario.scenarioId);
64
+ if (runs.length === 0) {
65
+ reports.push({ runSetId: runSet.runSetId, scenarioId: scenario.scenarioId, status: "NOT_COMPARABLE" });
66
+ continue;
67
+ }
68
+ const aggregate = aggregateBenchmarkRuns({ scenario, runs });
69
+ reports.push({
70
+ runSetId: runSet.runSetId,
71
+ scenarioId: scenario.scenarioId,
72
+ expectedProfile: scenario.expectedProfile,
73
+ status: statusForAggregate(aggregate, runs),
74
+ claimsAllowed: aggregate.claimsAllowed,
75
+ contextInflation: aggregate.contextInflation?.status ?? null,
76
+ });
77
+ }
78
+ }
79
+ const status = overallStatus(reports.map((report) => report.status));
80
+ const output = {
81
+ status,
82
+ blocking: false,
83
+ runSetCount: runSets.length,
84
+ scenarioCount: scenarios.length,
85
+ reports,
86
+ policy: "EFFICIENCY_REGRESSION is an observational release-quality warning and never a ForgeLoop lifecycle failure.",
87
+ };
88
+ if (options.json) console.log(JSON.stringify(output));
89
+ else {
90
+ console.log(`Efficiency regression status: ${status}`);
91
+ console.log(`Run sets: ${runSets.length}`);
92
+ console.log("This report is observational and does not change lifecycle completion.");
93
+ }
94
+ }
95
+
96
+ main().catch((error) => {
97
+ console.error(`ForgeLoop efficiency regression check: ${error.message}`);
98
+ process.exitCode = 1;
99
+ });
@@ -79,6 +79,41 @@ Package version: ${packageJson.version}
79
79
  6. Run forgeloop complete; accept completion only when the validator returns VALID.
80
80
  7. Run forgeloop next again and follow the returned lifecycle action to a terminal state or an explicit blocker.
81
81
 
82
+ ## Adaptive execution profiles
83
+
84
+ \`complianceMode\` controls how strongly project policy is enforced. The
85
+ orthogonal \`executionProfile\` controls process and context depth: \`auto\`
86
+ resolves deterministically to \`light\`, \`balanced\`, or \`full\` from route,
87
+ contract, and task metadata. CLI requests take precedence over project
88
+ configuration, but a safety floor always wins. Profiles never remove required
89
+ contracts, gates, verification, provenance, lifecycle phases, or validated
90
+ completion. Protocol-v1 routes without the field project to \`balanced\` for
91
+ compatibility without rewriting historical artifacts.
92
+
93
+ For \`light\` tasks, hosts should use \`next --compact\` or \`task-show --compact\`,
94
+ load only relevant guide sections, keep plans concise, and avoid optional
95
+ reflection, trajectory evaluation, handoff, attestation, and continuity
96
+ artifacts unless requested, required, or needed for recovery. The lifecycle
97
+ chronology remains unchanged.
98
+
99
+ The read-only \`task/context\` integration resource provides the canonical
100
+ profile-aware projection: objective, deliverables, constraints, selected guide
101
+ IDs, phase, next action, verification requirements, and context policy. A
102
+ profile changes presentation and optional context only; it never permits a
103
+ phase or required gate to be skipped.
104
+
105
+ Usage telemetry is provider or host reported when available, actor-reported
106
+ only through the explicit \`usage-record\` fallback, and \`UNKNOWN\` otherwise.
107
+ ForgeLoop never estimates tokens or treats usage as verification evidence.
108
+ \`efficiency --task\` is read-only and returns \`NOT_COMPARABLE\` unless a
109
+ project-local baseline has matching metadata.
110
+
111
+ Measured execution-profile benchmarks are observational. The reproducible
112
+ runner accepts provider or host usage, records actual timing, requires PASS
113
+ verification and matching metadata for comparisons, and reports \`NOT_MEASURED\`
114
+ or \`NOT_COMPARABLE\` when evidence is absent or incompatible. See
115
+ \`docs/EXECUTION_PROFILE_BENCHMARKS.md\` for the runner and schemas.
116
+
82
117
  ## Authority boundaries
83
118
 
84
119
  - Protocol-derived facts outrank actor-provided labels, free-form summaries, and guessed identities.