workspai 0.46.0 → 0.48.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +225 -81
- package/contracts/agent-customization-pack.v1.json +11 -1
- package/contracts/bootstrap-compliance.v1.json +14 -0
- package/contracts/cli-runtime-command-inventory.v1.snapshot.json +4 -0
- package/contracts/extension-cli-compatibility.v1.json +13 -2
- package/contracts/mirror-ops.v1.json +16 -0
- package/contracts/published-contract-catalog.v1.json +59 -1
- package/contracts/runtime-command-surface.v1.json +294 -2
- package/contracts/transparency-evidence.v1.json +13 -0
- package/contracts/workspace-contract.v1.json +78 -0
- package/contracts/workspace-intelligence/model-usage-event.v1.json +82 -0
- package/contracts/workspace-intelligence/workspace-context.v1.json +20 -0
- package/contracts/workspace-intelligence/workspace-graph-token-efficiency.v1.json +72 -0
- package/contracts/workspace-intelligence/workspace-intelligence-evaluation-comparison.v1.json +53 -0
- package/contracts/workspace-intelligence/workspace-intelligence-evaluation.v1.json +120 -0
- package/contracts/workspace-intelligence/workspace-intelligence-run.v1.json +6 -1
- package/contracts/workspace-intelligence/workspace-knowledge-graph-change-overlay.v1.json +200 -0
- package/contracts/workspace-intelligence/workspace-knowledge-graph.v1.json +260 -0
- package/contracts/workspace-intelligence/workspace-knowledge-search.v1.json +60 -0
- package/contracts/workspace-intelligence-architecture.v1.json +37 -5
- package/contracts/workspace-intelligence-chain.v1.json +58 -6
- package/contracts/workspace-share-bundle.v1.json +16 -0
- package/dist/analyze-EEEU3MIF.js +1 -0
- package/dist/artifact-remediation-plan-SPOUHMK5.js +3 -0
- package/dist/autopilot-release-XGVXPOZI.js +1 -0
- package/dist/chunk-32OJDBIG.js +2 -0
- package/dist/chunk-37CVKXBD.js +1 -0
- package/dist/chunk-3NU32T4A.js +2 -0
- package/dist/chunk-3VFA7D5T.js +1 -0
- package/dist/chunk-4HDYADHT.js +13 -0
- package/dist/chunk-54EP5CEV.js +8 -0
- package/dist/chunk-5S3DJQEP.js +8 -0
- package/dist/chunk-AFL3ACCR.js +2 -0
- package/dist/{chunk-VFDM65IE.js → chunk-BFLJ2R4D.js} +22 -22
- package/dist/chunk-BGPXQQNY.js +4 -0
- package/dist/chunk-BMWFQXGW.js +1 -0
- package/dist/{chunk-UQWOVV6V.js → chunk-CRHYBQI3.js} +1 -1
- package/dist/chunk-CV5HKU4P.js +1 -0
- package/dist/chunk-ESLPI3XZ.js +1 -0
- package/dist/chunk-FB7SCXAZ.js +1 -0
- package/dist/{chunk-WPEEC5BX.js → chunk-HDXNIN4N.js} +1 -1
- package/dist/chunk-HZDXO65G.js +36 -0
- package/dist/{chunk-J5PIZCAU.js → chunk-I46XEIPL.js} +1 -1
- package/dist/chunk-J5ENLXDF.js +1 -0
- package/dist/chunk-KB44JP4M.js +2 -0
- package/dist/chunk-LHOZXC2M.js +2 -0
- package/dist/chunk-NAJCUQ4X.js +2 -0
- package/dist/chunk-OA537ZQ5.js +1 -0
- package/dist/chunk-OW42TZFB.js +1 -0
- package/dist/chunk-P3D5YQB2.js +1 -0
- package/dist/chunk-PHXQR6PX.js +2 -0
- package/dist/{chunk-ZKAI3PJE.js → chunk-PRTR2DQ2.js} +1 -1
- package/dist/{chunk-QA5BGEQW.js → chunk-QNONOO4F.js} +1 -1
- package/dist/chunk-RHQW3DTP.js +1 -0
- package/dist/chunk-T4YR4RAI.js +2 -0
- package/dist/chunk-THIOE2PB.js +2 -0
- package/dist/{chunk-4LGXSBCN.js → chunk-VU7NZHPM.js} +1 -1
- package/dist/chunk-WDKNMTJQ.js +1 -0
- package/dist/{chunk-YUATNVOT.js → chunk-Y5UJLPS4.js} +16 -16
- package/dist/{chunk-6IIZJQLV.js → chunk-YJZOMRAS.js} +1 -1
- package/dist/{create-WCV3L6XH.js → create-DBQNAMKP.js} +1 -1
- package/dist/{doctor-5BWM2EMJ.js → doctor-4NNUDNGZ.js} +1 -1
- package/dist/{dotnet-webapi-clean-6TVFBTVI.js → dotnet-webapi-clean-A6MVDYXX.js} +4 -4
- package/dist/{gofiber-standard-2BL7GWZB.js → gofiber-standard-I5YPQG5V.js} +1 -1
- package/dist/{gogin-standard-XGP3KBXA.js → gogin-standard-VY2L4QT5.js} +1 -1
- package/dist/index.d.ts +56 -14
- package/dist/index.js +322 -320
- package/dist/pipeline-TQM43A3K.js +5 -0
- package/dist/{springboot-standard-JJNUID6M.js → springboot-standard-55XKCBIZ.js} +4 -4
- package/dist/{workspace-7OXW5YTJ.js → workspace-NCWRINEF.js} +1 -1
- package/dist/{workspace-agent-sync-O4IA6VOA.js → workspace-agent-sync-662QHXGF.js} +1 -1
- package/dist/{workspace-archive-H74NBBNW.js → workspace-archive-P76EDIUG.js} +1 -1
- package/dist/{workspace-context-R7IPUBPG.js → workspace-context-23YYCUCP.js} +1 -1
- package/dist/workspace-contract-TU2I7GC2.js +1 -0
- package/dist/workspace-dependency-graph-BP4EXYQ5.js +1 -0
- package/dist/workspace-explain-MWUEN643.js +1 -0
- package/dist/workspace-explain-contract-ZPI3JXJU.js +1 -0
- package/dist/{workspace-feedback-REOS36ZZ.js → workspace-feedback-SUVH2LUJ.js} +1 -1
- package/dist/{workspace-foundation-KXT4QI5O.js → workspace-foundation-WXJ6I7ES.js} +1 -1
- package/dist/workspace-graph-token-efficiency-CFGFCJ5V.js +1 -0
- package/dist/{workspace-history-OGOVSKZG.js → workspace-history-BANOJRQ2.js} +1 -1
- package/dist/workspace-intelligence-MFJE7W67.js +1 -0
- package/dist/workspace-intelligence-evaluation-IPH7M3WV.js +1 -0
- package/dist/workspace-intelligence-runner-THYLHHMF.js +1 -0
- package/dist/workspace-knowledge-graph-ARDC6HHG.js +1 -0
- package/dist/workspace-knowledge-graph-change-overlay-XG6FC4IX.js +1 -0
- package/dist/workspace-knowledge-graph-export-UYAYFTWX.js +10 -0
- package/dist/workspace-knowledge-graph-query-VOSPPH4W.js +1 -0
- package/dist/workspace-mcp-serve-EZR6O76D.js +3 -0
- package/dist/workspace-model-7OU2M3LE.js +1 -0
- package/dist/workspace-model-hash-MHXK5MEI.js +1 -0
- package/dist/{workspace-registry-summary-SZ46R5PD.js → workspace-registry-summary-ORDK7A36.js} +1 -1
- package/dist/workspace-run-RLIYSOTN.js +1 -0
- package/dist/{workspace-verify-MFQ7IXGD.js → workspace-verify-EBVL7FWT.js} +1 -1
- package/dist/workspace-watch-3BPGLFLB.js +1 -0
- package/docs/AI_DYNAMIC_INTEGRATION.md +73 -428
- package/docs/AI_EXAMPLES.md +37 -395
- package/docs/AI_FEATURES.md +76 -456
- package/docs/AI_QUICKSTART.md +49 -212
- package/docs/GLOSSARY.md +64 -0
- package/docs/OPEN_SOURCE_USER_SCENARIOS.md +68 -7
- package/docs/README.md +70 -41
- package/docs/README_CONTENT_CONTRACT.md +154 -0
- package/docs/commands-reference.md +64 -4
- package/docs/config-file-guide.md +6 -2
- package/docs/contracts/ARTIFACT_CATALOG.md +76 -37
- package/docs/contracts/README.md +47 -8
- package/docs/graph-benchmark-methodology.md +121 -0
- package/docs/workspace-intelligence-evaluation.md +147 -0
- package/docs/workspace-knowledge-graph.md +316 -0
- package/docs/workspace-operations.md +49 -0
- package/package.json +2 -1
- package/dist/analyze-BEBEZSZK.js +0 -1
- package/dist/artifact-remediation-plan-FFQSESAM.js +0 -3
- package/dist/autopilot-release-WUR4CQIT.js +0 -1
- package/dist/chunk-2G7FASAO.js +0 -2
- package/dist/chunk-4EPHWD27.js +0 -8
- package/dist/chunk-CVHMUSRX.js +0 -1
- package/dist/chunk-DIPD72H4.js +0 -2
- package/dist/chunk-EFYHGCGX.js +0 -2
- package/dist/chunk-FWRXA435.js +0 -2
- package/dist/chunk-HDURFXW5.js +0 -2
- package/dist/chunk-HMUKBW2S.js +0 -4
- package/dist/chunk-K4WNYXKK.js +0 -33
- package/dist/chunk-LG6RFLPZ.js +0 -1
- package/dist/chunk-N7DV5L7C.js +0 -1
- package/dist/chunk-PRBVYW3T.js +0 -1
- package/dist/chunk-QZLIURER.js +0 -13
- package/dist/chunk-RIEF2DDX.js +0 -8
- package/dist/chunk-SXMTSV5M.js +0 -1
- package/dist/chunk-SXPY523X.js +0 -1
- package/dist/chunk-V3LRQZ36.js +0 -1
- package/dist/chunk-WYFPXTTS.js +0 -2
- package/dist/pipeline-ORIWVVYM.js +0 -5
- package/dist/workspace-contract-HKCMOMFE.js +0 -1
- package/dist/workspace-explain-GOPQYTPQ.js +0 -1
- package/dist/workspace-explain-contract-SVFJAAEI.js +0 -1
- package/dist/workspace-intelligence-7IESQSXY.js +0 -1
- package/dist/workspace-intelligence-runner-6GJ5M4HB.js +0 -1
- package/dist/workspace-mcp-serve-FRVWBO36.js +0 -3
- package/dist/workspace-model-PPYX7B4S.js +0 -1
- package/dist/workspace-run-V3KKHTVF.js +0 -1
- package/dist/workspace-watch-SOPZHRWA.js +0 -1
|
@@ -72,22 +72,31 @@ the same evidence without losing the workspace source of truth.
|
|
|
72
72
|
|
|
73
73
|
## Workspace intelligence
|
|
74
74
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
|
80
|
-
|
|
|
81
|
-
| `
|
|
82
|
-
| `workspace
|
|
83
|
-
| `workspace
|
|
84
|
-
| `workspace
|
|
85
|
-
| `workspace
|
|
86
|
-
| `
|
|
87
|
-
| `workspace
|
|
88
|
-
| `workspace
|
|
89
|
-
| `workspace
|
|
90
|
-
| `workspace
|
|
75
|
+
Bare artifact names in this table are relative to `.workspai/reports/`.
|
|
76
|
+
Entries beginning with `reports/` are relative to `.workspai/`; paths such as
|
|
77
|
+
`AGENTS.md` are relative to the workspace root.
|
|
78
|
+
|
|
79
|
+
| Command | Artifact | Schema | Contract file |
|
|
80
|
+
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ---------------------------------------------------------------------------- |
|
|
81
|
+
| `workspace model --write` | `workspace-model.json` | `workspace-model.v1` | `contracts/workspace-intelligence/workspace-model.v1.json` |
|
|
82
|
+
| `workspace model --write` | `workspace-knowledge-graph.json` | `workspace-knowledge-graph.v1` | `contracts/workspace-intelligence/workspace-knowledge-graph.v1.json` |
|
|
83
|
+
| `workspace snapshot` | `workspace-model-snapshot.json` | `workspace-model-snapshot.v1` | `contracts/workspace-intelligence/workspace-model-snapshot.v1.json` |
|
|
84
|
+
| `workspace diff` | `workspace-model-diff-last-run.json` | `workspace-model-diff.v1` | `contracts/workspace-intelligence/workspace-model-diff.v1.json` |
|
|
85
|
+
| `workspace impact --from <diff>` | `workspace-impact-last-run.json` | `workspace-impact.v1` | `contracts/workspace-intelligence/workspace-impact.v1.json` |
|
|
86
|
+
| `analyze --json` | `analyze-last-run.json` | `rapidkit-analyze-v1` | `contracts/analyze-last-run.v1.json` |
|
|
87
|
+
| `workspace verify` | `workspace-verify-last-run.json` | `workspace-verify.v1` | `contracts/workspace-intelligence/workspace-verify.v1.json` |
|
|
88
|
+
| `workspace context --write` | `workspace-context-agent.json` | `workspace-context.v1` | `contracts/workspace-intelligence/workspace-context.v1.json` |
|
|
89
|
+
| `workspace agent-sync --write` | `reports/agent-customization-pack.json` | `rapidkit-agent-customization-pack.v1` | `contracts/workspace-intelligence/agent-customization-pack-report.v1.json` |
|
|
90
|
+
| `workspace agent-sync --write` | `reports/INDEX.json` | `rapidkit-agent-reports-index.v1` | `contracts/workspace-intelligence/agent-reports-index.v1.json` |
|
|
91
|
+
| `workspace agent-sync --write` | `reports/workspace-skills-index.json` | `workspace-skills-index.v1` | `contracts/workspace-intelligence/workspace-skills-index.v1.json` |
|
|
92
|
+
| `workspace agent-sync --write` | `reports/workspai-mcp-design.json`, `.workspai/skills/*.md`, `.workspai/AGENT-GROUNDING.md`, `AGENTS.md`, IDE agent surfaces | Mixed generated surfaces | See customization pack output inventory |
|
|
93
|
+
| `workspace explain --write` | `workspace-explain-last-run.json` | `workspace-explain.v1` | `contracts/workspace-intelligence/workspace-explain.v1.json` |
|
|
94
|
+
| `workspace why --write` | `workspace-why-last-run.json` | `workspace-explain.v1` | `contracts/workspace-intelligence/workspace-explain.v1.json` |
|
|
95
|
+
| `workspace trace --write` | `workspace-trace-last-run.json` | `workspace-explain.v1` | `contracts/workspace-intelligence/workspace-explain.v1.json` |
|
|
96
|
+
| `workspace intelligence run` | `workspace-intelligence-run-last-run.json` | `workspace-intelligence-run.v1` | `contracts/workspace-intelligence/workspace-intelligence-run.v1.json` |
|
|
97
|
+
| `workspace feedback record` / `doctor * --fix` | `workspace-intelligence-history.json` (`kind: agent-action`, `doctor-fix`) | `workspace-intelligence-history.v1` | `contracts/workspace-intelligence/workspace-intelligence-history.v1.json` |
|
|
98
|
+
| `workspace eval init` / `workspace eval record` | `workspace-intelligence-evaluation-live.json` | `workspace-intelligence-evaluation.v1` | `contracts/workspace-intelligence/workspace-intelligence-evaluation.v1.json` |
|
|
99
|
+
| `workspace eval report` | `workspace-intelligence-evaluation-last-run.json` | `workspace-intelligence-evaluation.v1` | `contracts/workspace-intelligence/workspace-intelligence-evaluation.v1.json` |
|
|
91
100
|
|
|
92
101
|
The unified runner report separates its execution envelope from the canonical
|
|
93
102
|
intelligence chain. `preflight` always contains exactly `sync` and `baseline`;
|
|
@@ -99,6 +108,18 @@ status/exit coherence, hard-failure skip propagation, and the aggregate verdict.
|
|
|
99
108
|
See [Unified Workspace Intelligence Runner](../workspace-intelligence-runner.md)
|
|
100
109
|
for the normative user and integration semantics.
|
|
101
110
|
|
|
111
|
+
`workspace-model.json` and `workspace-knowledge-graph.json` are published as one
|
|
112
|
+
recoverable artifact transaction. The graph carries a SHA-256 `source` binding
|
|
113
|
+
to the canonical model, so consumers must reject a graph whose source hash does
|
|
114
|
+
not equal the current structural model hash.
|
|
115
|
+
|
|
116
|
+
`workspace graph jsonld|graphml|gexf --output <path>` creates explicit interchange
|
|
117
|
+
projections from that bound graph. These files are portable exports, not competing
|
|
118
|
+
canonical or `last-run` artifacts: JSON-LD preserves semantic identifiers,
|
|
119
|
+
GraphML targets general graph tooling, and GEXF targets exploration and
|
|
120
|
+
visualization tools. Mermaid and DOT remain bounded text projections for docs
|
|
121
|
+
and diagrams.
|
|
122
|
+
|
|
102
123
|
**CLI semantics:** `workspace diff --from` expects a **model or snapshot** baseline. `workspace impact --from` expects a **diff report**.
|
|
103
124
|
Persisted artifacts retain their artifact schema. JSON command projections that add operation metadata
|
|
104
125
|
such as `outputPath`, `status`, or structured errors use
|
|
@@ -188,11 +209,18 @@ block, in `warn` mode they escalate to needs-attention.
|
|
|
188
209
|
a dependency change makes every dependent stale deterministically. The verdict compares against
|
|
189
210
|
the previously written verify report. Canonical source: `src/workspace-graph-freshness.ts`.
|
|
190
211
|
|
|
191
|
-
**Graph command surface.** `workspace graph` emits the graph plus
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
212
|
+
**Graph command surface.** `workspace graph` emits the dependency graph plus
|
|
213
|
+
integrity and hotspots. `explain <project>` returns centrality and
|
|
214
|
+
direct/transitive relationships. `search`, `entities`, `evidence`, and `path`
|
|
215
|
+
return bounded Knowledge Graph projections with proof references; `benchmark`
|
|
216
|
+
measures corpus-versus-retrieval payload; `overlay --from` compares a proposed
|
|
217
|
+
or earlier graph with the current graph; `emit` returns the complete
|
|
218
|
+
interchange graph; and `dot|mermaid` render deterministic dependency views.
|
|
219
|
+
Canonical sources are `src/workspace-graph.ts`,
|
|
220
|
+
`src/workspace-knowledge-graph-query.ts`,
|
|
221
|
+
`src/workspace-knowledge-graph-change-overlay.ts`, and
|
|
222
|
+
`src/workspace-graph-token-efficiency.ts`. These command surfaces are published
|
|
223
|
+
through `runtime-command-surface.v1` for IDE/CI capability detection.
|
|
196
224
|
|
|
197
225
|
### Model cache (`workspace-model-cache.v1`)
|
|
198
226
|
|
|
@@ -262,19 +290,19 @@ Canonical source: `src/observability/run-correlation.ts` (`attachRunCorrelation`
|
|
|
262
290
|
|
|
263
291
|
## Operational / platform
|
|
264
292
|
|
|
265
|
-
| Command | Artifact
|
|
266
|
-
| -------------------------------- |
|
|
267
|
-
| `workspace run` | `workspace-run-last.json`
|
|
268
|
-
| `autopilot release` (run stages) | same `workspace-run-last.json`
|
|
269
|
-
| `bootstrap` | `bootstrap-compliance-{ts}.json`, `bootstrap-compliance.latest.json`
|
|
270
|
-
| `mirror status` | `mirror-ops-{ts}.json`, `mirror-ops.latest.json`
|
|
271
|
-
| `mirror` (transparency) | `transparency-evidence-{ts}.json`, `transparency-evidence.latest.json` |
|
|
272
|
-
| `infra plan` | `infra-plan.json`
|
|
273
|
-
| `workspace archive` | `.workspai/archive-manifest.json` inside ZIP/ZIP64
|
|
274
|
-
| `workspace share` | `reports/share-bundle.json` (default)
|
|
275
|
-
| `import` | `{project}/.workspai/import.json`, `{project}/.workspai/import-readiness.json` | Copied/cloned project metadata and readiness
|
|
276
|
-
| `adopt` | `{project}/.workspai/adopt.json`, `{project}/.workspai/adopt-readiness.json` | In-place project metadata and readiness
|
|
277
|
-
| `workspace contract verify` | `workspace-contract-verify-last-run.json`
|
|
293
|
+
| Command | Artifact | Notes | Contract |
|
|
294
|
+
| -------------------------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
|
295
|
+
| `workspace run` | `workspace-run-last.json` | `workspace-run-v1` (multi-stage: `stages.test`, `stages.build`, …) | `contracts/workspace-run-last.v1.json` |
|
|
296
|
+
| `autopilot release` (run stages) | same `workspace-run-last.json` | Autopilot publishes test/build into aggregate (no separate `autopilot-workspace-run-*.json`) | — |
|
|
297
|
+
| `bootstrap` | `bootstrap-compliance-{ts}.json`, `bootstrap-compliance.latest.json` | `bootstrap-compliance.v1` | `contracts/bootstrap-compliance.v1.json` |
|
|
298
|
+
| `mirror status` | `mirror-ops-{ts}.json`, `mirror-ops.latest.json` | `mirror-ops.v1` | `contracts/mirror-ops.v1.json` |
|
|
299
|
+
| `mirror` (transparency) | `transparency-evidence-{ts}.json`, `transparency-evidence.latest.json` | `transparency-evidence.v1` | `contracts/transparency-evidence.v1.json` |
|
|
300
|
+
| `infra plan` | `infra-plan.json` | `rapidkit.infra-plan.v1` | — |
|
|
301
|
+
| `workspace archive` | `.workspai/archive-manifest.json` inside ZIP/ZIP64 | Streaming handoff; workspace payload is unlimited by default and safety budgets are opt-in | `contracts/workspace-archive-manifest.v1.json` |
|
|
302
|
+
| `workspace share` | `reports/share-bundle.json` (default) | Aggregation bundle (`1.1`) | `contracts/workspace-share-bundle.v1.json` |
|
|
303
|
+
| `import` | `{project}/.workspai/import.json`, `{project}/.workspai/import-readiness.json` | Copied/cloned project metadata and readiness | — |
|
|
304
|
+
| `adopt` | `{project}/.workspai/adopt.json`, `{project}/.workspai/adopt-readiness.json` | In-place project metadata and readiness | — |
|
|
305
|
+
| `workspace contract verify` | `workspace-contract-verify-last-run.json` | CLI verify cache | `contracts/workspace-intelligence/workspace-contract-verify.v1.json` |
|
|
278
306
|
|
|
279
307
|
## Static capability contracts
|
|
280
308
|
|
|
@@ -320,18 +348,29 @@ them as portable repository contracts. The portable source is
|
|
|
320
348
|
|
|
321
349
|
Under `{project}/.workspai/reports/` when commands run at project scope (e.g. project doctor). Workspace-level reports stay under `{workspace}/.workspai/reports/`.
|
|
322
350
|
|
|
351
|
+
After a Python Core bridge creates a project, Workspai validates and mirrors
|
|
352
|
+
legacy `.rapidkit/project.json`, `context.json`, and `file-hashes.json` into the
|
|
353
|
+
canonical project `.workspai/` directory without overwriting an existing
|
|
354
|
+
canonical file. Legacy files remain readable during the compatibility window.
|
|
355
|
+
|
|
323
356
|
## Consumer rules
|
|
324
357
|
|
|
325
358
|
1. **Project count:** read `workspace-registry.v1.json` (or run `workspace registry status --json`).
|
|
326
359
|
2. **Workspace Intelligence chain:** run `workspace intelligence run --for-agent codex --strict --json` to preserve Model → Diff → Impact → Doctor + Contract Verify + Analyze → Readiness → Verify → Context → Agent Sync → Explain. `pipeline` is the broader governance/release orchestrator and `autopilot` is a separate release surface; neither redefines the canonical chain. Use `pipeline-last-run.json` only for the pipeline orchestration summary.
|
|
327
360
|
3. **Do not** use `workspace.json.projects` (removed in schema 1.0).
|
|
328
361
|
4. Prefer `schemaVersion` constants in each artifact; legacy `v1` on readiness is accepted when reading old reports.
|
|
329
|
-
5. **Agent
|
|
362
|
+
5. **Agent retrieval:** start with `AGENTS.md` and `.workspai/reports/INDEX.json`, then use `workspace graph search <query> --limit <n> --json` or MCP `searchWorkspaceGraph` for question-sized facts. Follow returned proof paths to source evidence. Read the full context, model, or graph only when the bounded result is insufficient.
|
|
363
|
+
6. **Agent customization state:** use `.workspai/reports/agent-customization-pack.json` to inspect generated surfaces and drift; regenerate with `workspace agent-sync --write --refresh-context --preset enterprise`.
|
|
330
364
|
|
|
331
365
|
## Agent customization files (repo hooks)
|
|
332
366
|
|
|
333
367
|
Written by `workspace agent-sync --write --refresh-context --preset enterprise` (and by default after `workspace context --for-agent --write`):
|
|
334
368
|
|
|
369
|
+
The generated output inventory is committed as one journaled transaction. On
|
|
370
|
+
failure, all touched files are restored; an interrupted transaction is recovered
|
|
371
|
+
before the next agent-sync. `agent-customization-pack.json` is written last and
|
|
372
|
+
serves as the completed-generation marker.
|
|
373
|
+
|
|
335
374
|
| Path | Consumer |
|
|
336
375
|
| ----------------------------------------------------------------------- | -------------------------------------------------------------- |
|
|
337
376
|
| `AGENTS.md` | Copilot, Cursor, Claude Code, Codex, Grok (open standard) |
|
|
@@ -345,7 +384,7 @@ Written by `workspace agent-sync --write --refresh-context --preset enterprise`
|
|
|
345
384
|
| `.github/prompts/workspai-adopt-project.prompt.md` | Copilot adopt/import workflow prompt |
|
|
346
385
|
| `.github/skills/workspai-grounding/SKILL.md` | Copilot skills |
|
|
347
386
|
| `.github/skills/workspai-workspace-intelligence/SKILL.md` | Enterprise Workspace Intelligence skill |
|
|
348
|
-
| `.github/skills/workspai-workspace-intelligence/resources/mcp-tools.md` |
|
|
387
|
+
| `.github/skills/workspai-workspace-intelligence/resources/mcp-tools.md` | MCP tool and evidence-retrieval reference |
|
|
349
388
|
| `.github/agents/workspai-advisor.agent.md` | Read-only workspace advisor agent |
|
|
350
389
|
| `.github/agents/workspai-repair.agent.md` | Blocker repair agent |
|
|
351
390
|
| `.github/agents/workspai-release.agent.md` | Release safety agent |
|
|
@@ -353,7 +392,7 @@ Written by `workspace agent-sync --write --refresh-context --preset enterprise`
|
|
|
353
392
|
| `.cursor/rules/workspai-grounding.mdc` | Cursor always-on rule |
|
|
354
393
|
| `CLAUDE.md` | Claude Code (imports `@AGENTS.md`) |
|
|
355
394
|
| `.claude/rules/workspai-evidence.md` | Claude Code scoped evidence rule |
|
|
356
|
-
| `.claude/rules/rapidkit-evidence.md` | Legacy
|
|
395
|
+
| `.claude/rules/rapidkit-evidence.md` | Legacy compatibility alias pointing to the canonical rule |
|
|
357
396
|
| `.workspai/AGENT-GROUNDING.md` | Tool-agnostic operator doc |
|
|
358
397
|
| `.workspai/reports/agent-customization-pack.json` | Versioned output inventory, target matrix, drift state |
|
|
359
398
|
| `.workspai/reports/workspai-mcp-design.json` | Read-mostly MCP-ready design manifest |
|
package/docs/contracts/README.md
CHANGED
|
@@ -2,6 +2,24 @@
|
|
|
2
2
|
|
|
3
3
|
Contract documentation for JSON payloads, support matrices, and cross-repo parity.
|
|
4
4
|
|
|
5
|
+
## Complete contract discovery
|
|
6
|
+
|
|
7
|
+
The complete machine-readable inventory is
|
|
8
|
+
[`../../contracts/published-contract-catalog.v1.json`](../../contracts/published-contract-catalog.v1.json).
|
|
9
|
+
It is the source of truth for every published schema/capability path; the lists
|
|
10
|
+
below are grouped entry points, not a substitute for that catalog.
|
|
11
|
+
|
|
12
|
+
Installed consumers can discover the active package version and contract map
|
|
13
|
+
without scraping Markdown:
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
npx workspai --version --json
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Resolve contract files from the installed `workspai/contracts/` directory and
|
|
20
|
+
validate payloads against the exact catalog revision shipped with that CLI.
|
|
21
|
+
Do not copy a schema from `main` and assume it matches an older installed CLI.
|
|
22
|
+
|
|
5
23
|
## Monorepo workflow
|
|
6
24
|
|
|
7
25
|
Canonical JSON lives in **`../../contracts/`** (CLI package root, published in the tarball).
|
|
@@ -12,7 +30,7 @@ Canonical JSON lives in **`../../contracts/`** (CLI package root, published in t
|
|
|
12
30
|
| `npm run check:generated-contracts` | Verify committed JSON matches generators |
|
|
13
31
|
| `npm run sync:parity-snapshot` | Copy canonical → vscode `contracts/` mirror |
|
|
14
32
|
| `npm run check:parity-snapshot` | Verify mirrors match canonical |
|
|
15
|
-
| `npm run validate:contracts` | Shared-contract checks and focused contract tests
|
|
33
|
+
| `npm run validate:contracts` | Shared-contract checks and focused contract tests |
|
|
16
34
|
| `npm run contracts:validate` | Comprehensive generated/shared contract, parity, runtime-conformance, and adversarial gate |
|
|
17
35
|
| `npm run check:agent-customization-drift` | Verify generated agent customization files are committed in a consumer workspace |
|
|
18
36
|
|
|
@@ -20,19 +38,23 @@ Workflow: change code → `npm run generate:contracts` → `npm run sync:parity-
|
|
|
20
38
|
|
|
21
39
|
## Documents in this folder
|
|
22
40
|
|
|
23
|
-
| File | Purpose
|
|
24
|
-
| -------------------------------------------------------------- |
|
|
25
|
-
| [ARTIFACT_CATALOG.md](./ARTIFACT_CATALOG.md) | On-disk artifact paths, schema versions, and consumer rules
|
|
26
|
-
| [COMMAND_OWNERSHIP_MATRIX.md](./COMMAND_OWNERSHIP_MATRIX.md) | Which commands the npm wrapper owns vs Python Core
|
|
41
|
+
| File | Purpose |
|
|
42
|
+
| -------------------------------------------------------------- | ----------------------------------------------------------------- |
|
|
43
|
+
| [ARTIFACT_CATALOG.md](./ARTIFACT_CATALOG.md) | On-disk artifact paths, schema versions, and consumer rules |
|
|
44
|
+
| [COMMAND_OWNERSHIP_MATRIX.md](./COMMAND_OWNERSHIP_MATRIX.md) | Which commands the npm wrapper owns vs Python Core |
|
|
27
45
|
| [NAMING_AND_COEXISTENCE.md](./NAMING_AND_COEXISTENCE.md) | Workspace Intelligence command naming and generated surface rules |
|
|
28
|
-
| [RUNTIME_SUPPORT_MATRIX.md](./RUNTIME_SUPPORT_MATRIX.md) | Scaffold, import, lifecycle, and module support tiers
|
|
29
|
-
| [RUNTIME_ACCEPTANCE_MATRIX.md](./RUNTIME_ACCEPTANCE_MATRIX.md) | Runtime acceptance matrix expectations
|
|
30
|
-
| [rapidkit-cli-contracts.json](./rapidkit-cli-contracts.json) | Core CLI JSON schema fragments
|
|
46
|
+
| [RUNTIME_SUPPORT_MATRIX.md](./RUNTIME_SUPPORT_MATRIX.md) | Scaffold, import, lifecycle, and module support tiers |
|
|
47
|
+
| [RUNTIME_ACCEPTANCE_MATRIX.md](./RUNTIME_ACCEPTANCE_MATRIX.md) | Runtime acceptance matrix expectations |
|
|
48
|
+
| [rapidkit-cli-contracts.json](./rapidkit-cli-contracts.json) | Core CLI JSON schema fragments |
|
|
31
49
|
|
|
32
50
|
## Workspace intelligence schemas
|
|
33
51
|
|
|
34
52
|
Published under `../../contracts/` (not duplicated in this folder):
|
|
35
53
|
|
|
54
|
+
- `published-contract-catalog.v1.json` — complete machine-readable contract inventory
|
|
55
|
+
- `workspace-contract.v1.json` — canonical workspace project/relationship contract
|
|
56
|
+
- `runtime-command-surface.v1.json` and `cli-runtime-command-inventory.v1.snapshot.json` — supported command/capability discovery
|
|
57
|
+
- `workspace-intelligence-architecture.v1.json` and `workspace-intelligence-chain.v1.json` — architecture boundaries and ordered loop
|
|
36
58
|
- `workspace-registry.v1.json` — canonical project registry summary (see [ARTIFACT_CATALOG.md](./ARTIFACT_CATALOG.md))
|
|
37
59
|
- `release-readiness.v1.json` — release readiness gate evidence
|
|
38
60
|
- `workspace-run-last.v1.json` — multi-stage workspace run evidence
|
|
@@ -44,6 +66,10 @@ Published under `../../contracts/` (not duplicated in this folder):
|
|
|
44
66
|
- `project-entry-capability.v1.json` — open-ended adopt/import contract for readable projects
|
|
45
67
|
- `create-planner-capabilities.v1.json` — native, official, and existing capability lanes
|
|
46
68
|
- `agent-customization-pack.v1.json` — generated instructions, prompts, skills, agents, optional hooks, MCP-ready design metadata, target matrix, and drift state for AI agent surfaces
|
|
69
|
+
- `workspace-list.v1.json`, `workspace-sync.v1.json`, and `compatibility-matrix.v1.json` — workspace discovery, synchronization, and platform compatibility
|
|
70
|
+
- `project-archive.v1.json`, `workspace-snapshot.v1.json`, and `workspace-snapshot.v2.json` — recoverable lifecycle records
|
|
71
|
+
- `infra-plan.v1.json`, `private-product-manifest.v1.json`, and `product-factory-plan.v1.json` — infrastructure and product planning payloads
|
|
72
|
+
- `workspace-model-cache.v1.json`, `workspace-watch-event.v1.json`, `doctor-project-scan.v2.json`, and `doctor-workspace-cache.v2.json` — cache/watch/diagnostic support contracts
|
|
47
73
|
|
|
48
74
|
Workspace intelligence (`../../contracts/workspace-intelligence/`):
|
|
49
75
|
|
|
@@ -51,6 +77,13 @@ Workspace intelligence (`../../contracts/workspace-intelligence/`):
|
|
|
51
77
|
- `workspace-model.v1.json`
|
|
52
78
|
- `workspace-context.v1.json`
|
|
53
79
|
- `workspace-dependency-graph.v1.json`
|
|
80
|
+
- `workspace-knowledge-graph.v1.json` — proof-backed entities, relations, evidence, providers, and model binding
|
|
81
|
+
- `workspace-knowledge-graph-change-overlay.v1.json` — proposed/change-set facts and relations without mutating the base graph
|
|
82
|
+
- `workspace-knowledge-search.v1.json` — bounded ranked retrieval for CLI, MCP, IDE, and agent consumers
|
|
83
|
+
- `workspace-graph-token-efficiency.v1.json` — reproducible corpus-versus-retrieval payload measurement
|
|
84
|
+
- `model-usage-event.v1.json` — privacy-bounded model, tool, milestone, and verified-outcome events with explicit measurement provenance
|
|
85
|
+
- `workspace-intelligence-evaluation.v1.json` — live/final token, cost, latency, activity, and verified-outcome evaluation
|
|
86
|
+
- `workspace-intelligence-evaluation-comparison.v1.json` — task-aligned comparison of two completed evaluation strategies
|
|
54
87
|
- `workspace-model-snapshot.v1.json`
|
|
55
88
|
- `workspace-model-diff.v1.json`
|
|
56
89
|
- `workspace-impact.v1.json`
|
|
@@ -64,6 +97,12 @@ Workspace intelligence (`../../contracts/workspace-intelligence/`):
|
|
|
64
97
|
- `blocker-resolution.v1.json`
|
|
65
98
|
- `doctor-fix-result.v1.json`
|
|
66
99
|
- `studio-blocker-handoff.v1.json`
|
|
100
|
+
- `mcp-design.v1.json` and `agent-hooks.v1.json` — generated MCP/IDE integration surfaces
|
|
101
|
+
|
|
102
|
+
These schemas describe durable artifacts or bounded query results. A command's
|
|
103
|
+
stdout may wrap an artifact with operation metadata such as `status`,
|
|
104
|
+
`outputPath`, or a structured error; that envelope follows
|
|
105
|
+
`cli-operation-result.v1.json` and does not change the nested artifact contract.
|
|
67
106
|
|
|
68
107
|
CLI commands: see [commands-reference.md](../commands-reference.md) and the
|
|
69
108
|
[CLI README](../../README.md#one-intelligence-chain).
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
# Graph Retrieval Benchmark Methodology
|
|
2
|
+
|
|
3
|
+
This document defines what Workspai measures when it reports graph retrieval
|
|
4
|
+
payload reduction, how to reproduce a result, and what the result does—and does
|
|
5
|
+
not—prove.
|
|
6
|
+
|
|
7
|
+
## The question being measured
|
|
8
|
+
|
|
9
|
+
For one query, how much smaller is the bounded, proof-carrying graph response
|
|
10
|
+
than the readable source corpus that supplied the graph's proofs?
|
|
11
|
+
|
|
12
|
+
This is useful because AI agents do not need every indexed file for every
|
|
13
|
+
question. It is deliberately narrower than “How many tokens will this model
|
|
14
|
+
bill?” or “Will the model produce an equally good answer?”
|
|
15
|
+
|
|
16
|
+
## Run it
|
|
17
|
+
|
|
18
|
+
From a Workspai workspace:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
npx workspai workspace model --write --json
|
|
22
|
+
npx workspai workspace graph benchmark "authentication endpoint" --limit 12 --json
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The result conforms to
|
|
26
|
+
[`workspace-graph-token-efficiency.v1.json`](../contracts/workspace-intelligence/workspace-graph-token-efficiency.v1.json).
|
|
27
|
+
It records:
|
|
28
|
+
|
|
29
|
+
- the query and result limit;
|
|
30
|
+
- the graph schema, entity/relation/proof counts, source artifact, and source
|
|
31
|
+
model SHA-256;
|
|
32
|
+
- the number and size of readable, deduplicated proof-source artifacts;
|
|
33
|
+
- the bounded retrieval size and match count;
|
|
34
|
+
- unreadable artifacts rather than silently excluding them;
|
|
35
|
+
- the estimate formula, reduction ratio, percentage, and claim boundary.
|
|
36
|
+
|
|
37
|
+
## Baseline and formula
|
|
38
|
+
|
|
39
|
+
The current methodology is `indexed-corpus-vs-bounded-retrieval.v1`.
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
corpus characters = sum(unique readable proof-source files)
|
|
43
|
+
retrieval characters = compact JSON length of bounded search response
|
|
44
|
+
estimated tokens = ceil(characters / 4)
|
|
45
|
+
reduction ratio = corpus estimated tokens / retrieval estimated tokens
|
|
46
|
+
reduction percent = (corpus - retrieval) / corpus × 100
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`characters / 4` is intentionally labelled as an estimate. It is portable and
|
|
50
|
+
reproducible without downloading a tokenizer, but it is not exact for every
|
|
51
|
+
language, model, or tokenizer.
|
|
52
|
+
|
|
53
|
+
## Current fixture observation
|
|
54
|
+
|
|
55
|
+
The 16-project development workspace reproduced the following result on
|
|
56
|
+
2026-07-22 for `api endpoint --limit 8`:
|
|
57
|
+
|
|
58
|
+
| Measure | Observed value |
|
|
59
|
+
| ------------------------------- | -------------: |
|
|
60
|
+
| Graph entities | 1,738 |
|
|
61
|
+
| Graph relations | 2,244 |
|
|
62
|
+
| Graph proofs | 2,106 |
|
|
63
|
+
| Readable proof-source artifacts | 392 |
|
|
64
|
+
| Corpus estimated tokens | 134,105 |
|
|
65
|
+
| Retrieval estimated tokens | 2,812 |
|
|
66
|
+
| Returned entities | 8 |
|
|
67
|
+
| Retrieval ratio | 47.69× |
|
|
68
|
+
| Payload reduction | 97.9% |
|
|
69
|
+
|
|
70
|
+
Source-model SHA-256:
|
|
71
|
+
`2b8abd415420cc421707c726e6f6c96641554594e84440bf2e539f04ba5836e8`.
|
|
72
|
+
|
|
73
|
+
This row demonstrates that measurement is possible. It is not a representative
|
|
74
|
+
cross-project benchmark and must not be marketed as a universal Workspai result.
|
|
75
|
+
|
|
76
|
+
## What can be claimed today
|
|
77
|
+
|
|
78
|
+
Safe wording:
|
|
79
|
+
|
|
80
|
+
> Workspai can return bounded, proof-carrying workspace context instead of the
|
|
81
|
+
> complete indexed corpus. On the current 16-project development fixture, one
|
|
82
|
+
> `api endpoint` query reduced the estimated retrieval payload by 97.9%; results
|
|
83
|
+
> vary by workspace and query.
|
|
84
|
+
|
|
85
|
+
Unsafe wording:
|
|
86
|
+
|
|
87
|
+
- “Workspai always reduces model tokens by 97.9%.”
|
|
88
|
+
- “Agents are 47.69× cheaper with no quality loss.”
|
|
89
|
+
- “Workspai beats another product” without a shared corpus and evaluation
|
|
90
|
+
harness.
|
|
91
|
+
|
|
92
|
+
## Gate for a public headline benchmark
|
|
93
|
+
|
|
94
|
+
Before publishing a general token-efficiency number, the benchmark suite must:
|
|
95
|
+
|
|
96
|
+
1. pin public repositories and exact commit SHAs;
|
|
97
|
+
2. publish fixed question sets and graph configuration;
|
|
98
|
+
3. compare at least three baselines:
|
|
99
|
+
- entire readable corpus;
|
|
100
|
+
- a realistic grep/top-file retrieval strategy;
|
|
101
|
+
- bounded Workspai graph retrieval;
|
|
102
|
+
4. count with at least one real, named tokenizer in addition to the portable
|
|
103
|
+
character estimate;
|
|
104
|
+
5. measure answer relevance or task completion so smaller context is not treated
|
|
105
|
+
as automatically better context;
|
|
106
|
+
6. repeat runs and publish variance, failures, unreadable files, hardware, and
|
|
107
|
+
software versions;
|
|
108
|
+
7. publish raw machine-readable results and a one-command reproduction path;
|
|
109
|
+
8. report median and range—not only the best repository.
|
|
110
|
+
|
|
111
|
+
## Performance is a separate benchmark
|
|
112
|
+
|
|
113
|
+
Payload size and graph speed answer different questions. Build time, incremental
|
|
114
|
+
update time, peak memory, artifact size, and p50/p95 query latency must be
|
|
115
|
+
measured separately. Do not infer runtime performance from the token-efficiency
|
|
116
|
+
report.
|
|
117
|
+
|
|
118
|
+
For normal interactive use, prefer `workspace graph search` or the MCP
|
|
119
|
+
`searchWorkspaceGraph` tool. Use the complete graph artifact for interchange,
|
|
120
|
+
offline analysis, audits, and consumers that explicitly require the entire
|
|
121
|
+
workspace representation.
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# Measure Workspace Intelligence Usage
|
|
2
|
+
|
|
3
|
+
Workspai can record model usage, tool activity, cost provenance, and the final
|
|
4
|
+
verified outcome of an agent task. The resulting artifact is designed for CLI,
|
|
5
|
+
IDE, CI, and dashboard consumers.
|
|
6
|
+
|
|
7
|
+
It does not store prompt or response bodies. Optional SHA-256 hashes let a
|
|
8
|
+
consumer correlate calls without copying private content into the report.
|
|
9
|
+
|
|
10
|
+
## Start a measured task
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
npx workspai workspace eval init repair-readiness workspace-intelligence --json
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
The optional strategy is one of `full-corpus`, `grep`, `vector`, `graph`,
|
|
17
|
+
`workspace-intelligence`, or `custom`.
|
|
18
|
+
|
|
19
|
+
The live source of truth is:
|
|
20
|
+
|
|
21
|
+
```text
|
|
22
|
+
.workspai/reports/workspace-intelligence-evaluation-live.json
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Record events
|
|
26
|
+
|
|
27
|
+
Providers and IDE integrations append privacy-bounded events through stdin:
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
printf '%s\n' '{
|
|
31
|
+
"kind": "model-call",
|
|
32
|
+
"stage": "context",
|
|
33
|
+
"modelCall": {
|
|
34
|
+
"provider": "vscode-lm",
|
|
35
|
+
"model": "selected-model",
|
|
36
|
+
"source": "provider-reported",
|
|
37
|
+
"inputTokens": 18420,
|
|
38
|
+
"outputTokens": 2150,
|
|
39
|
+
"cachedInputTokens": 6200,
|
|
40
|
+
"reasoningTokens": null,
|
|
41
|
+
"latencyMs": 4820
|
|
42
|
+
}
|
|
43
|
+
}' | npx workspai workspace eval record --json
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
`source` is mandatory:
|
|
47
|
+
|
|
48
|
+
| Source | Meaning |
|
|
49
|
+
| ------------------- | -------------------------------------------------------------- |
|
|
50
|
+
| `provider-reported` | The provider returned the count |
|
|
51
|
+
| `tokenizer-counted` | A named tokenizer counted the exact serialized input or output |
|
|
52
|
+
| `estimated` | A documented estimate; never presented as provider billing |
|
|
53
|
+
| `unavailable` | The provider exposed no usable count |
|
|
54
|
+
|
|
55
|
+
Tool events record progress and repeated work without storing command output:
|
|
56
|
+
|
|
57
|
+
```json
|
|
58
|
+
{
|
|
59
|
+
"kind": "tool-call",
|
|
60
|
+
"stage": "verify",
|
|
61
|
+
"toolCall": {
|
|
62
|
+
"tool": "workspace verify",
|
|
63
|
+
"result": "passed",
|
|
64
|
+
"changedSource": false,
|
|
65
|
+
"durationMs": 814,
|
|
66
|
+
"artifact": ".workspai/reports/workspace-verify-last-run.json"
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Finish with a verified outcome event. Tokens per successful outcome remain
|
|
72
|
+
`null` until `status` is `passed` and `verified` is `true`.
|
|
73
|
+
|
|
74
|
+
```json
|
|
75
|
+
{
|
|
76
|
+
"kind": "outcome",
|
|
77
|
+
"stage": "verify",
|
|
78
|
+
"outcome": {
|
|
79
|
+
"status": "passed",
|
|
80
|
+
"verified": true,
|
|
81
|
+
"blockersResolved": 1,
|
|
82
|
+
"summary": "Readiness passed after dependency remediation."
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Event bodies conform to
|
|
88
|
+
[`model-usage-event.v1.json`](../contracts/workspace-intelligence/model-usage-event.v1.json).
|
|
89
|
+
|
|
90
|
+
## Inspect and finalize
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
npx workspai workspace eval status --json
|
|
94
|
+
npx workspai workspace eval report --json
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Finalization writes:
|
|
98
|
+
|
|
99
|
+
```text
|
|
100
|
+
.workspai/reports/workspace-intelligence-evaluation-last-run.json
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Both live and final reports conform to
|
|
104
|
+
[`workspace-intelligence-evaluation.v1.json`](../contracts/workspace-intelligence/workspace-intelligence-evaluation.v1.json).
|
|
105
|
+
|
|
106
|
+
## Compare strategies
|
|
107
|
+
|
|
108
|
+
Save a completed baseline, run the same fixed task using another strategy, then
|
|
109
|
+
compare:
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
npx workspai workspace eval compare --from .workspai/reports/baselines/full-corpus.json --json
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
The comparison reports token, model-call, tool-call, latency, and verified
|
|
116
|
+
outcome differences. A smaller prompt is not treated as a better result unless
|
|
117
|
+
the outcomes are task-aligned and independently verified.
|
|
118
|
+
|
|
119
|
+
Comparison JSON conforms to
|
|
120
|
+
[`workspace-intelligence-evaluation-comparison.v1.json`](../contracts/workspace-intelligence/workspace-intelligence-evaluation-comparison.v1.json),
|
|
121
|
+
so CI, IDE, and dashboard consumers do not need to infer its shape.
|
|
122
|
+
|
|
123
|
+
## Extension and dashboard contract
|
|
124
|
+
|
|
125
|
+
The VS Code extension should watch the live artifact and display its values; it
|
|
126
|
+
must not independently estimate or relabel them. At minimum the UI should show:
|
|
127
|
+
|
|
128
|
+
- run and session identity;
|
|
129
|
+
- live/final status and update time;
|
|
130
|
+
- input, output, cached-input, and reasoning token components;
|
|
131
|
+
- provider-reported, tokenizer-counted, estimated, and unavailable call counts;
|
|
132
|
+
- cost grouped by currency and source;
|
|
133
|
+
- tool calls, repeated artifact reads, and no-progress decisions;
|
|
134
|
+
- verified outcome and blockers resolved.
|
|
135
|
+
|
|
136
|
+
This makes the CLI artifact the source of truth for terminal, extension, CI,
|
|
137
|
+
and future hosted dashboards. MCP clients can read the same artifact through
|
|
138
|
+
`getWorkspaceEvaluation`; agent-sync also includes the finalized report in the
|
|
139
|
+
governed evidence index when it exists.
|
|
140
|
+
|
|
141
|
+
## Relationship to the graph benchmark
|
|
142
|
+
|
|
143
|
+
`workspace graph benchmark` measures **retrieval payload size** using a portable
|
|
144
|
+
character estimate. `workspace eval` measures **observed agent execution** and
|
|
145
|
+
requires outcome evidence. Keep both: the first is deterministic and
|
|
146
|
+
provider-independent; the second is the basis for real token, cost, and task
|
|
147
|
+
efficiency claims.
|