@klhapp/skillmux 1.12.0 → 1.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/README.md +18 -9
- package/bin/skillmux.js +89 -0
- package/docs/README.md +4 -4
- package/docs/cli.md +14 -4
- package/docs/concepts.md +2 -2
- package/docs/configuration.md +18 -3
- package/docs/deployment.md +3 -3
- package/docs/getting-started.md +32 -7
- package/docs/releasing.md +17 -2
- package/docs/sdd/cli-remote-target-parity/schema-bucket-b.yaml +297 -0
- package/docs/sdd/cli-remote-target-parity/spec-bucket-b.md +95 -0
- package/docs/sdd/cli-remote-target-parity/spec.md +111 -0
- package/docs/sdd/cli-remote-target-parity/think.md +183 -0
- package/docs/sdd/cli-surface-consistency/think.md +149 -0
- package/docs/sdd/config-authority-rerank-resilience/plan.md +338 -0
- package/docs/sdd/config-authority-rerank-resilience/review.md +124 -0
- package/docs/sdd/config-authority-rerank-resilience/spec.md +78 -0
- package/docs/sdd/feat-packaging-deployment/review.md +95 -0
- package/docs/sdd/fetch-outcome-flywheel/review.md +103 -0
- package/docs/sdd/fetch-outcome-flywheel/schema.json +347 -0
- package/docs/sdd/fetch-outcome-flywheel/spec.md +185 -0
- package/docs/sdd/npm-native-binaries/review.md +103 -0
- package/docs/sdd/npm-native-binaries/spec.md +148 -0
- package/docs/sdd/ranked-shortlist-2/pr3-calibration-removal.md +56 -0
- package/docs/sdd/refactor-builtin-target-paths/spec.md +41 -0
- package/docs/sdd/runtime-resource-hardening/review.md +86 -0
- package/docs/sdd/runtime-resource-hardening/spec.md +43 -0
- package/docs/sdd/security-hardening-phase/think.md +129 -0
- package/docs/sdd/security-observability/review.md +96 -0
- package/docs/sdd/security-observability/spec.md +55 -0
- package/docs/sdd/skill-provenance-update/review.md +152 -0
- package/docs/sdd/skill-provenance-update/schema.json +193 -0
- package/docs/sdd/skill-provenance-update/spec.md +134 -0
- package/docs/sdd/supply-chain-hardening/review.md +50 -0
- package/docs/sdd/supply-chain-hardening/spec.md +42 -0
- package/docs/sdd/target-marker-rehome/spec.md +33 -0
- package/docs/troubleshooting.md +18 -2
- package/package.json +15 -8
- package/src/adapters.ts +0 -438
- package/src/audit.ts +0 -21
- package/src/cli.ts +0 -928
- package/src/clients.ts +0 -368
- package/src/commands/audit.ts +0 -85
- package/src/commands/config.ts +0 -224
- package/src/commands/context.ts +0 -104
- package/src/commands/core.ts +0 -56
- package/src/commands/doctor.ts +0 -97
- package/src/commands/eval.ts +0 -88
- package/src/commands/init.ts +0 -672
- package/src/commands/install.ts +0 -155
- package/src/commands/local-vault.ts +0 -60
- package/src/commands/models.ts +0 -10
- package/src/commands/outdated.ts +0 -115
- package/src/commands/project.ts +0 -580
- package/src/commands/report.ts +0 -66
- package/src/commands/scan.ts +0 -71
- package/src/commands/shared.ts +0 -38
- package/src/commands/skill.ts +0 -33
- package/src/commands/sync.ts +0 -233
- package/src/commands/target.ts +0 -225
- package/src/commands/update.ts +0 -277
- package/src/completions.ts +0 -196
- package/src/concurrency-limiter.ts +0 -61
- package/src/config-service.ts +0 -398
- package/src/config-watcher.ts +0 -253
- package/src/config.ts +0 -583
- package/src/context.ts +0 -188
- package/src/db-audit.ts +0 -286
- package/src/db-index.ts +0 -238
- package/src/db.ts +0 -3
- package/src/deployment.ts +0 -39
- package/src/doctor.ts +0 -205
- package/src/eval.ts +0 -302
- package/src/global-flags.ts +0 -46
- package/src/init-agents.ts +0 -329
- package/src/init-instructions.ts +0 -192
- package/src/init.ts +0 -390
- package/src/install.ts +0 -232
- package/src/lifecycle.ts +0 -51
- package/src/logger.ts +0 -26
- package/src/manifest.ts +0 -359
- package/src/mcp-registration.ts +0 -89
- package/src/metrics.ts +0 -121
- package/src/models.ts +0 -20
- package/src/output.ts +0 -214
- package/src/project-setup.ts +0 -36
- package/src/prompts.ts +0 -124
- package/src/provenance.ts +0 -99
- package/src/rate-limiter.ts +0 -137
- package/src/readiness.ts +0 -30
- package/src/redact.ts +0 -52
- package/src/router-core.ts +0 -666
- package/src/rrf.ts +0 -31
- package/src/scan.ts +0 -300
- package/src/server.ts +0 -930
- package/src/setup.ts +0 -145
- package/src/snapshot.ts +0 -135
- package/src/stats.ts +0 -340
- package/src/sync.ts +0 -482
- package/src/toml-writer.ts +0 -51
- package/src/types.ts +0 -212
- package/src/vault.ts +0 -207
|
@@ -0,0 +1,297 @@
|
|
|
1
|
+
openapi: 3.1.0
|
|
2
|
+
info:
|
|
3
|
+
title: Skillmux Admin API — Bucket B additions
|
|
4
|
+
version: "1.0.0"
|
|
5
|
+
description: >
|
|
6
|
+
New endpoints only. Covers docs/sdd/cli-remote-target-parity/spec-bucket-b.md
|
|
7
|
+
AC3, AC6, AC7, AC10. `report` (AC1-2) and `doctor` (AC9) reuse existing
|
|
8
|
+
endpoints (GET /stats, GET /admin/v1/capabilities, GET /admin/v1/config) —
|
|
9
|
+
no new schema needed for those. Error shape matches the existing
|
|
10
|
+
/admin/v1/config convention ({error: CODE, message: human text}), not a
|
|
11
|
+
new one.
|
|
12
|
+
|
|
13
|
+
servers:
|
|
14
|
+
- url: /admin/v1
|
|
15
|
+
|
|
16
|
+
security:
|
|
17
|
+
- bearerAuth: []
|
|
18
|
+
|
|
19
|
+
paths:
|
|
20
|
+
/audit/prune:
|
|
21
|
+
post:
|
|
22
|
+
operationId: adminAuditPrune
|
|
23
|
+
summary: Prune audit/fetch/admin_audit rows older than a cutoff (AC3-5)
|
|
24
|
+
requestBody:
|
|
25
|
+
required: false
|
|
26
|
+
content:
|
|
27
|
+
application/json:
|
|
28
|
+
schema:
|
|
29
|
+
$ref: "#/components/schemas/AuditPruneRequest"
|
|
30
|
+
responses:
|
|
31
|
+
"200":
|
|
32
|
+
description: Prune executed or dry-run counted (AC3, AC4)
|
|
33
|
+
content:
|
|
34
|
+
application/json:
|
|
35
|
+
schema:
|
|
36
|
+
$ref: "#/components/schemas/AuditPruneResult"
|
|
37
|
+
"400":
|
|
38
|
+
description: >
|
|
39
|
+
Non-dry-run prune requested without confirm: true (AC5). No rows
|
|
40
|
+
are deleted.
|
|
41
|
+
content:
|
|
42
|
+
application/json:
|
|
43
|
+
schema:
|
|
44
|
+
$ref: "#/components/schemas/AdminError"
|
|
45
|
+
"401":
|
|
46
|
+
$ref: "#/components/responses/Unauthorized"
|
|
47
|
+
"403":
|
|
48
|
+
$ref: "#/components/responses/Forbidden"
|
|
49
|
+
|
|
50
|
+
/eval:
|
|
51
|
+
post:
|
|
52
|
+
operationId: adminEvalRun
|
|
53
|
+
summary: Run vault ranking evaluation against the server's own runtime (AC6)
|
|
54
|
+
requestBody:
|
|
55
|
+
required: false
|
|
56
|
+
responses:
|
|
57
|
+
"200":
|
|
58
|
+
description: Evaluation report, same shape as local `skillmux eval`
|
|
59
|
+
content:
|
|
60
|
+
application/json:
|
|
61
|
+
schema:
|
|
62
|
+
$ref: "#/components/schemas/EvalReport"
|
|
63
|
+
"401":
|
|
64
|
+
$ref: "#/components/responses/Unauthorized"
|
|
65
|
+
"403":
|
|
66
|
+
$ref: "#/components/responses/Forbidden"
|
|
67
|
+
|
|
68
|
+
/eval/promote:
|
|
69
|
+
post:
|
|
70
|
+
operationId: adminEvalPromote
|
|
71
|
+
summary: >
|
|
72
|
+
Return candidate promoted eval cases from the server's own audit db.
|
|
73
|
+
Read-only — writes no file server-side (AC7). The CLI applies dedup
|
|
74
|
+
and writes the local eval-case fixture file itself (AC8).
|
|
75
|
+
requestBody:
|
|
76
|
+
required: true
|
|
77
|
+
content:
|
|
78
|
+
application/json:
|
|
79
|
+
schema:
|
|
80
|
+
$ref: "#/components/schemas/EvalPromoteRequest"
|
|
81
|
+
responses:
|
|
82
|
+
"200":
|
|
83
|
+
description: >
|
|
84
|
+
Raw, undeduplicated candidates. Contains verbatim user query text
|
|
85
|
+
— same privacy note as local `eval promote`'s
|
|
86
|
+
"promoted eval cases contain raw user queries" warning.
|
|
87
|
+
content:
|
|
88
|
+
application/json:
|
|
89
|
+
schema:
|
|
90
|
+
$ref: "#/components/schemas/EvalPromoteResult"
|
|
91
|
+
"400":
|
|
92
|
+
description: Missing or invalid `since`
|
|
93
|
+
content:
|
|
94
|
+
application/json:
|
|
95
|
+
schema:
|
|
96
|
+
$ref: "#/components/schemas/AdminError"
|
|
97
|
+
"401":
|
|
98
|
+
$ref: "#/components/responses/Unauthorized"
|
|
99
|
+
"403":
|
|
100
|
+
$ref: "#/components/responses/Forbidden"
|
|
101
|
+
|
|
102
|
+
components:
|
|
103
|
+
securitySchemes:
|
|
104
|
+
bearerAuth:
|
|
105
|
+
type: http
|
|
106
|
+
scheme: bearer
|
|
107
|
+
description: Same shared admin token as /admin/v1/config (AC10).
|
|
108
|
+
|
|
109
|
+
responses:
|
|
110
|
+
Unauthorized:
|
|
111
|
+
description: Missing or empty bearer token
|
|
112
|
+
content:
|
|
113
|
+
application/json:
|
|
114
|
+
schema:
|
|
115
|
+
$ref: "#/components/schemas/AdminError"
|
|
116
|
+
Forbidden:
|
|
117
|
+
description: Invalid bearer token
|
|
118
|
+
content:
|
|
119
|
+
application/json:
|
|
120
|
+
schema:
|
|
121
|
+
$ref: "#/components/schemas/AdminError"
|
|
122
|
+
|
|
123
|
+
schemas:
|
|
124
|
+
AdminError:
|
|
125
|
+
type: object
|
|
126
|
+
additionalProperties: false
|
|
127
|
+
required: [error, message]
|
|
128
|
+
properties:
|
|
129
|
+
error:
|
|
130
|
+
type: string
|
|
131
|
+
description: Stable machine-readable code, e.g. "CONFIRMATION_REQUIRED".
|
|
132
|
+
message:
|
|
133
|
+
type: string
|
|
134
|
+
description: Human-readable explanation.
|
|
135
|
+
|
|
136
|
+
AuditPruneRequest:
|
|
137
|
+
type: object
|
|
138
|
+
additionalProperties: false
|
|
139
|
+
properties:
|
|
140
|
+
older_than:
|
|
141
|
+
type: string
|
|
142
|
+
description: >
|
|
143
|
+
Relative window (e.g. "30d") or absolute ISO-8601 date. Mirrors
|
|
144
|
+
CLI's --older-than. Omit to use the server's configured
|
|
145
|
+
audit.retention_days.
|
|
146
|
+
dry_run:
|
|
147
|
+
type: boolean
|
|
148
|
+
default: false
|
|
149
|
+
description: Mirrors CLI's --dry-run. When true, counts only, no deletion.
|
|
150
|
+
confirm:
|
|
151
|
+
type: boolean
|
|
152
|
+
default: false
|
|
153
|
+
description: >
|
|
154
|
+
Mirrors CLI's --yes. Required (true) for a non-dry-run prune to
|
|
155
|
+
execute (AC5) — an HTTP call has no interactive confirm, so the
|
|
156
|
+
server enforces this explicitly rather than trusting the caller.
|
|
157
|
+
|
|
158
|
+
AuditPruneResult:
|
|
159
|
+
type: object
|
|
160
|
+
additionalProperties: false
|
|
161
|
+
required: [audit_deleted, fetch_deleted, admin_audit_deleted, dry_run, cutoff]
|
|
162
|
+
properties:
|
|
163
|
+
audit_deleted:
|
|
164
|
+
type: integer
|
|
165
|
+
minimum: 0
|
|
166
|
+
fetch_deleted:
|
|
167
|
+
type: integer
|
|
168
|
+
minimum: 0
|
|
169
|
+
admin_audit_deleted:
|
|
170
|
+
type: integer
|
|
171
|
+
minimum: 0
|
|
172
|
+
dry_run:
|
|
173
|
+
type: boolean
|
|
174
|
+
cutoff:
|
|
175
|
+
type: ["string", "null"]
|
|
176
|
+
format: date-time
|
|
177
|
+
|
|
178
|
+
EvalMetrics:
|
|
179
|
+
type: object
|
|
180
|
+
additionalProperties: false
|
|
181
|
+
required: [recall_at_5, recall_at_10, mrr, ndcg_at_10]
|
|
182
|
+
properties:
|
|
183
|
+
recall_at_5:
|
|
184
|
+
type: number
|
|
185
|
+
recall_at_10:
|
|
186
|
+
type: number
|
|
187
|
+
mrr:
|
|
188
|
+
type: number
|
|
189
|
+
ndcg_at_10:
|
|
190
|
+
type: number
|
|
191
|
+
|
|
192
|
+
CandidateEvalDetail:
|
|
193
|
+
type: object
|
|
194
|
+
additionalProperties: false
|
|
195
|
+
required: [skill_id, lexical_rank, fused_rank]
|
|
196
|
+
properties:
|
|
197
|
+
skill_id:
|
|
198
|
+
type: string
|
|
199
|
+
lexical_rank:
|
|
200
|
+
type: ["integer", "null"]
|
|
201
|
+
fused_rank:
|
|
202
|
+
type: ["integer", "null"]
|
|
203
|
+
reranked_rank:
|
|
204
|
+
type: ["integer", "null"]
|
|
205
|
+
|
|
206
|
+
EvalCaseResult:
|
|
207
|
+
type: object
|
|
208
|
+
additionalProperties: false
|
|
209
|
+
required: [query, relevant_skill_ids, retrieval, latency_ms, recall_settings, candidates]
|
|
210
|
+
properties:
|
|
211
|
+
query:
|
|
212
|
+
type: string
|
|
213
|
+
relevant_skill_ids:
|
|
214
|
+
type: array
|
|
215
|
+
items:
|
|
216
|
+
type: string
|
|
217
|
+
retrieval:
|
|
218
|
+
type: string
|
|
219
|
+
degraded_from:
|
|
220
|
+
type: ["string", "null"]
|
|
221
|
+
degradation_reason:
|
|
222
|
+
type: ["string", "null"]
|
|
223
|
+
latency_ms:
|
|
224
|
+
type: number
|
|
225
|
+
recall_settings:
|
|
226
|
+
type: object
|
|
227
|
+
additionalProperties: false
|
|
228
|
+
required: [k_lexical, k_vector, k_rerank]
|
|
229
|
+
properties:
|
|
230
|
+
k_lexical:
|
|
231
|
+
type: integer
|
|
232
|
+
k_vector:
|
|
233
|
+
type: integer
|
|
234
|
+
k_rerank:
|
|
235
|
+
type: integer
|
|
236
|
+
candidates:
|
|
237
|
+
type: array
|
|
238
|
+
items:
|
|
239
|
+
$ref: "#/components/schemas/CandidateEvalDetail"
|
|
240
|
+
|
|
241
|
+
EvalReport:
|
|
242
|
+
type: object
|
|
243
|
+
additionalProperties: false
|
|
244
|
+
required: [queries, judged_queries, unjudged_queries, lexical, hybrid]
|
|
245
|
+
properties:
|
|
246
|
+
queries:
|
|
247
|
+
type: integer
|
|
248
|
+
minimum: 0
|
|
249
|
+
judged_queries:
|
|
250
|
+
type: integer
|
|
251
|
+
minimum: 0
|
|
252
|
+
unjudged_queries:
|
|
253
|
+
type: integer
|
|
254
|
+
minimum: 0
|
|
255
|
+
lexical:
|
|
256
|
+
$ref: "#/components/schemas/EvalMetrics"
|
|
257
|
+
hybrid:
|
|
258
|
+
$ref: "#/components/schemas/EvalMetrics"
|
|
259
|
+
cases:
|
|
260
|
+
type: array
|
|
261
|
+
items:
|
|
262
|
+
$ref: "#/components/schemas/EvalCaseResult"
|
|
263
|
+
|
|
264
|
+
EvalPromoteRequest:
|
|
265
|
+
type: object
|
|
266
|
+
additionalProperties: false
|
|
267
|
+
required: [since]
|
|
268
|
+
properties:
|
|
269
|
+
since:
|
|
270
|
+
type: string
|
|
271
|
+
description: Relative window (e.g. "7d") or absolute ISO-8601 date. Mirrors CLI's required --since.
|
|
272
|
+
|
|
273
|
+
EvalCase:
|
|
274
|
+
type: object
|
|
275
|
+
additionalProperties: false
|
|
276
|
+
required: [query, relevant_skill_ids]
|
|
277
|
+
properties:
|
|
278
|
+
query:
|
|
279
|
+
type: string
|
|
280
|
+
split:
|
|
281
|
+
type: string
|
|
282
|
+
description: Always "observed" for server-promoted candidates.
|
|
283
|
+
relevant_skill_ids:
|
|
284
|
+
type: array
|
|
285
|
+
items:
|
|
286
|
+
type: string
|
|
287
|
+
|
|
288
|
+
EvalPromoteResult:
|
|
289
|
+
type: object
|
|
290
|
+
additionalProperties: false
|
|
291
|
+
required: [candidates]
|
|
292
|
+
properties:
|
|
293
|
+
candidates:
|
|
294
|
+
type: array
|
|
295
|
+
items:
|
|
296
|
+
$ref: "#/components/schemas/EvalCase"
|
|
297
|
+
description: Raw, undeduplicated candidates. CLI dedups against its local file (AC8).
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
<!-- source: docs/sdd/cli-remote-target-parity/think.md -->
|
|
2
|
+
<!-- plane_project: SMX -->
|
|
3
|
+
<!-- plane_module: cli-remote-target-parity-bucket-b -->
|
|
4
|
+
|
|
5
|
+
# Spec: CLI Remote Target Parity — Bucket B (Remote Admin Parity)
|
|
6
|
+
|
|
7
|
+
Slice 3 of the feature. Slice 1 (Track 1, terminology fixes) shipped as
|
|
8
|
+
PR #163. Slice 2 (Track 2 / Bucket A, reject-cleanly for native/vault-local
|
|
9
|
+
commands) shipped as PR #165. This slice covers Track 2 / Bucket B: `report`,
|
|
10
|
+
`audit prune`, `eval`/`eval promote`, and `doctor` genuinely reach a remote
|
|
11
|
+
server's own admin state instead of `report` hand-rolling its own fetch and
|
|
12
|
+
the rest silently running local-only.
|
|
13
|
+
|
|
14
|
+
## Goal
|
|
15
|
+
`report`, `audit prune`, `eval`/`eval promote`, and `doctor` genuinely reach
|
|
16
|
+
a remote server's own admin state instead of `report` hand-rolling its own
|
|
17
|
+
fetch and the rest silently running local-only.
|
|
18
|
+
|
|
19
|
+
## Acceptance Criteria
|
|
20
|
+
|
|
21
|
+
1. `report --context <remote>`/`--server <url>` fetches via
|
|
22
|
+
`TargetAdapter`/`RemoteAdapter`'s existing shared `fetchJson()` (reusing
|
|
23
|
+
the already-shipped `GET /stats` endpoint) instead of its own hand-rolled
|
|
24
|
+
`fetch()`/loopback-check/auth logic (`cli.ts` ~1613-1640). Output shape
|
|
25
|
+
(`StatsResponse`, text/JSON rendering) is unchanged — client-side refactor
|
|
26
|
+
only.
|
|
27
|
+
2. If the resolved remote context's admin token env var is unset or empty,
|
|
28
|
+
`report` fails fast with a clear error before any network request —
|
|
29
|
+
replacing today's silent unauthenticated fetch.
|
|
30
|
+
3. A new `POST /admin/v1/audit/prune` endpoint (admin-token authenticated)
|
|
31
|
+
backs new `TargetAdapter.auditPrune()`/`auditCount()` methods, returning
|
|
32
|
+
the same `{audit_deleted, fetch_deleted, admin_audit_deleted, dry_run,
|
|
33
|
+
cutoff}` shape as local `audit prune`.
|
|
34
|
+
4. Remote `audit prune --dry-run` returns counts without mutating anything
|
|
35
|
+
server-side.
|
|
36
|
+
5. Remote `audit prune` (non-dry-run) requires `--yes`; omitting it prunes
|
|
37
|
+
nothing and returns a clear error — mirroring the existing local
|
|
38
|
+
non-interactive requirement, since an interactive y/n confirm has no
|
|
39
|
+
meaning over a stateless HTTP call.
|
|
40
|
+
6. A new `POST /admin/v1/eval` endpoint (admin-token authenticated) backs
|
|
41
|
+
`TargetAdapter.evalRun()`; the server runs `evalVault()` against its own
|
|
42
|
+
in-process runtime and returns the same `EvalReport` shape (lexical/hybrid
|
|
43
|
+
metrics) as local `eval`.
|
|
44
|
+
7. A new `POST /admin/v1/eval/promote` endpoint (admin-token authenticated)
|
|
45
|
+
backs `TargetAdapter.evalPromote()`; the server computes candidate
|
|
46
|
+
promoted cases from its *own* audit db and returns them as JSON — it does
|
|
47
|
+
not write any file server-side.
|
|
48
|
+
8. `eval promote --context <remote>` applies the existing local dedup
|
|
49
|
+
(`excludeExistingCases`) against the CLI's local `eval-observed.json`
|
|
50
|
+
before writing, exactly as local `eval promote` does today — the server
|
|
51
|
+
only supplies raw, undeduplicated candidates; the CLI remains the sole
|
|
52
|
+
writer of the local eval-case fixture file.
|
|
53
|
+
9. `doctor --context <remote>` shows a remote view built from the existing
|
|
54
|
+
`getCapabilities()`/`configStatus()` — no new endpoint. Local `doctor`
|
|
55
|
+
(default, no `--context`/`--server`) is completely unchanged.
|
|
56
|
+
10. All three new endpoints (`/admin/v1/audit/prune`, `/admin/v1/eval`,
|
|
57
|
+
`/admin/v1/eval/promote`) require the same admin bearer token as
|
|
58
|
+
`/admin/v1/config`; an unauthenticated or invalid-token request is
|
|
59
|
+
rejected (401/403) and performs no computation or mutation.
|
|
60
|
+
11. Bucket A's already-shipped reject-cleanly behavior (14 local-only
|
|
61
|
+
commands) and `config`/`context`'s existing correct remote handling are
|
|
62
|
+
unaffected — regression only, no behavior change.
|
|
63
|
+
12. `docs/cli.md` and `docs/concepts.md`'s HTTP-surfaces table document the
|
|
64
|
+
three new endpoints and the new remote behavior for `report`/
|
|
65
|
+
`audit prune`/`eval`/`eval promote`/`doctor`.
|
|
66
|
+
|
|
67
|
+
## Scope
|
|
68
|
+
- `report`'s remote path refactored onto `RemoteAdapter`'s shared fetch/auth
|
|
69
|
+
machinery (existing endpoint, no new route).
|
|
70
|
+
- New `TargetAdapter` methods: `auditPrune()`, `auditCount()`, `evalRun()`,
|
|
71
|
+
`evalPromote()`.
|
|
72
|
+
- New server endpoints: `POST /admin/v1/audit/prune`, `POST /admin/v1/eval`,
|
|
73
|
+
`POST /admin/v1/eval/promote`, all admin-token authenticated, all
|
|
74
|
+
synchronous (client blocks for the full result — no job/polling model).
|
|
75
|
+
- `doctor`'s optional remote view.
|
|
76
|
+
- Docs for all of the above.
|
|
77
|
+
|
|
78
|
+
## Out of Scope
|
|
79
|
+
- RBAC/SSO/access-control granularity beyond the single shared admin token —
|
|
80
|
+
reconfirmed per the 2026-08-29 decision.
|
|
81
|
+
- Any async/job-polling execution model for `eval` — stays synchronous,
|
|
82
|
+
matching local behavior.
|
|
83
|
+
- Any change to Bucket A's reject-cleanly commands or to `config`/`context`.
|
|
84
|
+
- Version negotiation for older servers lacking these endpoints — no such
|
|
85
|
+
infra exists elsewhere in this codebase; not introduced here.
|
|
86
|
+
- The `skillmux route "<query>" --context prod` MCP-preview command — still
|
|
87
|
+
no demand.
|
|
88
|
+
|
|
89
|
+
## Tasks
|
|
90
|
+
1. `report` refactor onto `RemoteAdapter`'s shared fetch (AC1-2).
|
|
91
|
+
2. `audit prune` remote: new endpoint + adapter methods + `--yes` requirement
|
|
92
|
+
(AC3-5).
|
|
93
|
+
3. `eval`/`eval promote` remote: new endpoints + adapter methods +
|
|
94
|
+
client-side dedup for promote (AC6-8).
|
|
95
|
+
4. `doctor` remote view + docs (AC9-10, 12).
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
<!-- source: docs/sdd/cli-remote-target-parity/think.md -->
|
|
2
|
+
<!-- plane_project: SMX -->
|
|
3
|
+
<!-- plane_module: cli-remote-target-parity -->
|
|
4
|
+
|
|
5
|
+
# Spec: CLI Remote Target Parity — Bucket A (Reject Cleanly)
|
|
6
|
+
|
|
7
|
+
Slice 2 of the feature. Slice 1 (Track 1 — terminology fixes) shipped as
|
|
8
|
+
PR #163, merged to main at `e942075`. This slice covers Track 2 / Bucket A
|
|
9
|
+
only: native/vault-local commands reject a remote context cleanly. Bucket B
|
|
10
|
+
(new remote admin capability for `report`/`audit prune`/`eval`) is a further
|
|
11
|
+
slice.
|
|
12
|
+
|
|
13
|
+
## Goal
|
|
14
|
+
Every native/vault-local CLI command rejects a remote `--context`/`--server`
|
|
15
|
+
with one clear, consistent error instead of silently running against local
|
|
16
|
+
state.
|
|
17
|
+
|
|
18
|
+
## Why this rejects (not routes)
|
|
19
|
+
|
|
20
|
+
`--context`/`--server` selects which Skillmux *admin instance* a command
|
|
21
|
+
talks to over `/admin/v1/*` — nothing more. That's one of three independent
|
|
22
|
+
axes in this system, and the recurring confusion (including mid-design-session
|
|
23
|
+
confusion this feature itself grew out of) comes from the name implying a
|
|
24
|
+
broader "local mode vs. remote mode" switch than it actually is:
|
|
25
|
+
|
|
26
|
+
1. **Which vault checkout backs Core/Project pinning** — always a local Git
|
|
27
|
+
checkout on the machine running the CLI. `install`/`update`/`sync`/`core`/
|
|
28
|
+
`project`/`target`/`local-vault`/`index`/`models`/`scan`/`init`/`outdated`/
|
|
29
|
+
`serve`/`skill which` operate on it. There is no remote version of this —
|
|
30
|
+
Git and the deployment process move content between checkouts, not
|
|
31
|
+
Skillmux's own commands (already documented, `concepts.md:101-104`).
|
|
32
|
+
2. **Which MCP server an agent queries for Routed retrieval** — the agent's
|
|
33
|
+
own MCP client configuration (local stdio vs. remote HTTP), entirely
|
|
34
|
+
separate from `skillmux context`.
|
|
35
|
+
3. **Which Skillmux instance the CLI's admin commands act on** — `config`,
|
|
36
|
+
`report`, and (in the next slice) `audit prune`/`eval`. This is the only
|
|
37
|
+
thing `--context`/`--server` actually selects.
|
|
38
|
+
|
|
39
|
+
The commands in this slice belong entirely to axis 1. `--context` has no
|
|
40
|
+
meaning for them — not "risky," a category error, the same way `--context
|
|
41
|
+
prod` wouldn't mean anything on `ls`. Today the CLI accepts the flag and
|
|
42
|
+
silently discards it, so a user can believe `skillmux install foo --context
|
|
43
|
+
prod` acted on `prod` when it only mutated their own machine. This slice
|
|
44
|
+
replaces that silent, misleading no-op with an explicit, correct refusal.
|
|
45
|
+
|
|
46
|
+
## Acceptance Criteria
|
|
47
|
+
|
|
48
|
+
1. When the resolved context is remote (`resolvedContext.type === "remote"`),
|
|
49
|
+
each of these commands is rejected before its handler runs, with a
|
|
50
|
+
non-zero exit: `install`, `update`, `outdated`, `sync`, `core`, `project`,
|
|
51
|
+
`target`, `local-vault`, `index`, `models`, `scan`, `init`, `serve`,
|
|
52
|
+
`skill which`.
|
|
53
|
+
2. The rejection is produced by one shared, centralized guard — not
|
|
54
|
+
per-command duplicated logic — reusing the existing Docker-host-management
|
|
55
|
+
rejection's shape (`docs/cli.md`): exit code 2, JSON error code
|
|
56
|
+
`REMOTE_CONTEXT_UNSUPPORTED`, includes `rejected_command`.
|
|
57
|
+
3. Human-mode error text follows one template naming the actual command:
|
|
58
|
+
`` `<command>` operates on the local vault only; --context/--server isn't
|
|
59
|
+
supported here ``.
|
|
60
|
+
4. In `--json` mode, the rejection is a structured error envelope
|
|
61
|
+
(`formatJsonEnvelope`-style) with `code: "REMOTE_CONTEXT_UNSUPPORTED"` and
|
|
62
|
+
`rejected_command`, not free text only.
|
|
63
|
+
5. When the resolved context is local (the default, no `--context`/
|
|
64
|
+
`--server`, or an explicit `--context local`), all Bucket A commands
|
|
65
|
+
behave exactly as before — zero regression, verified per-command.
|
|
66
|
+
6. `context`, `config`, `report`, `audit`, `eval`, `eval promote`, `doctor`
|
|
67
|
+
are explicitly untouched by this slice — `context`/`config` already
|
|
68
|
+
handle remote correctly, the rest are deferred to the next slice (Bucket
|
|
69
|
+
B) and must continue behaving exactly as they do today, including their
|
|
70
|
+
current silent-local behavior for `audit`/`eval`/`doctor`.
|
|
71
|
+
7. `docs/cli.md` documents the new rejection behavior for Bucket A commands
|
|
72
|
+
(same style as the existing Docker-host-management rejection section) and
|
|
73
|
+
states the three-axes distinction above, so the reason these commands
|
|
74
|
+
reject — rather than route — is written down, not just inferable from
|
|
75
|
+
code.
|
|
76
|
+
|
|
77
|
+
## Scope
|
|
78
|
+
- A single centralized guard (at CLI dispatch time, not scattered
|
|
79
|
+
per-handler) that rejects Bucket A commands when the resolved context is
|
|
80
|
+
remote.
|
|
81
|
+
- The shared JSON error contract (code + `rejected_command`) and
|
|
82
|
+
human-readable message template.
|
|
83
|
+
- Regression test coverage across all 14 Bucket A commands for both local
|
|
84
|
+
(unchanged) and remote (rejected) contexts.
|
|
85
|
+
- Docs update, including the three-axes rationale.
|
|
86
|
+
|
|
87
|
+
## Out of Scope
|
|
88
|
+
- Bucket B (`report` refactor onto `TargetAdapter`, new `audit prune`/
|
|
89
|
+
`eval`/`eval promote` remote capability, new `/admin/v1` endpoints,
|
|
90
|
+
`doctor`'s optional remote view) — next slice.
|
|
91
|
+
- RBAC/SSO/team accounts/multi-tenancy — deferred per the 2026-08-29
|
|
92
|
+
decision.
|
|
93
|
+
- Full remote vault mutation — rejected architecturally in the think.md, not
|
|
94
|
+
revisited.
|
|
95
|
+
- The `skillmux route "<query>" --context ...` MCP-preview command — no
|
|
96
|
+
demand, not bundled.
|
|
97
|
+
- Renaming `skill which` — already addressed via clarification only, in
|
|
98
|
+
Track 1 (PR #163); this slice only adds it to the reject-cleanly list
|
|
99
|
+
under its current name.
|
|
100
|
+
- Any change to how agents discover/load skills natively.
|
|
101
|
+
|
|
102
|
+
## Tasks
|
|
103
|
+
1. Add a single dispatch-level guard in `main()` (a `LOCAL_ONLY_COMMANDS` set
|
|
104
|
+
checked once, before the command switch invokes a handler).
|
|
105
|
+
2. Wire the JSON error envelope (`code: "REMOTE_CONTEXT_UNSUPPORTED"`,
|
|
106
|
+
`rejected_command`) and human message template, mirroring the existing
|
|
107
|
+
Docker-rejection contract.
|
|
108
|
+
3. Regression tests: each Bucket A command × {local context: unchanged,
|
|
109
|
+
remote context: rejected}.
|
|
110
|
+
4. Update `docs/cli.md` with the rejection behavior and the three-axes
|
|
111
|
+
rationale.
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
# CLI Remote Target Parity — Design Doc
|
|
2
|
+
|
|
3
|
+
## Job to Be Done
|
|
4
|
+
|
|
5
|
+
When an operator or agent points the Skillmux CLI at a target — a local
|
|
6
|
+
install (stdio-colocated runtime) or a remote shared server (HTTP deployment)
|
|
7
|
+
— every command should either honor that target correctly or fail with a
|
|
8
|
+
clear, explicit reason. Today many commands silently ignore `--context`/
|
|
9
|
+
`--server` and run against local state instead, so a user can believe they
|
|
10
|
+
just acted on a remote server when they didn't.
|
|
11
|
+
|
|
12
|
+
## Current State
|
|
13
|
+
|
|
14
|
+
- `resolveTarget()`/`ResolvedTarget` (`{type:"local"}|{type:"remote"}`) and
|
|
15
|
+
`TargetAdapter` (`LocalAdapter`/`RemoteAdapter`) already exist
|
|
16
|
+
(`context.ts`, `adapters.ts`), decided and documented 2026-07-21.
|
|
17
|
+
- Only `context`, `config`, and `report` actually consume the resolved
|
|
18
|
+
target. `report` reimplements its own ad hoc `fetch()`/auth-header/
|
|
19
|
+
loopback-check logic (`cli.ts` ~1557-1607) instead of going through
|
|
20
|
+
`RemoteAdapter` — a second, diverging implementation of the same pattern
|
|
21
|
+
`config` already built.
|
|
22
|
+
- Every other command (`install`, `update`, `outdated`, `sync`, `index`,
|
|
23
|
+
`scan`, `eval`, `eval promote`, `doctor`, `skill which`, `audit prune`,
|
|
24
|
+
`core`, `project`, `target`, `local-vault`, `models`) ignores
|
|
25
|
+
`--context`/`--server` entirely — the flag parses successfully and is
|
|
26
|
+
silently discarded. There is no comment anywhere flagging this as a known
|
|
27
|
+
gap; a reader has to trace every call site to discover which commands
|
|
28
|
+
honor the resolved target.
|
|
29
|
+
- The distinction that actually matters is not "local vs. remote
|
|
30
|
+
embeddings" or any inference-specific detail — it's whether the CLI
|
|
31
|
+
process has the runtime (vault, index, audit db, embeddings/reranker
|
|
32
|
+
clients) loaded in-process (stdio-colocated = `local` target) or is a thin
|
|
33
|
+
network client to a separate process that owns that runtime (HTTP =
|
|
34
|
+
`remote` target). This distinction is stated nowhere in code —
|
|
35
|
+
`ResolvedTarget` and `TargetAdapter` carry zero doc comments — and is
|
|
36
|
+
stated inconsistently across docs.
|
|
37
|
+
- A read-only audit this session found a live, demonstrated defect from
|
|
38
|
+
that inconsistency: `docs/cli.md:13-15` correctly says a local *target*
|
|
39
|
+
doesn't imply local *inference*; `docs/cli.md:27-28` contradicts that,
|
|
40
|
+
twelve lines later, by calling `eval` "local embeddings" only;
|
|
41
|
+
`docs/concepts.md:167-170` gets it right again ("inference location and
|
|
42
|
+
deployment location use separate settings"). `src/cli.ts:677`'s error
|
|
43
|
+
message ("eval requires local embeddings") matches the wrong version.
|
|
44
|
+
This produced incorrect reasoning mid-design-session before it was
|
|
45
|
+
caught — not a hypothetical risk.
|
|
46
|
+
- `skill which` (`cli.ts:719-736`) owns the CLI's most routing-suggestive
|
|
47
|
+
name but is pure local vault-checkout shadow resolution ("which root
|
|
48
|
+
wins for this skill id"). It has nothing to do with `resolve_skill`/
|
|
49
|
+
`fetch_skill` semantic routing. No CLI command currently exercises those
|
|
50
|
+
MCP tools — they're reachable only by agents over the MCP transport.
|
|
51
|
+
- Native/vault-mutating commands (`install`, `update`, `sync`, `core`/
|
|
52
|
+
`project` pin, `target`) are already documented
|
|
53
|
+
(`concepts.md:101-104`) as host-only: "the server ... never
|
|
54
|
+
manage[s] host agent directories." Agents only read skill files from
|
|
55
|
+
local, product-specific directories (`~/.claude/skills`,
|
|
56
|
+
`~/.agents/skills`, etc.), so no remote server could make a pin take
|
|
57
|
+
effect there regardless of what the server-side implementation did.
|
|
58
|
+
|
|
59
|
+
## Proposed Solution
|
|
60
|
+
|
|
61
|
+
Two tracks. Track 1 is a prerequisite: the terminology defects found this
|
|
62
|
+
session actively misled reasoning about this very design, so they get fixed
|
|
63
|
+
before (or alongside) the capability work, not after.
|
|
64
|
+
|
|
65
|
+
### Track 1 — Fix the terminology/contract defects
|
|
66
|
+
|
|
67
|
+
- Reconcile the three-way doc contradiction: `docs/cli.md:27-28` should
|
|
68
|
+
match `docs/concepts.md:167-170` — inference location and deployment
|
|
69
|
+
location are independent settings.
|
|
70
|
+
- Fix `src/cli.ts:677`'s error message to state the real requirement: "eval
|
|
71
|
+
requires an embeddings client (local model or a configured remote
|
|
72
|
+
endpoint)."
|
|
73
|
+
- Add a doc comment to `ResolvedTarget` (`context.ts`) and `TargetAdapter`
|
|
74
|
+
(`adapters.ts`) stating the real distinction: `local` = the CLI has the
|
|
75
|
+
runtime in-process; `remote` = the CLI is a thin network client to a
|
|
76
|
+
separate process that owns the runtime.
|
|
77
|
+
- Rename `ResolvedTarget`/`resolveTarget()` (`context.ts`) to
|
|
78
|
+
`ResolvedContext`/`resolveContext()`. The CLI-facing vocabulary for "which
|
|
79
|
+
server/instance am I talking to" is already settled as *context*
|
|
80
|
+
(`--context` flag, `SKILLMUX_CONTEXT` env var, `contexts.toml`, the
|
|
81
|
+
`skillmux context` subcommand — mirroring the kubectl/docker context
|
|
82
|
+
model). The internal type is the only place still using a different word,
|
|
83
|
+
and that word collides with the unrelated `skillmux target` subcommand
|
|
84
|
+
(which manages local agent skill directories, not server selection).
|
|
85
|
+
Renaming the internal type to match the CLI surface removes the collision
|
|
86
|
+
permanently: *context* always means "which server," *target* always means
|
|
87
|
+
"which local directory." Internal-only rename, no CLI-facing or
|
|
88
|
+
compatibility impact — safe to do outright, not gated on an open
|
|
89
|
+
question the way the `skill which` rename is.
|
|
90
|
+
- Add a one-line comment at the flag-parsing site in `cli.ts`'s `main()`
|
|
91
|
+
noting which command handlers don't yet consume `resolvedTarget`, so the
|
|
92
|
+
gap is documented rather than discoverable only by tracing every call
|
|
93
|
+
site.
|
|
94
|
+
- Rename or clarify `skill which` vs. semantic routing (e.g. `skill
|
|
95
|
+
locate`, or at minimum a `--help`/docs clarification) so the name stops
|
|
96
|
+
misleading readers — including future agents working in this repo.
|
|
97
|
+
|
|
98
|
+
### Track 2 — Close the remote-target gap, by bucket
|
|
99
|
+
|
|
100
|
+
- **Bucket A — Native/vault-local (reject remote cleanly).** `install`,
|
|
101
|
+
`update`, `outdated`, `sync`, `core`, `project`, `target`,
|
|
102
|
+
`local-vault`, `index`, `models`, `scan`, `init`, `serve`, `skill
|
|
103
|
+
which`/`locate`. When `resolvedTarget.type === "remote"`, these throw one
|
|
104
|
+
consistent, clear error ("`<command>` operates on the local vault only;
|
|
105
|
+
`--context`/`--server` isn't supported here") instead of silently running
|
|
106
|
+
locally. No new server capability needed.
|
|
107
|
+
- **Bucket B — Server-instance admin state (extend to full remote
|
|
108
|
+
parity).** `config` (done, unchanged). `report` (refactor onto
|
|
109
|
+
`TargetAdapter` instead of its hand-rolled fetch; add `getStats(since)`
|
|
110
|
+
to the interface). `audit prune` (new: `auditPrune()`/`auditCount()` on
|
|
111
|
+
`TargetAdapter` + a new authenticated `/admin/v1/audit` endpoint).
|
|
112
|
+
`eval`/`eval promote` (new: `evalRun()`/`evalPromote()` on
|
|
113
|
+
`TargetAdapter` with server-side execution — `evalVault()` must run
|
|
114
|
+
wherever the runtime, including whatever backs its embeddings, actually
|
|
115
|
+
lives). `doctor` gains an optional remote view that delegates to the
|
|
116
|
+
existing `getCapabilities()`/`configStatus()` rather than inventing new
|
|
117
|
+
endpoints.
|
|
118
|
+
- **Bucket C — `context`.** Stays as the mechanism, unchanged.
|
|
119
|
+
|
|
120
|
+
## Scope
|
|
121
|
+
|
|
122
|
+
- Fixing the terminology/doc/comment defects found in this session's audit
|
|
123
|
+
(Track 1).
|
|
124
|
+
- Making every CLI command either properly honor `--context`/`--server` or
|
|
125
|
+
explicitly and clearly refuse it (Track 2, Bucket A).
|
|
126
|
+
- Extending `TargetAdapter`/`RemoteAdapter` with new methods for stats,
|
|
127
|
+
audit prune, and eval so those genuinely reach a remote server's own
|
|
128
|
+
admin state (Track 2, Bucket B).
|
|
129
|
+
- Unifying `report` onto the same adapter pattern as `config` instead of
|
|
130
|
+
its own hand-rolled fetch.
|
|
131
|
+
|
|
132
|
+
## Out of Scope
|
|
133
|
+
|
|
134
|
+
- RBAC, SSO, team accounts, multi-tenancy (deferred per the 2026-08-29
|
|
135
|
+
decision) — this stays a single shared admin-token model.
|
|
136
|
+
- Full remote vault mutation (`install`/`update`/`sync` executing on a
|
|
137
|
+
remote server). Rejected on architectural grounds, not just risk: agents
|
|
138
|
+
only read skill files from local, product-specific directories, so a
|
|
139
|
+
remote server "installing" something accomplishes nothing for the agent
|
|
140
|
+
that needs it. Not revisited unless that architectural fact changes.
|
|
141
|
+
- A new CLI command that acts as a genuine MCP client to preview
|
|
142
|
+
`resolve_skill`/`fetch_skill` routing (e.g. a hypothetical `skillmux
|
|
143
|
+
route "<query>" --context prod`). Floated during discussion; no existing
|
|
144
|
+
command needs it. Worth a separate future spec if there's real demand,
|
|
145
|
+
not bundled here.
|
|
146
|
+
- Any change to how agents discover/load skills natively (the existing
|
|
147
|
+
core/project/sync mechanism is untouched).
|
|
148
|
+
|
|
149
|
+
## Alternatives Considered
|
|
150
|
+
|
|
151
|
+
- **Full remote parity ("Approach A").** New `/admin/v1` endpoints so
|
|
152
|
+
`install`/`update`/`sync` execute server-side. Rejected: contradicts the
|
|
153
|
+
already-documented "server never manages host agent directories"
|
|
154
|
+
promise, and wouldn't accomplish anything for the agent, which only
|
|
155
|
+
reads local files — an architectural dead end, not just a risk tradeoff.
|
|
156
|
+
- **Leave everything as-is except `report`.** Rejected: leaves the
|
|
157
|
+
dangerous silent-ignore bug live on every other command, and leaves the
|
|
158
|
+
terminology defects that actively misled this session's own design
|
|
159
|
+
discussion.
|
|
160
|
+
|
|
161
|
+
## Open Questions
|
|
162
|
+
|
|
163
|
+
- Exact wording/UX for Bucket A's "operates locally only" refusal — one
|
|
164
|
+
shared helper vs. a per-command message; needs concrete examples during
|
|
165
|
+
`/sdd-spec`.
|
|
166
|
+
- Whether the new `eval`/`audit` remote endpoints need any access-control
|
|
167
|
+
granularity beyond the existing single admin token (e.g. read-only vs.
|
|
168
|
+
mutate). Probably not, given RBAC stays deferred, but worth a sentence in
|
|
169
|
+
the spec confirming.
|
|
170
|
+
- Whether `skill which` gets renamed (a breaking change for scripts) or
|
|
171
|
+
just clarified in place (alias the old name, deprecate over a release) —
|
|
172
|
+
a compatibility call for `/sdd-spec`.
|
|
173
|
+
- Slicing: this doc covers Track 1 and Track 2 conceptually together, but
|
|
174
|
+
following this repo's own precedent (security-hardening-phase sliced
|
|
175
|
+
into 3 PRs), `/sdd-spec` should decide whether Track 1 (terminology
|
|
176
|
+
fixes) ships as its own fast slice before Track 2 (new capability), or
|
|
177
|
+
bundled with Bucket A's reject-cleanly behavior.
|
|
178
|
+
|
|
179
|
+
## Next Step
|
|
180
|
+
|
|
181
|
+
Run `/sdd-spec` to turn this into acceptance criteria — likely sliced per
|
|
182
|
+
the last open question above (terminology-fix slice first, or bundled with
|
|
183
|
+
Bucket A, then Bucket B's new remote admin endpoints as a further slice).
|