@design-ai/cli 4.60.0 → 4.61.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +25 -0
- package/README.ko.md +3 -3
- package/README.md +3 -3
- package/cli/commands/help.mjs +1 -1
- package/cli/commands/search.mjs +81 -2
- package/cli/lib/search-eval.mjs +241 -0
- package/cli/lib/search.mjs +56 -1
- package/docs/AI-LEARNING-PHASE2.md +2 -2
- package/docs/NEXT-SURFACE-DECISION.md +11 -2
- package/docs/ROADMAP.md +37 -0
- package/docs/external-status.md +6 -6
- package/package.json +1 -1
- package/tools/audit/smoke_assertions.py +10 -5
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "design-ai",
|
|
3
|
-
"version": "4.
|
|
3
|
+
"version": "4.61.0",
|
|
4
4
|
"description": "Senior product designer for any AI coding agent. 20 skills, 17 commands, 4 review agents covering UI/UX, website improvement, design systems, motion, illustration, print, video, game UI, conversational, and spatial design. Korean market depth.",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Sungjin Park",
|
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,31 @@
|
|
|
2
2
|
|
|
3
3
|
User-facing release notes for design-ai. Versions follow semver.
|
|
4
4
|
|
|
5
|
+
## v4.61.0 — Ranked-search eval checkpoint (2026-07)
|
|
6
|
+
|
|
7
|
+
Closes the last open follow-up from the AI-learning Phase A review (FU-3 in `docs/AI-LEARNING-PHASE2.md`): `design-ai search` gains eval-checkpoint modes mirroring the route/prompt/pack/learn pattern, run through the shipped ranked (BM25) search path. This is the standing evidence artifact for any future decision to promote `--ranked` to the default search mode. Additive and backward-compatible; default `search` behavior is unchanged.
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
- `search --eval-template [--json]` — emit a versioned ranked-search checkpoint template. The default template ships six cases that all pass against the shipped corpus, including the Korean particle-form regression queries (`버튼`, `버튼을`, `접근성이`) from the Hangul-bigram work, so the eval doubles as the retrieval-level Korean tokenization regression.
|
|
11
|
+
- `search --eval [--strict] [--json]` with `--from-file file` / `--stdin` — run a checkpoint file's cases through the ranked search path and report pass/warn/fail per case; `--strict` exits nonzero on any failure. Case assertions: `expectRelPathIn` (top hit), `minHits`, and `matchedTokenIncludes`.
|
|
12
|
+
|
|
13
|
+
### Changed
|
|
14
|
+
- `search` help/usage lines (CLI `help` topic and command help) document the new eval modes; the shared smoke assertion pins were updated in lockstep.
|
|
15
|
+
|
|
16
|
+
### Verified
|
|
17
|
+
- All 8 audits passed.
|
|
18
|
+
- `npm run release:check`.
|
|
19
|
+
- `npm run release:metadata`.
|
|
20
|
+
- `git diff --check`.
|
|
21
|
+
- Main-branch GitHub Actions (`Design-AI audit`, `Deploy doc site`) passed for the constituent commits.
|
|
22
|
+
|
|
23
|
+
### Versions
|
|
24
|
+
- `package.json` + `.claude-plugin/plugin.json`: 4.60.0 → 4.61.0.
|
|
25
|
+
- `vscode-extension/package.json`: remains 0.4.1.
|
|
26
|
+
|
|
27
|
+
### What this enables
|
|
28
|
+
- Any future `--ranked` default promotion is now evidence-backed and announced: the checkpoint demonstrates ranked-search quality (including Korean agglutinative queries) reproducibly, per the FU-3 gate set in the Phase A implementation review.
|
|
29
|
+
|
|
5
30
|
## v4.60.0 — Agent SDK types and local-write namespace (2026-07)
|
|
6
31
|
|
|
7
32
|
Extends the Agent SDK (`@design-ai/cli/sdk`) two ways, both additive and backward-compatible. First, hand-written TypeScript declarations now ship with the package, so TypeScript and editors resolve `@design-ai/cli/sdk` with full types — no `@types` install and no build step. Second, an opt-in `learn.*` namespace adds the SDK's first local-write verbs, keeping the eight Phase A verbs read-only and unchanged.
|
package/README.ko.md
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
|
|
13
13
|
> **모델이 아니에요. 파인튜닝도 아니에요.** 디자인 전문 지식을 구조화한 코퍼스 + 에이전트가 바로 실행할 수 있는 지시문이에요. 범용 LLM을 이번 세션에서만큼은 시니어 디자이너로 바꿔주는 셈이에요.
|
|
14
14
|
|
|
15
|
-
> **배포 상태, 2026-07-
|
|
15
|
+
> **배포 상태, 2026-07-05 확인:** 로컬 `npm run release:check`는 통과했고, GitHub Pages 문서는 live 상태이며, GitHub Release `v4.60.0`과 npm `@design-ai/cli@4.60.0`(`latest`) publish가 provenance와 함께 확인됐어요. 이번 릴리스는 Agent SDK(`@design-ai/cli/sdk`)에 TypeScript 선언과 opt-in `learn.*` 로컬 쓰기 네임스페이스를 추가했고, 읽기 전용 8개 verb는 그대로라 CLI 출력도 변함없어요. live `npm run registry:smoke`가 published `@design-ai/cli@4.60.0`에 대해 깨끗이 통과했어요(retrieval 표면과 route enrichment 포함). Homebrew formula는 `v4.60.0`에 pinning되어 있고, VS Code Marketplace에는 `sungjin.design-ai-vscode@0.4.1`이 공개되어 있어요. 자세한 내용은 [`docs/external-status.md`](docs/external-status.md)를 확인하세요.
|
|
16
16
|
|
|
17
17
|
## 한눈에 보는 커버리지
|
|
18
18
|
|
|
@@ -75,7 +75,7 @@ design-ai install
|
|
|
75
75
|
/design-from-brief 프리랜서를 위한 한국 핀테크
|
|
76
76
|
```
|
|
77
77
|
|
|
78
|
-
CLI 명령어: `design-ai install [--json]`, `update [--dry-run] [--json]`, `uninstall [--json]`, `status [--json]`, `list [skills|commands|agents] [--json]`, `route <brief|--from-file file|--stdin|--list|--eval-template|--eval> [--limit N] [--explain] [--strict] [--json]`, `routes [--json]`, `prompt <brief|--from-file file|--stdin|--eval-template|--eval> [--out file] [--route id] [--with-learning] [--learning-category kind] [--learning-limit N] [--with-recall] [--recall-limit N] [--strict] [--json]`, `pack <brief|--from-file file|--stdin|--eval-template|--eval> [--out file] [--route id] [--with-learning] [--learning-category kind] [--learning-limit N] [--with-recall] [--recall-limit N] [--max-bytes N] [--strict] [--json]`, `learn [--init|--remember text|--feedback text|--list|--export|--query text|--explain|--recall query|--backup|--redact|--verify|--diff|--restore|--restore-backups [--prune]|--import|--audit [--fix]|--curate|--stats|--usage|--signals [--strict]|--agent-backlog [--strict]|--propose-skills [--min-evidence N] [--review-file path] [--review-check|--apply-plan] [--strict]|--eval-template|--eval [--strict]|--forget id|--clear] [--json|--report|--patch|--review-template] [--out file]`, `check <artifact.md|--stdin|--examples> [--route id|--all-routes] [--issues-only] [--strict] [--learn [--yes] [--learning-file path]] [--json]`, `workspace [--root path] [--learning-file path] [--learning-usage path] [--learning-eval path] [--strict] [--json]`, `site <workspace.json|--stdin> [--strict] [--json|--mcp-check [--probes]|--mcp-plan [--probes] [--json]|--next-actions [--json]|--graph|--tasks|--bundle|--report|--prompts|--prompt id [--task id]] [--out file] | site <bundle-dir> --bundle-check [--json] | site <bundle-dir> --bundle-compare other-bundle-dir [--json] | site <bundle-dir> --bundle-handoff [--task id] [--json] | site <bundle-dir> --bundle-repair [--yes] [--json] [--out file] | site --init --name name --live-url url [--next-actions] [--out file] | site --init --name name --live-url url --bundle --out dir | site --from-intake file.md|--stdin [--json|--next-actions [--json]|--tasks|--bundle [--tasks] --out dir] [--out file] | site --intake-template [--language en|ko] [--json] [--out file] | site --sample [--out file] | site --prompt-list [--json]`, `examples [query|--route id] [--limit N] [--json]`, `search <query> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider "cmd args"]] [--json]`, `index [--build|--status|--verify] [--json] [--embeddings [--provider "cmd args"]]`, `show <file[:line]> [--lines N:M] [--context N] [--json]`, `audit [--strict] [--quiet] [--json]`, `doctor [--strict] [--json] [--fix]`, `mcp`, `version [--json]`, `help [command|--json]`.
|
|
78
|
+
CLI 명령어: `design-ai install [--json]`, `update [--dry-run] [--json]`, `uninstall [--json]`, `status [--json]`, `list [skills|commands|agents] [--json]`, `route <brief|--from-file file|--stdin|--list|--eval-template|--eval> [--limit N] [--explain] [--strict] [--json]`, `routes [--json]`, `prompt <brief|--from-file file|--stdin|--eval-template|--eval> [--out file] [--route id] [--with-learning] [--learning-category kind] [--learning-limit N] [--with-recall] [--recall-limit N] [--strict] [--json]`, `pack <brief|--from-file file|--stdin|--eval-template|--eval> [--out file] [--route id] [--with-learning] [--learning-category kind] [--learning-limit N] [--with-recall] [--recall-limit N] [--max-bytes N] [--strict] [--json]`, `learn [--init|--remember text|--feedback text|--list|--export|--query text|--explain|--recall query|--backup|--redact|--verify|--diff|--restore|--restore-backups [--prune]|--import|--audit [--fix]|--curate|--stats|--usage|--signals [--strict]|--agent-backlog [--strict]|--propose-skills [--min-evidence N] [--review-file path] [--review-check|--apply-plan] [--strict]|--eval-template|--eval [--strict]|--forget id|--clear] [--json|--report|--patch|--review-template] [--out file]`, `check <artifact.md|--stdin|--examples> [--route id|--all-routes] [--issues-only] [--strict] [--learn [--yes] [--learning-file path]] [--json]`, `workspace [--root path] [--learning-file path] [--learning-usage path] [--learning-eval path] [--strict] [--json]`, `site <workspace.json|--stdin> [--strict] [--json|--mcp-check [--probes]|--mcp-plan [--probes] [--json]|--next-actions [--json]|--graph|--tasks|--bundle|--report|--prompts|--prompt id [--task id]] [--out file] | site <bundle-dir> --bundle-check [--json] | site <bundle-dir> --bundle-compare other-bundle-dir [--json] | site <bundle-dir> --bundle-handoff [--task id] [--json] | site <bundle-dir> --bundle-repair [--yes] [--json] [--out file] | site --init --name name --live-url url [--next-actions] [--out file] | site --init --name name --live-url url --bundle --out dir | site --from-intake file.md|--stdin [--json|--next-actions [--json]|--tasks|--bundle [--tasks] --out dir] [--out file] | site --intake-template [--language en|ko] [--json] [--out file] | site --sample [--out file] | site --prompt-list [--json]`, `examples [query|--route id] [--limit N] [--json]`, `search <query|--eval-template|--eval> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider "cmd args"]] [--strict] [--json]`, `index [--build|--status|--verify] [--json] [--embeddings [--provider "cmd args"]]`, `show <file[:line]> [--lines N:M] [--context N] [--json]`, `audit [--strict] [--quiet] [--json]`, `doctor [--strict] [--json] [--fix]`, `mcp`, `version [--json]`, `help [command|--json]`.
|
|
79
79
|
|
|
80
80
|
자세한 내용은 [`docs/DISTRIBUTION.ko.md`](docs/DISTRIBUTION.ko.md)를 확인하세요.
|
|
81
81
|
|
|
@@ -172,7 +172,7 @@ design-ai는 한국 시장을 1순위로 만들어졌고, 글로벌 시장 패
|
|
|
172
172
|
|
|
173
173
|
## 상태
|
|
174
174
|
|
|
175
|
-
전체 단계 로그는 [`docs/ROADMAP.md`](docs/ROADMAP.md), 현재 완료 범위는 [`docs/PRODUCT-READINESS.md`](docs/PRODUCT-READINESS.md)에서 확인하세요. 현재 **v4.
|
|
175
|
+
전체 단계 로그는 [`docs/ROADMAP.md`](docs/ROADMAP.md), 현재 완료 범위는 [`docs/PRODUCT-READINESS.md`](docs/PRODUCT-READINESS.md)에서 확인하세요. 현재 **v4.60.0**: Agent SDK(`@design-ai/cli/sdk`) — TypeScript 선언 + opt-in `learn.*` 로컬 쓰기 네임스페이스, public npm publish, provenance-backed GitHub Actions release, public registry smoke, Website Console MCP readiness, workspace learning restore/eval coverage, handoff bundle verification, 90%+ component coverage가 완료됐어요.
|
|
176
176
|
|
|
177
177
|
핵심 디자인 컨설팅 워크플로우는 로컬 릴리스 기준으로 준비되어 있어요. 웹사이트 개선 컨트롤 타워는 zero-dependency static Web App과 `website-improvement` route/skill/command로 제공되고, Site Profile, audit checklist, MCP readiness, refactor prompt, handoff evidence tracking, bundle export/verify/repair를 한 번에 다뤄요. 로컬 학습 선호도는 `design-ai learn`으로 관리해요 — profile bootstrap, feedback 캡처, 읽기 전용 signal registry, 반복 QA 신호에서 만드는 skill 제안, 그리고 backup/restore/curate/audit까지 전부 로컬에서만 동작하는 opt-in 기능이에요. AI 모델 학습이나 fine-tuning은 여전히 현재 배포 범위 밖이에요.
|
|
178
178
|
|
package/README.md
CHANGED
|
@@ -12,7 +12,7 @@ A model-agnostic design knowledge base + skill system. Drop it in front of any A
|
|
|
12
12
|
|
|
13
13
|
> **Not a model. Not a fine-tune.** A structured corpus of design expertise + agent-ready instructions that turn a general-purpose LLM into an expert.
|
|
14
14
|
|
|
15
|
-
> **Distribution status, checked 2026-07-
|
|
15
|
+
> **Distribution status, checked 2026-07-05:** local `npm run release:check` passes, GitHub Pages docs are live, GitHub Release `v4.60.0` is published, `@design-ai/cli@4.60.0` is public on npm (`latest`) with provenance, and the live `npm run registry:smoke` passes cleanly against `@design-ai/cli@4.60.0`. The Homebrew formula is pinned to `v4.60.0`, and `sungjin.design-ai-vscode@0.4.1` is public on the VS Code Marketplace. See [`docs/external-status.md`](docs/external-status.md).
|
|
16
16
|
|
|
17
17
|
## Coverage at a glance
|
|
18
18
|
|
|
@@ -75,7 +75,7 @@ Any available path installs all 20 skills, 17 commands, and 4 agents under `~/.c
|
|
|
75
75
|
/design-from-brief Korean fintech for freelancers
|
|
76
76
|
```
|
|
77
77
|
|
|
78
|
-
CLI commands: `design-ai install [--json]`, `update [--dry-run] [--json]`, `uninstall [--json]`, `status [--json]`, `list [skills|commands|agents] [--json]`, `route <brief|--from-file file|--stdin|--list|--eval-template|--eval> [--limit N] [--explain] [--strict] [--json]`, `routes [--json]`, `prompt <brief|--from-file file|--stdin|--eval-template|--eval> [--out file] [--route id] [--with-learning] [--learning-category kind] [--learning-limit N] [--with-recall] [--recall-limit N] [--strict] [--json]`, `pack <brief|--from-file file|--stdin|--eval-template|--eval> [--out file] [--route id] [--with-learning] [--learning-category kind] [--learning-limit N] [--with-recall] [--recall-limit N] [--max-bytes N] [--strict] [--json]`, `learn [--init|--remember text|--feedback text|--list|--export|--query text|--explain|--recall query|--backup|--redact|--verify|--diff|--restore|--restore-backups [--prune]|--import|--audit [--fix]|--curate|--stats|--usage|--signals [--strict]|--agent-backlog [--strict]|--propose-skills [--min-evidence N] [--review-file path] [--review-check|--apply-plan] [--strict]|--eval-template|--eval [--strict]|--forget id|--clear] [--json|--report|--patch|--review-template] [--out file]`, `check <artifact.md|--stdin|--examples> [--route id|--all-routes] [--issues-only] [--strict] [--learn [--yes] [--learning-file path]] [--json]`, `workspace [--root path] [--learning-file path] [--learning-usage path] [--learning-eval path] [--strict] [--json]`, `site <workspace.json|--stdin> [--strict] [--json|--mcp-check [--probes]|--mcp-plan [--probes] [--json]|--next-actions [--json]|--graph|--tasks|--bundle|--report|--prompts|--prompt id [--task id]] [--out file] | site <bundle-dir> --bundle-check [--json] | site <bundle-dir> --bundle-compare other-bundle-dir [--json] | site <bundle-dir> --bundle-handoff [--task id] [--json] | site <bundle-dir> --bundle-repair [--yes] [--json] [--out file] | site --init --name name --live-url url [--next-actions] [--out file] | site --init --name name --live-url url --bundle --out dir | site --from-intake file.md|--stdin [--json|--next-actions [--json]|--tasks|--bundle [--tasks] --out dir] [--out file] | site --intake-template [--language en|ko] [--json] [--out file] | site --sample [--out file] | site --prompt-list [--json]`, `examples [query|--route id] [--limit N] [--json]`, `search <query> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider "cmd args"]] [--json]`, `index [--build|--status|--verify] [--json] [--embeddings [--provider "cmd args"]]`, `show <file[:line]> [--lines N:M] [--context N] [--json]`, `audit [--strict] [--quiet] [--json]`, `doctor [--strict] [--json] [--fix]`, `mcp`, `version [--json]`, `help [command|--json]`.
|
|
78
|
+
CLI commands: `design-ai install [--json]`, `update [--dry-run] [--json]`, `uninstall [--json]`, `status [--json]`, `list [skills|commands|agents] [--json]`, `route <brief|--from-file file|--stdin|--list|--eval-template|--eval> [--limit N] [--explain] [--strict] [--json]`, `routes [--json]`, `prompt <brief|--from-file file|--stdin|--eval-template|--eval> [--out file] [--route id] [--with-learning] [--learning-category kind] [--learning-limit N] [--with-recall] [--recall-limit N] [--strict] [--json]`, `pack <brief|--from-file file|--stdin|--eval-template|--eval> [--out file] [--route id] [--with-learning] [--learning-category kind] [--learning-limit N] [--with-recall] [--recall-limit N] [--max-bytes N] [--strict] [--json]`, `learn [--init|--remember text|--feedback text|--list|--export|--query text|--explain|--recall query|--backup|--redact|--verify|--diff|--restore|--restore-backups [--prune]|--import|--audit [--fix]|--curate|--stats|--usage|--signals [--strict]|--agent-backlog [--strict]|--propose-skills [--min-evidence N] [--review-file path] [--review-check|--apply-plan] [--strict]|--eval-template|--eval [--strict]|--forget id|--clear] [--json|--report|--patch|--review-template] [--out file]`, `check <artifact.md|--stdin|--examples> [--route id|--all-routes] [--issues-only] [--strict] [--learn [--yes] [--learning-file path]] [--json]`, `workspace [--root path] [--learning-file path] [--learning-usage path] [--learning-eval path] [--strict] [--json]`, `site <workspace.json|--stdin> [--strict] [--json|--mcp-check [--probes]|--mcp-plan [--probes] [--json]|--next-actions [--json]|--graph|--tasks|--bundle|--report|--prompts|--prompt id [--task id]] [--out file] | site <bundle-dir> --bundle-check [--json] | site <bundle-dir> --bundle-compare other-bundle-dir [--json] | site <bundle-dir> --bundle-handoff [--task id] [--json] | site <bundle-dir> --bundle-repair [--yes] [--json] [--out file] | site --init --name name --live-url url [--next-actions] [--out file] | site --init --name name --live-url url --bundle --out dir | site --from-intake file.md|--stdin [--json|--next-actions [--json]|--tasks|--bundle [--tasks] --out dir] [--out file] | site --intake-template [--language en|ko] [--json] [--out file] | site --sample [--out file] | site --prompt-list [--json]`, `examples [query|--route id] [--limit N] [--json]`, `search <query|--eval-template|--eval> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider "cmd args"]] [--strict] [--json]`, `index [--build|--status|--verify] [--json] [--embeddings [--provider "cmd args"]]`, `show <file[:line]> [--lines N:M] [--context N] [--json]`, `audit [--strict] [--quiet] [--json]`, `doctor [--strict] [--json] [--fix]`, `mcp`, `version [--json]`, `help [command|--json]`.
|
|
79
79
|
|
|
80
80
|
See [`docs/DISTRIBUTION.md`](docs/DISTRIBUTION.md) for the full distribution guide.
|
|
81
81
|
|
|
@@ -214,7 +214,7 @@ Refresh refs/ on demand: `./tools/extractors/run-all.sh`.
|
|
|
214
214
|
|
|
215
215
|
## Status
|
|
216
216
|
|
|
217
|
-
See [`docs/ROADMAP.md`](docs/ROADMAP.md) for the full phase log and [`docs/PRODUCT-READINESS.md`](docs/PRODUCT-READINESS.md) for the current completion boundary. Currently at **v4.
|
|
217
|
+
See [`docs/ROADMAP.md`](docs/ROADMAP.md) for the full phase log and [`docs/PRODUCT-READINESS.md`](docs/PRODUCT-READINESS.md) for the current completion boundary. Currently at **v4.60.0**: the Agent SDK (`@design-ai/cli/sdk`) with shipped TypeScript types and the opt-in `learn.*` local-write namespace, public npm publish, provenance-backed GitHub Actions release, public registry smoke, Website Console MCP readiness, workspace learning restore/eval coverage, handoff bundle verification, and 90%+ component coverage are complete.
|
|
218
218
|
|
|
219
219
|
Core design consulting workflows are locally release-ready. The website improvement control tower ships as a zero-dependency static Web App plus a `website-improvement` route/skill/command, covering Site Profiles, audit checklists, MCP readiness, refactor prompts, handoff evidence tracking, and bundle export/verify/repair. Local learning preferences are available through `design-ai learn` — profile bootstrap, feedback capture, a read-only signals registry, skill-proposal generation from repeated QA signals, and full backup/restore/curate/audit tooling, all local-only and opt-in. AI model training or fine-tuning remains outside the shipped scope.
|
|
220
220
|
|
package/cli/commands/help.mjs
CHANGED
|
@@ -33,7 +33,7 @@ export const HELP_COMMANDS = [
|
|
|
33
33
|
{ topic: "uninstall", usage: "uninstall [--json]", description: "Remove symlinks (keeps source files)" },
|
|
34
34
|
{ topic: "status", usage: "status [--json]", description: "Show what's installed (use VERBOSE=1 for full list)" },
|
|
35
35
|
{ topic: "list", usage: "list [skills|commands|agents] [--json]", description: "List catalog from the plugin manifest" },
|
|
36
|
-
{ topic: "search", usage: "search <query> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider \"cmd args\"]] [--json]", description: "Search the local markdown corpus" },
|
|
36
|
+
{ topic: "search", usage: "search <query|--eval-template|--eval> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider \"cmd args\"]] [--strict] [--json]", description: "Search the local markdown corpus and ranked-search eval checkpoints" },
|
|
37
37
|
{ topic: "index", usage: "index [--build|--status|--verify] [--json] [--embeddings [--provider \"cmd args\"]]", description: "Build, inspect, and verify the local retrieval index" },
|
|
38
38
|
{ topic: "show", usage: "show <file[:line]> [--lines N:M] [--context N] [--json]", description: "Print a corpus file or line range" },
|
|
39
39
|
{ topic: "route", usage: "route <brief|--from-file file|--stdin|--list|--eval-template|--eval> [--limit N]", description: "Recommend commands, skills, knowledge, and route eval checkpoints" },
|
package/cli/commands/search.mjs
CHANGED
|
@@ -2,13 +2,19 @@
|
|
|
2
2
|
|
|
3
3
|
import { DESIGN_AI_HOME } from "../lib/paths.mjs";
|
|
4
4
|
import { dim, header, info, warn } from "../lib/log.mjs";
|
|
5
|
+
import { resolveBriefInput } from "../lib/brief.mjs";
|
|
5
6
|
import { configuredEmbeddingProvider, defaultConfigFile } from "../lib/local-config.mjs";
|
|
7
|
+
import { buildSearchEvalTemplate, searchEvalReport } from "../lib/search-eval.mjs";
|
|
6
8
|
import { embeddingRerankSearch, rankedSearchCorpus } from "../lib/search-ranked.mjs";
|
|
7
9
|
import { formatSearchJson, parseSearchArgs, searchCorpus } from "../lib/search.mjs";
|
|
8
10
|
|
|
9
11
|
function printHelp() {
|
|
10
|
-
console.log("Usage: design-ai search <query> [--limit N] [--dir kind] [--ranked] [--embeddings [--provider \"cmd args\"]] [--json]
|
|
11
|
-
console.log("
|
|
12
|
+
console.log("Usage: design-ai search <query> [--limit N] [--dir kind] [--ranked] [--embeddings [--provider \"cmd args\"]] [--json]");
|
|
13
|
+
console.log(" design-ai search --eval-template [--json]");
|
|
14
|
+
console.log(" design-ai search --eval --from-file search-eval.json [--strict] [--json]");
|
|
15
|
+
console.log(" cat search-eval.json | design-ai search --eval --stdin [--strict] [--json]\n");
|
|
16
|
+
console.log("Searches markdown files across knowledge, examples, skills, docs, agents, and commands.");
|
|
17
|
+
console.log("Search eval is read-only and checks deterministic ranked-search fixtures.\n");
|
|
12
18
|
console.log("Options:");
|
|
13
19
|
console.log(" --limit N Maximum hits to return, 1-500. Default: 20");
|
|
14
20
|
console.log(" --dir kind Restrict to one corpus directory. Repeatable.");
|
|
@@ -18,7 +24,18 @@ function printHelp() {
|
|
|
18
24
|
console.log(" Requires --ranked and a configured provider; falls back to the");
|
|
19
25
|
console.log(" lexical ranking (with a notice) if no provider or sidecar is available");
|
|
20
26
|
console.log(" --provider \"cmd args\" Embedding provider command for this invocation (overrides config)");
|
|
27
|
+
console.log(" --from-file file Read ranked-search eval JSON from a file (with --eval)");
|
|
28
|
+
console.log(" --stdin Read ranked-search eval JSON from standard input (with --eval)");
|
|
29
|
+
console.log(" --eval-template Generate a runnable ranked-search eval checkpoint JSON template");
|
|
30
|
+
console.log(" --eval Run deterministic ranked-search checkpoint cases");
|
|
31
|
+
console.log(" --strict With --eval, exit non-zero on warning or failure");
|
|
21
32
|
console.log(" --json Emit machine-readable results");
|
|
33
|
+
console.log("");
|
|
34
|
+
console.log("Examples:");
|
|
35
|
+
console.log(" design-ai search Pretendard");
|
|
36
|
+
console.log(" design-ai search \"button component\" --ranked");
|
|
37
|
+
console.log(" design-ai search --eval-template --json > search-eval.json");
|
|
38
|
+
console.log(" design-ai search --eval --from-file search-eval.json --strict --json");
|
|
22
39
|
}
|
|
23
40
|
|
|
24
41
|
function parseProviderFlag(value) {
|
|
@@ -45,6 +62,37 @@ function printRankedHits(query, hits, backend) {
|
|
|
45
62
|
info(`Hits: ${hits.length}`);
|
|
46
63
|
}
|
|
47
64
|
|
|
65
|
+
function printSearchEvalTemplate(template) {
|
|
66
|
+
header("design-ai search", "Ranked-search eval checkpoint template");
|
|
67
|
+
info(`Source: ${DESIGN_AI_HOME}`);
|
|
68
|
+
info(`Cases: ${template.cases.length}`);
|
|
69
|
+
console.log();
|
|
70
|
+
console.log("Write the JSON below to a file, edit cases if needed, then run:");
|
|
71
|
+
console.log("design-ai search --eval --from-file search-eval.json --strict");
|
|
72
|
+
console.log();
|
|
73
|
+
console.log(formatSearchJson(template));
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
function printSearchEvalReport(report) {
|
|
77
|
+
header("design-ai search", "Ranked-search eval report");
|
|
78
|
+
info(`Source: ${report.source}`);
|
|
79
|
+
info(`Status: ${report.status}`);
|
|
80
|
+
info(`Cases: ${report.summary.total} (${report.summary.pass} pass, ${report.summary.warn} warn, ${report.summary.fail} fail)`);
|
|
81
|
+
console.log();
|
|
82
|
+
|
|
83
|
+
for (const result of report.cases) {
|
|
84
|
+
const top = result.topRelPath || "none";
|
|
85
|
+
console.log(`${result.status.toUpperCase()} ${result.id}`);
|
|
86
|
+
console.log(` query: ${result.query}`);
|
|
87
|
+
console.log(` top: ${top} ${dim(`(hits ${result.hitCount})`)}`);
|
|
88
|
+
if (result.matchedTokens.length > 0) {
|
|
89
|
+
console.log(` matched: ${result.matchedTokens.join(", ")}`);
|
|
90
|
+
}
|
|
91
|
+
console.log(` result: ${result.message}`);
|
|
92
|
+
console.log();
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
|
|
48
96
|
export async function runSearch(args) {
|
|
49
97
|
const parsed = parseSearchArgs(args);
|
|
50
98
|
if (parsed.help) {
|
|
@@ -52,6 +100,37 @@ export async function runSearch(args) {
|
|
|
52
100
|
return;
|
|
53
101
|
}
|
|
54
102
|
|
|
103
|
+
if (parsed.evalTemplate) {
|
|
104
|
+
const template = buildSearchEvalTemplate({ sourceRoot: DESIGN_AI_HOME });
|
|
105
|
+
if (parsed.json) {
|
|
106
|
+
console.log(formatSearchJson(template));
|
|
107
|
+
return;
|
|
108
|
+
}
|
|
109
|
+
printSearchEvalTemplate(template);
|
|
110
|
+
return;
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
if (parsed.eval) {
|
|
114
|
+
const evalText = resolveBriefInput(parsed);
|
|
115
|
+
const report = searchEvalReport({
|
|
116
|
+
evalText,
|
|
117
|
+
source: parsed.fromFile || "stdin",
|
|
118
|
+
sourceRoot: DESIGN_AI_HOME,
|
|
119
|
+
limit: parsed.limit,
|
|
120
|
+
});
|
|
121
|
+
|
|
122
|
+
if (parsed.json) {
|
|
123
|
+
console.log(formatSearchJson(report));
|
|
124
|
+
} else {
|
|
125
|
+
printSearchEvalReport(report);
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
if (parsed.strict && report.status !== "pass") {
|
|
129
|
+
process.exitCode = 1;
|
|
130
|
+
}
|
|
131
|
+
return;
|
|
132
|
+
}
|
|
133
|
+
|
|
55
134
|
if (parsed.embeddings && !parsed.ranked) {
|
|
56
135
|
throw new Error("--embeddings requires --ranked");
|
|
57
136
|
}
|
|
@@ -0,0 +1,241 @@
|
|
|
1
|
+
// Ranked-search eval checkpoint generation and reports for `design-ai search`.
|
|
2
|
+
// Mirrors cli/lib/route.mjs's eval-checkpoint pattern (docs/AI-LEARNING-PHASE2.md
|
|
3
|
+
// FU-3): runs hand-authored / generated cases through the shipped deterministic
|
|
4
|
+
// ranked search path (rankedSearchCorpus, BM25-style lexical scorer) and reports
|
|
5
|
+
// pass/warn/fail. This is the standing evidence artifact for any future decision
|
|
6
|
+
// to promote `--ranked` to the default `search` behavior.
|
|
7
|
+
|
|
8
|
+
import { rankedSearchCorpus } from "./search-ranked.mjs";
|
|
9
|
+
import { DEFAULT_SEARCH_DIRS } from "./search.mjs";
|
|
10
|
+
|
|
11
|
+
export const SEARCH_EVAL_VERSION = 1;
|
|
12
|
+
// Default per-case result limit when a case does not specify its own `limit`.
|
|
13
|
+
// Kept small and eval-appropriate (route's default is 3; search corpus hits are
|
|
14
|
+
// noisier, so 10 gives enough headroom for `minHits` assertions without needing
|
|
15
|
+
// a large recall window).
|
|
16
|
+
export const SEARCH_EVAL_DEFAULT_LIMIT = 10;
|
|
17
|
+
|
|
18
|
+
function isoTimestamp(now = new Date()) {
|
|
19
|
+
return (now instanceof Date ? now : new Date(now)).toISOString();
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
export function buildSearchEvalTemplate({ sourceRoot, generatedAt = new Date() } = {}) {
|
|
23
|
+
return {
|
|
24
|
+
version: SEARCH_EVAL_VERSION,
|
|
25
|
+
generatedAt: isoTimestamp(generatedAt),
|
|
26
|
+
description: "Deterministic ranked-search checkpoints for design-ai's BM25-style lexical retrieval.",
|
|
27
|
+
cases: [
|
|
28
|
+
{
|
|
29
|
+
id: "korean-button-stem",
|
|
30
|
+
query: "버튼",
|
|
31
|
+
minHits: 1,
|
|
32
|
+
},
|
|
33
|
+
{
|
|
34
|
+
id: "korean-button-particle",
|
|
35
|
+
query: "버튼을",
|
|
36
|
+
minHits: 1,
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
id: "korean-accessibility-particle",
|
|
40
|
+
query: "접근성이",
|
|
41
|
+
minHits: 1,
|
|
42
|
+
},
|
|
43
|
+
{
|
|
44
|
+
id: "accessibility-english",
|
|
45
|
+
query: "accessibility",
|
|
46
|
+
minHits: 5,
|
|
47
|
+
matchedTokenIncludes: ["accessibility"],
|
|
48
|
+
},
|
|
49
|
+
{
|
|
50
|
+
id: "button-component-spec-english",
|
|
51
|
+
query: "button component spec",
|
|
52
|
+
minHits: 5,
|
|
53
|
+
matchedTokenIncludes: ["button", "component", "spec"],
|
|
54
|
+
},
|
|
55
|
+
{
|
|
56
|
+
id: "color-palette-english",
|
|
57
|
+
query: "color palette",
|
|
58
|
+
expectRelPathIn: ["commands/palette-from-brand.md"],
|
|
59
|
+
minHits: 5,
|
|
60
|
+
matchedTokenIncludes: ["color", "palette"],
|
|
61
|
+
},
|
|
62
|
+
],
|
|
63
|
+
};
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
function searchEvalStatus(counts) {
|
|
67
|
+
if (counts.fail > 0) return "fail";
|
|
68
|
+
if (counts.warn > 0) return "warn";
|
|
69
|
+
return "pass";
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
export function normalizeSearchEvalPayload(evalText, source = "search-eval.json") {
|
|
73
|
+
let payload;
|
|
74
|
+
try {
|
|
75
|
+
payload = JSON.parse(evalText);
|
|
76
|
+
} catch (err) {
|
|
77
|
+
throw new Error(`Could not parse search eval JSON from ${source}: ${err.message}`);
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
if (!payload || typeof payload !== "object" || Array.isArray(payload)) {
|
|
81
|
+
throw new Error("Search eval payload must be a JSON object");
|
|
82
|
+
}
|
|
83
|
+
if (payload.version !== SEARCH_EVAL_VERSION) {
|
|
84
|
+
throw new Error(`Search eval payload version must be ${SEARCH_EVAL_VERSION}`);
|
|
85
|
+
}
|
|
86
|
+
if (!Array.isArray(payload.cases)) {
|
|
87
|
+
throw new Error("Search eval payload must include a cases array");
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
return payload;
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
function normalizeStringList(value, { field, id }) {
|
|
94
|
+
if (value === undefined || value === null) return [];
|
|
95
|
+
if (!Array.isArray(value)) {
|
|
96
|
+
throw new Error(`Search eval case ${id} field ${field} must be an array`);
|
|
97
|
+
}
|
|
98
|
+
return value.map((item) => String(item || "").trim()).filter(Boolean);
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
export function normalizeSearchEvalCase(rawCase, index) {
|
|
102
|
+
if (!rawCase || typeof rawCase !== "object" || Array.isArray(rawCase)) {
|
|
103
|
+
throw new Error(`Search eval case ${index + 1} must be a JSON object`);
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
const id = String(rawCase.id || `case-${index + 1}`).trim();
|
|
107
|
+
if (!id) throw new Error(`Search eval case ${index + 1} is missing id`);
|
|
108
|
+
|
|
109
|
+
const query = String(rawCase.query || "").trim();
|
|
110
|
+
if (!query) throw new Error(`Search eval case ${id} is missing query`);
|
|
111
|
+
|
|
112
|
+
const dirs = normalizeStringList(rawCase.dirs, { field: "dirs", id });
|
|
113
|
+
for (const dir of dirs) {
|
|
114
|
+
if (!DEFAULT_SEARCH_DIRS.includes(dir)) {
|
|
115
|
+
throw new Error(`Search eval case ${id} field dirs has invalid value: ${dir}`);
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
const limit = rawCase.limit === undefined || rawCase.limit === null
|
|
120
|
+
? SEARCH_EVAL_DEFAULT_LIMIT
|
|
121
|
+
: Number(rawCase.limit);
|
|
122
|
+
if (!Number.isInteger(limit) || limit < 1 || limit > 500) {
|
|
123
|
+
throw new Error(`Search eval case ${id} limit must be an integer from 1 to 500`);
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
const expectRelPathIn = normalizeStringList(rawCase.expectRelPathIn, { field: "expectRelPathIn", id });
|
|
127
|
+
const matchedTokenIncludes = normalizeStringList(rawCase.matchedTokenIncludes, { field: "matchedTokenIncludes", id })
|
|
128
|
+
.map((token) => token.toLowerCase());
|
|
129
|
+
|
|
130
|
+
let minHits;
|
|
131
|
+
if (rawCase.minHits === undefined || rawCase.minHits === null) {
|
|
132
|
+
minHits = undefined;
|
|
133
|
+
} else {
|
|
134
|
+
minHits = Number(rawCase.minHits);
|
|
135
|
+
if (!Number.isInteger(minHits) || minHits < 0) {
|
|
136
|
+
throw new Error(`Search eval case ${id} field minHits must be a non-negative integer`);
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
return {
|
|
141
|
+
id,
|
|
142
|
+
query,
|
|
143
|
+
dirs: dirs.length > 0 ? dirs : DEFAULT_SEARCH_DIRS,
|
|
144
|
+
limit,
|
|
145
|
+
expectRelPathIn,
|
|
146
|
+
minHits,
|
|
147
|
+
matchedTokenIncludes,
|
|
148
|
+
};
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
export function evaluateSearchEvalCase(testCase, sourceRoot, defaultLimit) {
|
|
152
|
+
const limit = testCase.limit || defaultLimit || SEARCH_EVAL_DEFAULT_LIMIT;
|
|
153
|
+
const { hits } = rankedSearchCorpus({
|
|
154
|
+
query: testCase.query,
|
|
155
|
+
designAiPath: sourceRoot,
|
|
156
|
+
dirs: testCase.dirs,
|
|
157
|
+
limit,
|
|
158
|
+
});
|
|
159
|
+
const topHit = hits[0] || null;
|
|
160
|
+
const topRelPath = topHit?.relPath || "";
|
|
161
|
+
const topMatchedTokens = (topHit?.matchedTokens || []).map((token) => token.toLowerCase());
|
|
162
|
+
|
|
163
|
+
const hasAssertions = testCase.expectRelPathIn.length > 0
|
|
164
|
+
|| testCase.minHits !== undefined
|
|
165
|
+
|| testCase.matchedTokenIncludes.length > 0;
|
|
166
|
+
|
|
167
|
+
let status = "pass";
|
|
168
|
+
let message = "All checkpoint expectations were satisfied.";
|
|
169
|
+
|
|
170
|
+
if (!hasAssertions) {
|
|
171
|
+
status = "warn";
|
|
172
|
+
message = "Case has no expectRelPathIn, minHits, or matchedTokenIncludes assertion.";
|
|
173
|
+
} else if (testCase.expectRelPathIn.length > 0 && !testCase.expectRelPathIn.includes(topRelPath)) {
|
|
174
|
+
status = "fail";
|
|
175
|
+
message = topRelPath
|
|
176
|
+
? `Expected top hit in [${testCase.expectRelPathIn.join(", ")}], but top hit was ${topRelPath}.`
|
|
177
|
+
: `Expected top hit in [${testCase.expectRelPathIn.join(", ")}], but there were no hits.`;
|
|
178
|
+
} else if (testCase.minHits !== undefined && hits.length < testCase.minHits) {
|
|
179
|
+
status = "fail";
|
|
180
|
+
message = `Expected at least ${testCase.minHits} hit(s), got ${hits.length}.`;
|
|
181
|
+
} else if (
|
|
182
|
+
testCase.matchedTokenIncludes.length > 0
|
|
183
|
+
&& !testCase.matchedTokenIncludes.every((token) => topMatchedTokens.includes(token))
|
|
184
|
+
) {
|
|
185
|
+
status = "fail";
|
|
186
|
+
const missing = testCase.matchedTokenIncludes.filter((token) => !topMatchedTokens.includes(token));
|
|
187
|
+
message = topHit
|
|
188
|
+
? `Top hit ${topRelPath} did not match expected token(s): ${missing.join(", ")}.`
|
|
189
|
+
: `Expected matched tokens [${missing.join(", ")}], but there were no hits.`;
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
return {
|
|
193
|
+
id: testCase.id,
|
|
194
|
+
status,
|
|
195
|
+
message,
|
|
196
|
+
query: testCase.query,
|
|
197
|
+
dirs: testCase.dirs,
|
|
198
|
+
limit,
|
|
199
|
+
topRelPath,
|
|
200
|
+
hitCount: hits.length,
|
|
201
|
+
matchedTokens: topHit?.matchedTokens || [],
|
|
202
|
+
expectRelPathIn: testCase.expectRelPathIn,
|
|
203
|
+
minHits: testCase.minHits ?? null,
|
|
204
|
+
matchedTokenIncludes: testCase.matchedTokenIncludes,
|
|
205
|
+
hits,
|
|
206
|
+
};
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
export function searchEvalReport({
|
|
210
|
+
evalText,
|
|
211
|
+
source = "search-eval.json",
|
|
212
|
+
sourceRoot,
|
|
213
|
+
limit = SEARCH_EVAL_DEFAULT_LIMIT,
|
|
214
|
+
generatedAt = new Date(),
|
|
215
|
+
}) {
|
|
216
|
+
const payload = normalizeSearchEvalPayload(evalText, source);
|
|
217
|
+
const normalizedCases = payload.cases.map((testCase, index) => normalizeSearchEvalCase(testCase, index));
|
|
218
|
+
const results = normalizedCases.map((testCase) => evaluateSearchEvalCase(testCase, sourceRoot, limit));
|
|
219
|
+
const counts = results.reduce(
|
|
220
|
+
(acc, result) => ({
|
|
221
|
+
...acc,
|
|
222
|
+
[result.status]: acc[result.status] + 1,
|
|
223
|
+
}),
|
|
224
|
+
{ pass: 0, warn: 0, fail: 0 },
|
|
225
|
+
);
|
|
226
|
+
const summary = {
|
|
227
|
+
total: results.length,
|
|
228
|
+
pass: counts.pass,
|
|
229
|
+
warn: counts.warn,
|
|
230
|
+
fail: counts.fail,
|
|
231
|
+
};
|
|
232
|
+
|
|
233
|
+
return {
|
|
234
|
+
evalVersion: payload.version,
|
|
235
|
+
source,
|
|
236
|
+
generatedAt: isoTimestamp(generatedAt),
|
|
237
|
+
status: searchEvalStatus(counts),
|
|
238
|
+
summary,
|
|
239
|
+
cases: results,
|
|
240
|
+
};
|
|
241
|
+
}
|
package/cli/lib/search.mjs
CHANGED
|
@@ -7,6 +7,7 @@ import {
|
|
|
7
7
|
} from "node:fs";
|
|
8
8
|
import path from "node:path";
|
|
9
9
|
|
|
10
|
+
import { parseBriefSourceFlag } from "./brief.mjs";
|
|
10
11
|
import { expectedValueMessage, unknownOptionMessage } from "./suggest.mjs";
|
|
11
12
|
|
|
12
13
|
export const DEFAULT_SEARCH_DIRS = [
|
|
@@ -20,7 +21,21 @@ export const DEFAULT_SEARCH_DIRS = [
|
|
|
20
21
|
|
|
21
22
|
const PREVIEW_LEN = 120;
|
|
22
23
|
const PREVIEW_BEFORE = 50;
|
|
23
|
-
const SEARCH_OPTIONS = [
|
|
24
|
+
const SEARCH_OPTIONS = [
|
|
25
|
+
"-h",
|
|
26
|
+
"--help",
|
|
27
|
+
"--json",
|
|
28
|
+
"--limit",
|
|
29
|
+
"--dir",
|
|
30
|
+
"--ranked",
|
|
31
|
+
"--embeddings",
|
|
32
|
+
"--provider",
|
|
33
|
+
"--eval-template",
|
|
34
|
+
"--eval",
|
|
35
|
+
"--strict",
|
|
36
|
+
"--from-file",
|
|
37
|
+
"--stdin",
|
|
38
|
+
];
|
|
24
39
|
|
|
25
40
|
function exists(p) {
|
|
26
41
|
try {
|
|
@@ -118,11 +133,17 @@ export function parseSearchArgs(args) {
|
|
|
118
133
|
ranked: false,
|
|
119
134
|
embeddings: false,
|
|
120
135
|
provider: "",
|
|
136
|
+
evalTemplate: false,
|
|
137
|
+
eval: false,
|
|
138
|
+
strict: false,
|
|
139
|
+
fromFile: "",
|
|
140
|
+
stdin: false,
|
|
121
141
|
help: false,
|
|
122
142
|
};
|
|
123
143
|
|
|
124
144
|
for (let i = 0; i < args.length; i += 1) {
|
|
125
145
|
const arg = args[i];
|
|
146
|
+
out.index = i;
|
|
126
147
|
if (arg === "-h" || arg === "--help") {
|
|
127
148
|
out.help = true;
|
|
128
149
|
} else if (arg === "--json") {
|
|
@@ -131,9 +152,17 @@ export function parseSearchArgs(args) {
|
|
|
131
152
|
out.ranked = true;
|
|
132
153
|
} else if (arg === "--embeddings") {
|
|
133
154
|
out.embeddings = true;
|
|
155
|
+
} else if (arg === "--eval-template") {
|
|
156
|
+
out.evalTemplate = true;
|
|
157
|
+
} else if (arg === "--eval") {
|
|
158
|
+
out.eval = true;
|
|
159
|
+
} else if (arg === "--strict") {
|
|
160
|
+
out.strict = true;
|
|
134
161
|
} else if (arg === "--provider") {
|
|
135
162
|
out.provider = args[i + 1] || "";
|
|
136
163
|
i += 1;
|
|
164
|
+
} else if (parseBriefSourceFlag(args, out)) {
|
|
165
|
+
i = out.index;
|
|
137
166
|
} else if (arg === "--limit") {
|
|
138
167
|
const next = args[i + 1];
|
|
139
168
|
const limit = Number(next);
|
|
@@ -156,8 +185,34 @@ export function parseSearchArgs(args) {
|
|
|
156
185
|
}
|
|
157
186
|
}
|
|
158
187
|
|
|
188
|
+
if (out.eval && out.evalTemplate) {
|
|
189
|
+
throw new Error("Choose either --eval-template or --eval, not both");
|
|
190
|
+
}
|
|
191
|
+
if (out.strict && !out.eval) {
|
|
192
|
+
throw new Error("--strict can only be used with --eval");
|
|
193
|
+
}
|
|
194
|
+
if (
|
|
195
|
+
out.evalTemplate
|
|
196
|
+
&& (out.fromFile || out.stdin || out.queryParts.length > 0 || out.ranked || out.embeddings || out.provider || out.dirs.length > 0)
|
|
197
|
+
) {
|
|
198
|
+
throw new Error("--eval-template cannot be combined with a query, --from-file, --stdin, --ranked, --embeddings, --provider, or --dir");
|
|
199
|
+
}
|
|
200
|
+
if (out.eval && (!out.fromFile && !out.stdin)) {
|
|
201
|
+
throw new Error("--eval requires --from-file or --stdin");
|
|
202
|
+
}
|
|
203
|
+
if ((out.fromFile || out.stdin) && !out.eval) {
|
|
204
|
+
throw new Error("--from-file and --stdin require --eval");
|
|
205
|
+
}
|
|
206
|
+
if (
|
|
207
|
+
out.eval
|
|
208
|
+
&& (out.queryParts.length > 0 || out.ranked || out.embeddings || out.provider || out.dirs.length > 0)
|
|
209
|
+
) {
|
|
210
|
+
throw new Error("--eval cannot be combined with an inline query, --ranked, --embeddings, --provider, or --dir");
|
|
211
|
+
}
|
|
212
|
+
|
|
159
213
|
return {
|
|
160
214
|
...out,
|
|
215
|
+
index: undefined,
|
|
161
216
|
query: out.queryParts.join(" ").trim(),
|
|
162
217
|
dirs: out.dirs.length > 0 ? out.dirs : DEFAULT_SEARCH_DIRS,
|
|
163
218
|
};
|
|
@@ -202,8 +202,8 @@ This review answers the five open questions against the shipped Phase A implemen
|
|
|
202
202
|
|
|
203
203
|
- **FU-1 (Q4, before Phase B):** Add Hangul-aware tokenization (CJK bigramming or particle stripping) in `cli/lib/lexical.mjs` behind Korean `learn --eval` checkpoints; regression-test that `버튼`, `버튼을`, `버튼이` converge on the same button docs. _Done (2026-07-03): Hangul runs >= 2 chars now emit overlapping character bigrams alongside the surface form; the review's zero-hit queries recover (`버튼` 0 → 3 ranked hits, `접근성이` 0 → 3), with unit regression coverage in `lexical.test.mjs`._
|
|
204
204
|
- **FU-2 (Q2, before Phase B index-as-source-of-truth):** Key the corpus index by corpus digest / record `designAiPath` in the payload so multiple checkouts do not overwrite each other once the index is read for content rather than staleness. _Done (2026-07-03): sidecar format bumped to version 2 (auto-invalidating v1 files); the corpus payload records resolved `designAiPath` and the learning payload the resolved `learningFile`, and `index --status` reports `sourceMatch` and treats identity mismatch as not fresh._
|
|
205
|
-
- **FU-3 (Q1, gates default promotion):** Land a ranked-search eval checkpoint so any future `--ranked` default promotion is evidence-backed and announced.
|
|
206
|
-
- **FU-4 (Q5, Phase B):** Specify and implement `~/.design-ai/config.json` as the Phase B provider config home with per-invocation `--embeddings` still required.
|
|
205
|
+
- **FU-3 (Q1, gates default promotion):** Land a ranked-search eval checkpoint so any future `--ranked` default promotion is evidence-backed and announced. _Done (2026-07-05): `design-ai search --eval-template`/`--eval [--strict]` landed in `cli/lib/search-eval.mjs` and `cli/commands/search.mjs`, mirroring the route/prompt/pack/learn eval-checkpoint pattern; cases run through `rankedSearchCorpus` (the shipped ranked path), and the default template includes the three Korean particle-form regression cases from FU-1 (버튼/버튼을/접근성이) plus English cases, doubling as a retrieval-level Hangul-bigram regression check._
|
|
206
|
+
- **FU-4 (Q5, Phase B):** Specify and implement `~/.design-ai/config.json` as the Phase B provider config home with per-invocation `--embeddings` still required. _Done (2026-07-03, Phase B): `cli/lib/local-config.mjs` reads `~/.design-ai/config.json` (or `DESIGN_AI_CONFIG_FILE`); the config only supplies the provider — every invocation still requires the explicit `--embeddings` flag to arm it._
|
|
207
207
|
|
|
208
208
|
### Phase B gate: cleared-with-conditions
|
|
209
209
|
|
|
@@ -118,8 +118,17 @@ Cross-check any surface work against the release gate: `npm run audit` (8/8, lin
|
|
|
118
118
|
|
|
119
119
|
## Decision record
|
|
120
120
|
|
|
121
|
-
- **Status:**
|
|
122
|
-
- **Date:** 2026-07-03
|
|
121
|
+
- **Status:** accepted and executed — both recommended surfaces shipped
|
|
122
|
+
- **Date:** 2026-07-03 (proposed) / 2026-07-05 (executed)
|
|
123
123
|
- **Deciders:** maintainer
|
|
124
124
|
- **Supersedes:** none
|
|
125
125
|
- **Related:** [`PRODUCT-READINESS.md`](PRODUCT-READINESS.md), [`external-status.md`](external-status.md), `AI-LEARNING-PHASE2.md`
|
|
126
|
+
|
|
127
|
+
### Outcome (2026-07-05)
|
|
128
|
+
|
|
129
|
+
The recommendation was executed in order and both surfaces are live:
|
|
130
|
+
|
|
131
|
+
- **CLI deepening** shipped as v4.57.0 (local retrieval memory: `index`, `search --ranked`, `--with-recall`, `learn --recall`, opt-in embedding rerank) and v4.58.0 (`route --explain` related knowledge, recall quality, corpus depth). The dogfood friction this document identified — missing corpus and missing flags — is retired; retrieval spans every CLI/MCP surface.
|
|
132
|
+
- **Agent SDK** followed exactly on the stated revisit trigger (recall interface stable across two releases): design doc [`AGENT-SDK.md`](AGENT-SDK.md), then v4.59.0 (Phase A — read-only `@design-ai/cli/sdk`, 8 verbs) and v4.60.0 (TypeScript declarations + the opt-in `learn.*` local-write namespace).
|
|
133
|
+
|
|
134
|
+
The deferred surfaces (VS Code deepening, Web UI, Figma plugin) remain deferred with the same revisit triggers. The next-surface question is open again; re-evaluate against the criteria above when the next planning window opens, factoring in that the SDK now also serves programmatic adopters.
|
package/docs/ROADMAP.md
CHANGED
|
@@ -1,5 +1,42 @@
|
|
|
1
1
|
# Roadmap
|
|
2
2
|
|
|
3
|
+
## Phase 763 — Ranked-search Eval Checkpoint Release (v4.61.0) ✓ ready
|
|
4
|
+
|
|
5
|
+
Ships the FU-3 ranked-search eval checkpoint (Phase 762) as an npm version: `search --eval-template` / `search --eval [--strict]` over the shipped BM25 path, with the Korean particle-form regression cases in the default template. Additive and backward-compatible; default `search` behavior is unchanged. With this release the AI-learning Phase A follow-ups (FU-1–FU-4) are all closed and published.
|
|
6
|
+
|
|
7
|
+
### Verified
|
|
8
|
+
- All 8 audits passed.
|
|
9
|
+
- `npm run release:check` (unit tests incl. search-eval and search-command suites, strict audits, whitespace, package contents, release metadata, release self-tests, packed-tarball smoke).
|
|
10
|
+
- `npm run release:metadata`.
|
|
11
|
+
- `git diff --check`.
|
|
12
|
+
- Main-branch GitHub Actions (`Design-AI audit`, `Deploy doc site`) passed for the constituent commits.
|
|
13
|
+
|
|
14
|
+
### Versions
|
|
15
|
+
- `package.json` + `.claude-plugin/plugin.json`: 4.60.0 → 4.61.0.
|
|
16
|
+
- `vscode-extension/package.json`: remains 0.4.1.
|
|
17
|
+
|
|
18
|
+
### What this enables
|
|
19
|
+
- The `--ranked` default-promotion decision becomes evidence-backed for npm users too: the published CLI carries the checkpoint tooling, so the eval can be re-run against any installed version — including Korean agglutinative-query regressions — before and after such a change.
|
|
20
|
+
|
|
21
|
+
### What's still ahead
|
|
22
|
+
- The `--ranked` default-promotion decision itself (separate, evidence-backed call).
|
|
23
|
+
- Next-surface re-evaluation per the updated decision record in [NEXT-SURFACE-DECISION.md](NEXT-SURFACE-DECISION.md).
|
|
24
|
+
|
|
25
|
+
## Phase 762 — Ranked-search eval checkpoint (FU-3, implemented, unreleased)
|
|
26
|
+
|
|
27
|
+
Closes the last open follow-up from the AI-learning Phase A review ([AI-LEARNING-PHASE2.md](AI-LEARNING-PHASE2.md), FU-3): `design-ai search` gains `--eval-template` / `--eval [--strict]` checkpoint modes mirroring the route/prompt/pack/learn eval pattern, run through the shipped ranked (BM25) search path. This is the standing evidence artifact for any future decision to promote `--ranked` to the default; with FU-1/FU-2/FU-4 already annotated done, all four Phase A follow-ups are now closed.
|
|
28
|
+
|
|
29
|
+
### Delivered
|
|
30
|
+
- [x] `cli/lib/search-eval.mjs` — `SEARCH_EVAL_VERSION = 1`, `buildSearchEvalTemplate()`, `searchEvalReport()` over `rankedSearchCorpus`; route-style payload normalization and error messages. Case schema: `id`, `query`, optional `dirs`/`limit`, assertions `expectRelPathIn` / `minHits` / `matchedTokenIncludes`.
|
|
31
|
+
- [x] Default template includes the FU-1 Korean particle-form regression cases (`버튼`, `버튼을`, `접근성이`) plus three English cases — so the eval doubles as the retrieval-level Hangul-bigram regression. All six verified passing against the shipped corpus (`--strict` exit 0).
|
|
32
|
+
- [x] CLI wiring in `cli/lib/search.mjs` / `cli/commands/search.mjs` (`--eval-template`, `--eval`, `--strict`, `--from-file`, `--stdin`, mutual-exclusivity validation), help topic usage line, and the shared `smoke_assertions.py` pins (4 spots — shared constants, so both package and registry smokes stay aligned).
|
|
33
|
+
- [x] Tests: `search-eval.test.mjs` (template, pass/warn/fail reports, malformed payloads) and `search-command.test.mjs` (CLI-level, strict exit semantics). 578/578 total.
|
|
34
|
+
- [x] `docs/AI-LEARNING-PHASE2.md` FU-3 and FU-4 done annotations.
|
|
35
|
+
|
|
36
|
+
### Remaining
|
|
37
|
+
- [ ] Release: bundle into the next npm version.
|
|
38
|
+
- [ ] The `--ranked` default-promotion decision itself — a future, separate call that this checkpoint now makes evidence-backed.
|
|
39
|
+
|
|
3
40
|
## Phase 761 — Agent SDK types + Phase B Release (v4.60.0) ✓ ready
|
|
4
41
|
|
|
5
42
|
Ships the Agent SDK TypeScript declarations (Phase 759) and the `learn.*` local-write namespace (Phase 760) together as one npm version. Both are additive and backward-compatible: the eight Phase A verbs stay read-only and unchanged, and TypeScript types resolve automatically for `@design-ai/cli/sdk` with no build step on either side.
|
package/docs/external-status.md
CHANGED
|
@@ -1,22 +1,22 @@
|
|
|
1
1
|
# External Publication Status
|
|
2
2
|
|
|
3
|
-
> Checked: 2026-07-
|
|
3
|
+
> Checked: 2026-07-05
|
|
4
4
|
> Scope: npm registry, GitHub Pages, Homebrew tap, VS Code Marketplace, Claude/Codex MCP
|
|
5
5
|
|
|
6
6
|
## Summary
|
|
7
7
|
|
|
8
|
-
npm is publicly published at `@design-ai/cli@4.
|
|
8
|
+
npm is publicly published at `@design-ai/cli@4.60.0` (npm dist-tag `latest` = `4.60.0`); publish run from tag `v4.60.0` succeeded with provenance and all pre-publish gates green. This release extends the Agent SDK (`@design-ai/cli/sdk`) with shipped TypeScript declarations (via an `exports` `types` condition) and the opt-in `learn.*` local-write namespace; the eight read-only verbs are unchanged, so CLI output is unchanged. The live `npm run registry:smoke` passes cleanly against published `@design-ai/cli@4.60.0` ("Registry smoke passed"), covering the retrieval surfaces (index/ranked/embeddings/recall) and route enrichment. GitHub Release `v4.60.0` is published, and the Homebrew tap formula points at the `v4.60.0` release source tarball with a verified SHA-256. The VS Code extension `sungjin.design-ai-vscode` remains published at `0.4.1`. GitHub Pages docs are publicly reachable.
|
|
9
9
|
|
|
10
10
|
## Results
|
|
11
11
|
|
|
12
12
|
| Surface | Checked target | Result | Evidence |
|
|
13
13
|
|---|---|---|---|
|
|
14
|
-
| npm registry | `@design-ai/cli` | Published latest is `4.
|
|
14
|
+
| npm registry | `@design-ai/cli` | Published latest is `4.60.0` (tag `v4.60.0`, provenance). Pre-publish packed-tarball smoke (incl. SDK import + `learn.remember` write) and live `npm run registry:smoke` both pass for `@design-ai/cli@4.60.0`. | `npm view @design-ai/cli version` → `4.60.0`; live registry smoke "Registry smoke passed" |
|
|
15
15
|
| GitHub Pages | `https://sungjin9288.github.io/design-ai/` | Published and reachable: HTTP `200`, design-ai MkDocs page rendered | `evidence/cli-logs/github-pages-status.log` |
|
|
16
|
-
| Homebrew tap | `Formula/design-ai.rb` | Formula pinned to `v4.
|
|
16
|
+
| Homebrew tap | `Formula/design-ai.rb` | Formula pinned to `v4.60.0` release source tarball with SHA-256 `fa34af51c32d8e59dba4c1d293746688820e85ae03efa3b0bae4f3b3b2a34583` (recomputed from the published tag tarball) | `Formula/design-ai.rb` |
|
|
17
17
|
| VS Code Marketplace | `sungjin.design-ai-vscode` | Published: run `28431571256` published `v0.4.1`, and the Marketplace Gallery API returned visible version `0.4.1` on 2026-07-02. | `evidence/cli-logs/vscode-marketplace-status.log`, `evidence/cli-logs/vscode-marketplace-secret-status.log`, `evidence/cli-logs/vscode-extension-vsce-package.log`, `evidence/cli-logs/vscode-publish-workflow-status.log` |
|
|
18
|
-
| GitHub Release | `v4.
|
|
19
|
-
| MCP server | `@design-ai/cli@4.
|
|
18
|
+
| GitHub Release | `v4.60.0` | Published for tag `v4.60.0` at commit `2d73c42` | `gh release view v4.60.0` |
|
|
19
|
+
| MCP server | `@design-ai/cli@4.60.0` / local clone | Public npm `design-ai-mcp` responds to initialize and tools/list with 10 tools; local Codex and Claude Code both report `design-ai` MCP as configured and connected. | `evidence/cli-logs/npm-registry-smoke.log`, `evidence/cli-logs/design-ai-mcp-client-status.log` |
|
|
20
20
|
|
|
21
21
|
## Interpretation
|
|
22
22
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@design-ai/cli",
|
|
3
|
-
"version": "4.
|
|
3
|
+
"version": "4.61.0",
|
|
4
4
|
"description": "Senior product designer for any AI coding agent. Installs design-ai (20 skills, 17 commands, 4 agents) into Claude Code globally. Korean market depth plus website improvement control tower.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"design-ai": "cli/bin/design-ai.mjs",
|
|
@@ -160,7 +160,7 @@ EXPECTED_HELP_TOPIC_USAGES = {
|
|
|
160
160
|
"uninstall": "design-ai uninstall [--json]",
|
|
161
161
|
"status": "design-ai status [--json]",
|
|
162
162
|
"list": "design-ai list [skills|commands|agents] [--json]",
|
|
163
|
-
"search": "design-ai search <query> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider \"cmd args\"]] [--json]",
|
|
163
|
+
"search": "design-ai search <query|--eval-template|--eval> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider \"cmd args\"]] [--strict] [--json]",
|
|
164
164
|
"index": "design-ai index [--build|--status|--verify] [--json] [--embeddings [--provider \"cmd args\"]]",
|
|
165
165
|
"show": "design-ai show <file[:line]> [--lines N:M] [--context N] [--json]",
|
|
166
166
|
"route": "design-ai route <brief|--from-file file|--stdin|--list|--eval-template|--eval> [--limit N]",
|
|
@@ -184,7 +184,12 @@ EXPECTED_HELP_TOPIC_FRAGMENTS = {
|
|
|
184
184
|
"uninstall": ("Usage:", "design-ai uninstall [--json]"),
|
|
185
185
|
"status": ("Usage:", "design-ai status [--json]"),
|
|
186
186
|
"list": ("Usage:", "design-ai list [skills|commands|agents]"),
|
|
187
|
-
"search": (
|
|
187
|
+
"search": (
|
|
188
|
+
"Usage:",
|
|
189
|
+
"design-ai search <query> [--limit N] [--dir kind] [--ranked] [--embeddings [--provider \"cmd args\"]] [--json]",
|
|
190
|
+
"design-ai search --eval-template [--json]",
|
|
191
|
+
"design-ai search --eval --from-file search-eval.json [--strict] [--json]",
|
|
192
|
+
),
|
|
188
193
|
"index": ("Usage:", "design-ai index [--build|--status|--verify] [--json] [--embeddings [--provider \"cmd args\"]]"),
|
|
189
194
|
"show": ("Usage:", "design-ai show <file[:line|start-end]> [--lines N:M] [--context N] [--json]"),
|
|
190
195
|
"route": ("Usage:", "design-ai route <brief>", "design-ai route --list [--json]", "design-ai route --eval-template [--json]"),
|
|
@@ -364,7 +369,7 @@ EXPECTED_MAIN_HELP_FRAGMENTS = (
|
|
|
364
369
|
"Usage: design-ai <command> [args]",
|
|
365
370
|
"design-ai help [command|--json]",
|
|
366
371
|
"install",
|
|
367
|
-
"search <query>",
|
|
372
|
+
"search <query|--eval-template|--eval>",
|
|
368
373
|
"show <file[:line]>",
|
|
369
374
|
"route <brief|--from-file file|--stdin|--list|--eval-template|--eval>",
|
|
370
375
|
"improve homepage conversion and SEO",
|
|
@@ -4669,8 +4674,8 @@ def passing_main_help_output() -> str:
|
|
|
4669
4674
|
" design-ai help [command|--json]",
|
|
4670
4675
|
"",
|
|
4671
4676
|
" install Symlink design-ai into Claude Code (~/.claude)",
|
|
4672
|
-
" search <query> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider \"cmd args\"]] [--json]",
|
|
4673
|
-
" Search the local markdown corpus",
|
|
4677
|
+
" search <query|--eval-template|--eval> [--dir kind] [--limit N] [--ranked] [--embeddings [--provider \"cmd args\"]] [--strict] [--json]",
|
|
4678
|
+
" Search the local markdown corpus and ranked-search eval checkpoints",
|
|
4674
4679
|
" index [--build|--status|--verify] [--json] [--embeddings [--provider \"cmd args\"]]",
|
|
4675
4680
|
" Build, inspect, and verify the local retrieval index",
|
|
4676
4681
|
" show <file[:line]> [--lines N:M] [--context N] [--json] Print a corpus file or line range",
|