@adia-ai/a2ui-corpus 0.0.2 → 0.0.4

package/CHANGELOG.md

@@ -9,6 +9,94 @@ gap registry. Consumed by `@adia-ai/a2ui-retrieval` and
 
 ## [Unreleased]
 
+## [0.0.4] - 2026-04-28
+
+Same-day follow-up to `0.0.3` shipped earlier today. Adds **semantic
+search vectors** to the corpus and ships them in the tarball so consumers
+get embedding-based retrieval out of the box (no post-install build step).
+Pure-data addition; no schema changes.
+
+### Added
+
+- **`pattern-embeddings.json`** — refreshed against the current
+  113-pattern set (was last built 2026-04-23 against 109 patterns).
+  OpenAI text-embedding-3-small, 1536 dims, ~3.3 MB.
+- **`chunk-embeddings.json`** — NEW. 702 chunks × 1536 dims, ~20 MB.
+  Embeds each chunk's name + kind + primary tag + slot names + nested
+  chunk names + indexed attributes + cluster prefix. HTML body is
+  deliberately NOT embedded (would dominate retrieval noise; the LLM
+  gets the HTML at retrieval time).
+
+### Changed
+
+- `package.json` `files` field now ships both embedding indices in the
+  published tarball. Tarball size grows from ~3.5 MB to ~24 MB; the
+  trade-off is that consumers can use semantic search out of the box
+  without re-running embedding generation.
+- Companion repo tooling (not in tarball but used to generate the
+  corpus):
+  - `scripts/build/embeddings.mjs` — broken `import` paths fixed (the
+    script had been dead since some directory move; load-env.mjs +
+    embedding-provider.js were both at wrong relative paths).
+  - **`scripts/build/embeddings-chunks.mjs`** — new sibling script that
+    walks `chunks/<name>.json` and generates `chunk-embeddings.json`.
+    Auto-detects Voyage / OpenAI / null per the same conventions as the
+    pattern script. Honors `--kind`, `--batch`, `--out`, `--dry-run`.
+  - `npm run build:embeddings` / `build:embeddings:chunks` /
+    `build:embeddings:all` pipeline scripts.
+
+### Performance
+
+- The chunk-aware composition synthesizer (`@adia-ai/a2ui-mcp@0.0.3`)
+  uses these embeddings via `chunk-embedding-retriever.js` (cosine
+  similarity, blended 5× with keyword score). Real-LLM eval improved
+  from 7/10 → **9/10 retrieval hits** out of 10 hold-out intents — only
+  novel composites still need synthesis fallback.
+
+## [0.0.3] - 2026-04-28
+
+Same-day extension after `0.0.2` shipped. Adds the **gen-UI training-chunk
+corpus** — 702 unique chunks (852 instances) harvested from `data-chunk`-marked
+elements across `site/pages/*` and the corpus exemplars. Pure-data addition;
+existing pattern/fragment/composition surface unchanged.
+
+### Added (gen-UI chunks)
+
+- **`chunks/<name>.json`** × 702 — one record per chunk; carries
+  `name`, `kind` (`block` | `panel` | `page`), `primary` (outer tag),
+  `slots[]` (named regions for compositional reasoning), `nested[]`
+  (child chunk names), `attrs` (indexed for retrieval), `html`
+  (bounding HTML, descendants intact), `source` (original file path),
+  `captured_at`. Multi-instance reusable slot chunks
+  (e.g. `auth-card-header`, `reg-step-header`) carry an `instances`
+  array — one entry per page where the chunk appears.
+- **`chunks/_index.json`** — aggregate index (metadata only, no HTML)
+  for fast lookup. Records total counts, by-kind tallies, per-chunk
+  pages + slot names + nested-chunk references.
+- **852 chunk instances** across ~150 distinct pages (70 template +
+  82 component primitive + 16 pattern). Coverage:
+  `block=844, panel=7, page=1`.
+
+### Added (companion tooling — not in tarball)
+
+- `scripts/build/harvest-chunks.mjs` (repo root) — walks site/pages
+  + corpus exemplars; depth-tracked HTML parser; writes per-chunk
+  records + aggregate index. Wired into
+  `packages/a2ui/corpus/scripts/run-pipeline.mjs` Step 1.
+- `packages/a2ui/corpus/scripts/chunk-library.js` — in-memory loader
+  for the chunk corpus; powers the new MCP retrieval tools (consumed
+  by `@adia-ai/a2ui-mcp@0.0.2`).
+- `npm run harvest:chunks` / `harvest:chunks:dry` — pipeline scripts.
+
+### Convention reference
+
+- Spec: [`docs/specs/genui-chunk-marker.md`](../../../docs/specs/genui-chunk-marker.md)
+  (Draft v0.1.0) — three attributes (`data-chunk`, `data-chunk-kind`,
+  `data-chunk-slot`); naming conventions per cluster; visual marker
+  rules; harvester contract.
+- Inventory: [`docs/specs/genui-chunk-inventory-001.md`](../../../docs/specs/genui-chunk-inventory-001.md)
+  — full list with intent labels + review status.
+
 ## [0.0.2] - 2026-04-27
 
 Three-day batch since `0.0.1`. Catalog regenerated against the

package/README.md

@@ -6,8 +6,8 @@ fixtures. Pure data plus the scripts that maintain it. No runtime.
 
 > The pipeline reads this package; this package never reads the pipeline.
 > See [`@adia-ai/a2ui-compose`](../compose) for engine code,
-> [`@adia-ai/web-components`](../web-components) for UI atoms,
-> [`@adia-ai/a2ui-utils`](../a2ui/utils) for the A2UI runtime (renderer,
+> [`@adia-ai/web-components`](../../web-components) for UI atoms,
+> [`@adia-ai/a2ui-utils`](../utils) for the A2UI runtime (renderer,
 > registry, streams, wiring),
 > [`@adia-ai/a2ui-mcp`](../mcp) for the MCP server.
 
@@ -47,16 +47,21 @@ a2ui/corpus/
 │                              extract.js compiles these → chunks/ + catalog
 │
 ├── evals/                   held-out.jsonl + eval fixtures
-├── chunks/                  extracted A2UI fragments  (build output)
+├── chunks/                  gen-UI training chunks (702 unique, 852 instances)
+│                              — harvested from data-chunk markers across
+│                              site/pages/* + corpus exemplars (since 0.0.3).
+│                              Powers compose_from_chunks in @adia-ai/a2ui-mcp.
 ├── pages/                   extracted full-page A2UI   (build output)
 ├── feedback/                daily JSONL — user feedback events
 ├── gaps/                    gap registry — prompts with missing coverage
 ├── scripts/                 maintenance tooling (extract, ingest, feedback, ticket)
+│   └── chunk-library.js       in-memory loader + keyword/semantic search over chunks/
 │
 ├── catalog-a2ui_0_9.json       aggregated artifact — what a2ui-compose + a2ui-mcp read
 ├── catalog-a2ui_0_9_rules.txt  natural-language composition rules (per component)
 ├── common_types.json           shared A2UI type shapes
-├── pattern-embeddings.json     pre-computed embeddings for semantic search
+├── pattern-embeddings.json     pre-computed embeddings for pattern semantic search
+├── chunk-embeddings.json       pre-computed embeddings for chunk semantic search (0.0.4+)
 ├── functions.json              declarative wiring-engine function catalog
 ├── manifest.json               extraction metadata (what / when / counts)
 ├── pattern-specs.md            written specs for each pattern category
@@ -103,12 +108,22 @@ Script inventory (`scripts/`):
 |---------------------------|------------------------------------------------------|
 | `extract.js`              | Exemplar HTML → A2UI chunks & pages                  |
 | `ingest.js`               | Index extracted chunks into the retrieval corpus     |
-| `run-pipeline.mjs`        | Orchestrates extract + ingest                        |
-| `build-pattern-index.mjs` | Computes pattern-embeddings.json for semantic search |
+| `run-pipeline.mjs`        | Orchestrates harvest → extract → ingest → index      |
+| `build-pattern-index.mjs` | Computes the keyword inverted index (`patterns/_index.json`) |
+| `chunk-library.js`        | In-memory loader + keyword/semantic search over `chunks/` |
 | `feedback-report.js`      | Aggregates feedback JSONL into a readable digest     |
 | `feedback-promote.js`     | Moves high-confidence feedback into the pattern lib  |
 | `ticket.mjs`              | Corpus/pipeline issue tracker                        |
 
+Repo-side build scripts (not in tarball; run from the workspace root):
+
+| Script                              | Purpose                                                  |
+|-------------------------------------|----------------------------------------------------------|
+| `npm run harvest:chunks`            | Walks site/pages/* + corpus exemplars, harvests every `[data-chunk]` element, writes `chunks/<name>.json` + `_index.json` |
+| `npm run build:embeddings`          | Generates `pattern-embeddings.json` (113 patterns × 1536d) |
+| `npm run build:embeddings:chunks`   | Generates `chunk-embeddings.json` (702 chunks × 1536d)   |
+| `npm run build:embeddings:all`      | Both embedding indices in sequence                       |
+
 ## Exports
 
 ```javascript
@@ -123,6 +138,11 @@ import fixture from '@adia-ai/a2ui-corpus/evals/held-out.jsonl?raw';
 
 // Exemplar source (for scripts + docs)
 import html from '@adia-ai/a2ui-corpus/exemplars/ui/pricing-tiers.html?raw';
+
+// Chunk corpus (since 0.0.3 / 0.0.4)
+import chunkIndex from '@adia-ai/a2ui-corpus/chunks';   // _index.json (metadata only)
+import { searchChunks, searchChunksAsync, getChunk }
+  from '@adia-ai/a2ui-corpus/chunk-library';            // in-memory query API
 ```
 
 ## Authoring order — demo page → exemplar → training