@adia-ai/a2ui-corpus 0.0.3 → 0.0.5

package/CHANGELOG.md

@@ -9,6 +9,74 @@ gap registry. Consumed by `@adia-ai/a2ui-retrieval` and
 
 ## [Unreleased]
 
+## [0.0.5] - 2026-04-28
+
+Corpus refresh — drops a non-chunk-shaped entry and regenerates both
+embedding indices. Pure-data; no schema changes; no consumer-facing
+API impact beyond a slight retrieval-quality bump.
+
+### Removed
+
+- **`genui-feed-message` chunk** — `<col-ui id="gen-feed">` on
+  `gen-ui-feed/index.html` is sample-data scaffolding (the feed
+  structure varies entirely per session), not a discrete trainable
+  pattern. The `data-chunk` marker was dropped; the source page
+  itself stays. Inventory entry struck through, kept as a strikethrough
+  audit trail.
+
+### Changed
+
+- **`chunks/_index.json` re-harvested** — 702 → **701 unique** chunks;
+  852 → **851 instances**; by-kind `block=843, panel=7, page=1`.
+- **`chunk-embeddings.json` regenerated** — 701 × 1536d (was 702).
+  Same provider / model / dims; just one fewer entry.
+- **`pattern-embeddings.json` regenerated** — same 113 × 1536d count;
+  fresh timestamp.
+
+## [0.0.4] - 2026-04-28
+
+Same-day follow-up to `0.0.3` shipped earlier today. Adds **semantic
+search vectors** to the corpus and ships them in the tarball so consumers
+get embedding-based retrieval out of the box (no post-install build step).
+Pure-data addition; no schema changes.
+
+### Added
+
+- **`pattern-embeddings.json`** — refreshed against the current
+  113-pattern set (was last built 2026-04-23 against 109 patterns).
+  OpenAI text-embedding-3-small, 1536 dims, ~3.3 MB.
+- **`chunk-embeddings.json`** — NEW. 702 chunks × 1536 dims, ~20 MB.
+  Embeds each chunk's name + kind + primary tag + slot names + nested
+  chunk names + indexed attributes + cluster prefix. HTML body is
+  deliberately NOT embedded (would dominate retrieval noise; the LLM
+  gets the HTML at retrieval time).
+
+### Changed
+
+- `package.json` `files` field now ships both embedding indices in the
+  published tarball. Tarball size grows from ~3.5 MB to ~24 MB; the
+  trade-off is that consumers can use semantic search out of the box
+  without re-running embedding generation.
+- Companion repo tooling (not in tarball but used to generate the
+  corpus):
+  - `scripts/build/embeddings.mjs` — broken `import` paths fixed (the
+    script had been dead since some directory move; load-env.mjs +
+    embedding-provider.js were both at wrong relative paths).
+  - **`scripts/build/embeddings-chunks.mjs`** — new sibling script that
+    walks `chunks/<name>.json` and generates `chunk-embeddings.json`.
+    Auto-detects Voyage / OpenAI / null per the same conventions as the
+    pattern script. Honors `--kind`, `--batch`, `--out`, `--dry-run`.
+  - `npm run build:embeddings` / `build:embeddings:chunks` /
+    `build:embeddings:all` pipeline scripts.
+
+### Performance
+
+- The chunk-aware composition synthesizer (`@adia-ai/a2ui-mcp@0.0.3`)
+  uses these embeddings via `chunk-embedding-retriever.js` (cosine
+  similarity, blended 5× with keyword score). Real-LLM eval improved
+  from 7/10 → **9/10 retrieval hits** out of 10 hold-out intents — only
+  novel composites still need synthesis fallback.
+
 ## [0.0.3] - 2026-04-28
 
 Same-day extension after `0.0.2` shipped. Adds the **gen-UI training-chunk

package/README.md

@@ -47,16 +47,21 @@ a2ui/corpus/
 │                              extract.js compiles these → chunks/ + catalog
 │
 ├── evals/                   held-out.jsonl + eval fixtures
-├── chunks/                  extracted A2UI fragments  (build output)
+├── chunks/                  gen-UI training chunks (702 unique, 852 instances)
+│                              — harvested from data-chunk markers across
+│                              site/pages/* + corpus exemplars (since 0.0.3).
+│                              Powers compose_from_chunks in @adia-ai/a2ui-mcp.
 ├── pages/                   extracted full-page A2UI   (build output)
 ├── feedback/                daily JSONL — user feedback events
 ├── gaps/                    gap registry — prompts with missing coverage
 ├── scripts/                 maintenance tooling (extract, ingest, feedback, ticket)
+│   └── chunk-library.js       in-memory loader + keyword/semantic search over chunks/
 │
 ├── catalog-a2ui_0_9.json       aggregated artifact — what a2ui-compose + a2ui-mcp read
 ├── catalog-a2ui_0_9_rules.txt  natural-language composition rules (per component)
 ├── common_types.json           shared A2UI type shapes
-├── pattern-embeddings.json     pre-computed embeddings for semantic search
+├── pattern-embeddings.json     pre-computed embeddings for pattern semantic search
+├── chunk-embeddings.json       pre-computed embeddings for chunk semantic search (0.0.4+)
 ├── functions.json              declarative wiring-engine function catalog
 ├── manifest.json               extraction metadata (what / when / counts)
 ├── pattern-specs.md            written specs for each pattern category
@@ -103,12 +108,22 @@ Script inventory (`scripts/`):
 |---------------------------|------------------------------------------------------|
 | `extract.js`              | Exemplar HTML → A2UI chunks & pages                  |
 | `ingest.js`               | Index extracted chunks into the retrieval corpus     |
-| `run-pipeline.mjs`        | Orchestrates extract + ingest                        |
-| `build-pattern-index.mjs` | Computes pattern-embeddings.json for semantic search |
+| `run-pipeline.mjs`        | Orchestrates harvest → extract → ingest → index      |
+| `build-pattern-index.mjs` | Computes the keyword inverted index (`patterns/_index.json`) |
+| `chunk-library.js`        | In-memory loader + keyword/semantic search over `chunks/` |
 | `feedback-report.js`      | Aggregates feedback JSONL into a readable digest     |
 | `feedback-promote.js`     | Moves high-confidence feedback into the pattern lib  |
 | `ticket.mjs`              | Corpus/pipeline issue tracker                        |
 
+Repo-side build scripts (not in tarball; run from the workspace root):
+
+| Script                              | Purpose                                                  |
+|-------------------------------------|----------------------------------------------------------|
+| `npm run harvest:chunks`            | Walks site/pages/* + corpus exemplars, harvests every `[data-chunk]` element, writes `chunks/<name>.json` + `_index.json` |
+| `npm run build:embeddings`          | Generates `pattern-embeddings.json` (113 patterns × 1536d) |
+| `npm run build:embeddings:chunks`   | Generates `chunk-embeddings.json` (702 chunks × 1536d)   |
+| `npm run build:embeddings:all`      | Both embedding indices in sequence                       |
+
 ## Exports
 
 ```javascript
@@ -123,6 +138,11 @@ import fixture from '@adia-ai/a2ui-corpus/evals/held-out.jsonl?raw';
 
 // Exemplar source (for scripts + docs)
 import html from '@adia-ai/a2ui-corpus/exemplars/ui/pricing-tiers.html?raw';
+
+// Chunk corpus (since 0.0.3 / 0.0.4)
+import chunkIndex from '@adia-ai/a2ui-corpus/chunks';   // _index.json (metadata only)
+import { searchChunks, searchChunksAsync, getChunk }
+  from '@adia-ai/a2ui-corpus/chunk-library';            // in-memory query API
 ```
 
 ## Authoring order — demo page → exemplar → training