@adia-ai/a2ui-corpus 0.4.6 → 0.4.8

package/CHANGELOG.md

@@ -19,6 +19,31 @@ gap registry. Consumed by `@adia-ai/a2ui-retrieval` and
 
 _No pending changes._
 
+## [0.4.8] - 2026-05-12
+
+### Changed — `catalog-a2ui_0_9.json` regen (v0.4.8 ride-along)
+
+`catalog-a2ui_0_9.json` regenerated to pick up v0.4.8 source-side additions across `packages/web-components/` + `packages/web-modules/`: §80 event-type declarations (38 non-form primitives now have `x-adiaui.events` populated for codegen consumption) + the new `composes:` field on web-module YAMLs (ADR-0027 wiring; lifted to `components[name].x-adiaui.composes` on the canonical catalog) + §88 free-form composition mode's component additions. Pure regen — no schema changes; corpus consumers get the new fields automatically.
+
+## [0.4.7] - 2026-05-12
+
+### Removed — `patterns/` + `compositions/` retired from disk + tarball (§72, the §65 carry-over from v0.4.6)
+
+Hand-authored `corpus/patterns/<domain>/*.{json,yaml}` (~225 files) and `corpus/compositions/<domain>/*.json` (~177 files) deleted. The corpus is now one-format and harvester-driven — all retrievable units trace to a real page via `data-chunk-*` annotations per the v0.4.6 grounding rule.
+
+**Exports removed:**
+- `./patterns` → `./patterns/_components.json` — the alias-map subpath. Migrated to `components[name].x-adiaui.synonyms.tags` on the canonical `catalog-a2ui_0_9.json` (still exported as the package root `.`).
+- `./exemplars/*` — exemplar HTML pages (their retrievable content was already harvested into `chunks/`).
+- `./pattern-embeddings` → `./pattern-embeddings.json` — pattern-embeddings index (companion `embedding-retriever.js` + builder `scripts/build/embeddings.mjs` retired in `@adia-ai/a2ui-retrieval` and repo-side, respectively).
+
+**`files:` array** drops `patterns/`, `compositions/`. The remaining published tarball surface: `catalog-a2ui_0_9.json`, `catalog-a2ui_0_9_rules.txt`, `manifest.json`, `chunks/`, `chunk-embeddings.json`, `scripts/chunk-library.js`, `README.md`, `CHANGELOG.md`.
+
+**Alias-data migration:** the 18 alias-bearing components in the legacy `_components.json` had their `aliases` arrays folded into per-component `synonyms.tags` on the source yaml under `packages/web-components/`. Four legacy-only entries (`Chat`, `Panel`, `SegmentedControl`, `Toggle`) merged into their modern canonicals (`ChatThread`, `Pane`, `Segmented`, `Switch`). `catalog-a2ui_0_9.json` regenerated; the alias data lives there now under `components[name].x-adiaui.synonyms.tags`.
+
+### Changed — `README.md`, `data-flow.md`, `package.json` description
+
+`README.md` — Exports section rewritten to show the catalog-with-aliases pattern; line referencing `data-flow.md`'s "chunks → fragments → compositions → feedback" simplified to "chunks → feedback" reflecting the retired stages. `data-flow.md` — full rewrite describing the harvester-driven flow (no more extract.js → ingest.js). `package.json` description — "patterns, compositions, chunks, eval fixtures…" → "canonical v0.9 catalog + chunks + eval fixtures…".
+
 ## [0.4.6] - 2026-05-12
 
 ### Removed — `corpus/scripts/` retirement (§64 step 3, v0.4.6)

package/README.md

@@ -1,7 +1,7 @@
 # @adia-ai/a2ui-corpus
 
 Corpus and operational-learning artifacts for the gen-UI pipeline —
-patterns, fragments, compositions, chunks, feedback, gaps, eval
+**chunks** (the canonical retrieval surface), feedback, gaps, eval
 fixtures. Pure data plus the scripts that maintain it. No runtime.
 
 > The pipeline reads this package; this package never reads the pipeline.
@@ -34,55 +34,52 @@ web-components, so the read-direction stays one-way.
 
 ## Glossary
 
-Three overlapping concepts circulate through this package. Knowing which
-is which prevents reading bugs from creeping into authoring.
+The corpus has converged on **one** retrievable concept: the **chunk**.
 
 | Term | Source of truth | Granularity | What it carries | Engine that consumes it |
 | --- | --- | --- | --- | --- |
-| **chunk** | `chunks/<id>.json` (one file per chunk) + `_index.json` | Smallest — a single labeled HTML region, harvested via `data-chunk` markers | `html`, `intent`, `domain`, `kind` (`block` / `data-chunk` / `field`), `keywords` | `chunk-zettel` + `monolithic-pro` (retrieve → synthesize or inject as structural reference) |
-| **pattern** | `patterns/<domain>/<name>.json` | Full-canvas A2UI template (≈whole-page) | A2UI message tree, retrieval keywords, exemplar provenance | `monolithic-instant` / `monolithic-pro` (retrieve → emit verbatim) |
-| **composition** | `compositions/<domain>/<name>.json` | Multi-section A2UI surface (between pattern and chunk) | Flat A2UI template, keywords + tags | `zettel` (retrieve → emit verbatim) |
-
-**One-line distinctions:**
-
-- **Chunk vs composition** — chunks are *harvested* from `data-chunk` markers in real production HTML. Compositions are *authored* A2UI templates that don't necessarily correspond to a real page surface.
-- **Composition vs pattern** — both are full-surface A2UI trees. Patterns are flat (single-page idea), compositions are multi-section (composed surface). The distinction is fuzzy and may converge over time.
-
-**When in doubt, follow the engine:**
-
-- Looking at `monolithic-*` retrieval? You want a **pattern**.
-- Looking at `zettel` retrieval? You want a **composition**.
-- Looking at `chunk-zettel` or chunk-aware monolithic-pro injection? You want a **chunk**.
-
-> **Note:** *Fragments* (atomic A2UI sub-trees with named slots) were
-> a fourth concept until §37 (2026-05-12). They were retired because
-> the slot-binding machinery added authoring + maintenance cost
-> without enough payoff (10/41 fragments had production examples,
-> 31/41 were stale). Their compositions were inlined into flat A2UI
-> templates per Option C (LLM handles parameterization at gen-time).
-> See [docs/journal/2026/05/2026-05-12.md §37](../../../docs/journal/2026/05/2026-05-12.md).
+| **chunk** | `chunks/<id>.json` (one file per chunk) + `_index.json` | A single labeled HTML region, harvested via `data-chunk` markers | `html`, `intent`, `domain`, `kind` (`block` / `page` / `panel`), `keywords`, `metadata` (when annotated), `template` (transpiled A2UI tree, when annotated) | `chunk-zettel` + `monolithic-pro` + `zettel` (composition-library wraps annotated chunks) |
+
+**Annotated vs raw chunks**: every chunk has `source`/`page` provenance,
+but only chunks carrying `data-chunk-{domain,description,keywords,kind}`
+attributes on their source HTML become retrievable as compositions
+(the harvester's transpile pass produces a `template` + lifts the
+`metadata` block onto them). Raw chunks remain as substrate for
+`nested-expand` reference resolution but don't compete in retrieval.
+
+**Grounding rule (locked v0.4.6, enforced v0.4.7):** every retrievable
+unit MUST trace to a real page under `site/pages/`, `apps/`,
+`playgrounds/`, or `catalog/` via its `data-chunk-*` annotations. No
+hand-authored ungrounded JSON lives in `corpus/`.
+
+> **Historical note:** *Fragments* (atomic A2UI sub-trees with named
+> slots, §37, 2026-05-12), *patterns* (hand-authored full-canvas A2UI
+> templates, `patterns/` dir), and *compositions* (hand-authored
+> multi-section A2UI surfaces, `compositions/` dir) were earlier corpus
+> formats. Fragments retired in v0.4.4 (§37). Patterns + compositions
+> retired in v0.4.7 (§72 — the v0.4.6 carryover of §65). Their
+> retrievable equivalents were either (a) already represented in the
+> chunks corpus via annotated `data-chunk-*` regions, or (b) deemed
+> non-grounded under the locked rule and DELETE'd in §73 Mode-C triage
+> (target for v0.4.7). See [`docs/journal/2026/05/2026-05-12.md`](../../../docs/journal/2026/05/2026-05-12.md)
+> §§ 36-§42, §65, §72 for the multi-arc retirement narrative.
 
 ## Layout
 
 ```
 a2ui/corpus/
-├── patterns/                ~97 full-canvas A2UI templates (retrieval corpus)
-│   ├── <domain>/<name>.json   agent · data · forms · layout · navigation · settings
-│   ├── _components.json       canonical name + aliases map (Carousel → Swiper)
-│   └── _taxonomy.json         concept synonyms (auth → [login, signup, otp])
-│
-├── compositions/            named multi-section A2UI templates (flat,
-│   ├── agent/ data/ form/ layout/ navigation/ settings/
-│                              ~100 surfaces; each = flat A2UI template
-│                              with keywords + tags. Inlined per §37 —
-│                              no $fragment refs remain.
+├── chunks/                  retrievable + raw chunks (~190 entries)
+│                              — one JSON per chunk; carries `source`,
+│                              `metadata` (when annotated), `template`
+│                              (transpiled A2UI tree, when annotated).
+│                              Harvested by `scripts/build/harvest-chunks.mjs`
+│                              from data-chunk markers across site/pages/*,
+│                              apps/*, playgrounds/*, catalog/*.
+│   └── _index.json            harvester output — name + by-kind tallies +
+│                              normalized chunk list (shape consumed by
+│                              chunk-loader + composition-library)
 │
 ├── evals/                   held-out.jsonl + eval fixtures
-├── chunks/                  gen-UI training chunks (329 unique, ~852 instances)
-│                              — harvested from data-chunk markers across
-│                              site/pages/*, apps/*, playgrounds/*, catalog/*.
-│                              Powers compose_from_chunks in @adia-ai/a2ui-mcp.
-├── pages/                   extracted full-page A2UI   (build output)
 ├── feedback/                daily JSONL — user feedback events
 ├── gaps/                    gap registry — prompts with missing coverage
 ├── scripts/                 maintenance tooling (extract, ingest, feedback, ticket)
@@ -91,50 +88,53 @@ a2ui/corpus/
 ├── catalog-a2ui_0_9.json       aggregated artifact — what a2ui-compose + a2ui-mcp read
 ├── catalog-a2ui_0_9_rules.txt  natural-language composition rules (per component)
 ├── common_types.json           shared A2UI type shapes
-├── pattern-embeddings.json     pre-computed embeddings for pattern semantic search
 ├── chunk-embeddings.json       pre-computed embeddings for chunk semantic search (0.0.4+)
 ├── functions.json              declarative wiring-engine function catalog
 ├── manifest.json               extraction metadata (what / when / counts)
-├── pattern-specs.md            written specs for each pattern category
-└── data-flow.md                how the 4 signal sources feed the pipeline
+├── pattern-specs.md            written specs for each pattern category (historical reference)
+└── data-flow.md                how the signal sources feed the pipeline
 ```
 
 ## What's committed vs generated vs published
 
 | Kind                       | Committed | Published to npm | Source of truth                        |
 |----------------------------|:---------:|:----------------:|----------------------------------------|
-| Patterns (`patterns/`)     | ✓         | ✓                | Hand-authored JSON                     |
-| Fragments/compositions     | ✓         | ✓                | Hand-authored JSON                     |
 | Chunks (`chunks/`)         | ✓         | ✓                | `scripts/build/harvest-chunks.mjs` (from `data-chunk` markers across site/pages, apps, playgrounds, catalog) |
-| `catalog-a2ui_0_9.json`    | ✓         | ✓                | `scripts/build-catalog.js` (aggregated)|
+| `catalog-a2ui_0_9.json`    | ✓         | ✓                | `npm run components` (assembled from yamls in `packages/web-components/`) |
 | `chunk-embeddings.json`    | ✓         | **✗ since 0.2.1**| `scripts/build/embeddings-chunks.mjs`  |
-| `pattern-embeddings.json`  | ✓         | **✗ since 0.2.1**| `scripts/build/embeddings.mjs`         |
 | Feedback JSONL             | ✓         | ✓                | Written by `@adia-ai/a2ui-retrieval` at runtime |
 | Gap registry               | ✓         | ✓                | Written by `@adia-ai/a2ui-retrieval` at runtime |
 
 Extracted artifacts are committed for convenience (avoids a build step to
 read the pipeline), but the scripts are authoritative — regenerate via
-`npm run pipeline` if anything drifts.
+`npm run harvest:chunks` (chunks), `npm run components` (catalog), or
+`npm run build:embeddings:chunks` (chunk embeddings) if anything drifts.
 
 ### Why embeddings ship via git, not npm
 
-`chunk-embeddings.json` (~20 MB) and `pattern-embeddings.json` (~3 MB) are
-**committed in git** so the monorepo's own pipeline runs without a network
-round-trip, but **excluded from the published npm tarball** — every
-`npm i @adia-ai/a2ui-corpus` was pulling 23 MB of pre-computed float
-arrays that consumers had no reliable way to address (the
-`chunk-embedding-retriever` resolves them via a relative-path that breaks
-under a `node_modules/@adia-ai/a2ui-corpus/` install layout).
+`chunk-embeddings.json` (~20 MB) is **committed in git** so the monorepo's
+own pipeline runs without a network round-trip, but **excluded from the
+published npm tarball** — every `npm i @adia-ai/a2ui-corpus` was pulling
+~20 MB of pre-computed float arrays that consumers had no reliable way
+to address (the `chunk-embedding-retriever` resolves them via a
+relative-path that breaks under a `node_modules/@adia-ai/a2ui-corpus/`
+install layout).
+
+> The companion `pattern-embeddings.json` retired in v0.4.7 §72 along
+> with the `patterns/` source directory. Its only consumer was
+> `concept-mapper.js` (dead post-v0.4.6 §64 retirement of
+> `pattern-library.js`). The pattern-embeddings build script
+> (`scripts/build/embeddings.mjs`) and the matching
+> `embedding-retriever.js` were retired in the same arc.
 
 Consumers who want embedding-based retrieval either:
 
-1. Regenerate locally — `npm run build:embeddings:all` produces both
-   files in your `node_modules/@adia-ai/a2ui-corpus/` checkout. Requires
-   API access to your embedding provider.
-2. Use the keyword-only fallback — `chunk-library.searchChunks()` and
-   `pattern-library.searchPatterns()` work without embeddings; the
-   embedding-aware `searchChunksAsync` / `searchPatternsAsync` paths
-   fall through to keyword scoring when the index file is absent
+1. Regenerate locally — `npm run build:embeddings:chunks` produces the
+   `chunk-embeddings.json` file in your `node_modules/@adia-ai/a2ui-corpus/`
+   checkout. Requires API access to your embedding provider.
+2. Use the keyword-only fallback — `chunk-library.searchChunks()` works
+   without embeddings; the embedding-aware `searchChunksAsync` path
+   falls through to keyword scoring when the index file is absent
    (`chunk-embedding-retriever.js` returns `null` gracefully).
 
 ### Embedding model pinning
@@ -164,7 +164,7 @@ When upgrading to a new model (e.g. `text-embedding-3-small` →
 `text-embedding-3-large`):
 
 1. Update the default in `embedding-provider.js`.
-2. Rebuild **both** indexes (`npm run build:embeddings:all`).
+2. Rebuild the chunk index (`npm run build:embeddings:chunks`).
 3. Verify with `npm run check:embeddings-fresh` that both index headers
    record the new model.
 4. Re-run `npm run eval:diff -- --engine zettel` to confirm the new
@@ -182,13 +182,12 @@ but the rankings will be incomparable across the two corpora.
 All run from repo root via npm:
 
 ```bash
-npm run pipeline             # extract + ingest (full refresh of chunks/ + catalog)
-npm run pipeline:stats       # report counts without writing
-npm run build:catalog        # aggregate patterns/fragments/compositions → catalog
-npm run build:catalog -- --verify   # fail if catalog is stale vs source
+npm run harvest:chunks       # full re-harvest of chunks/ from data-chunk markers
+npm run components           # regenerate v0.9 sidecars + assemble catalog-a2ui_0_9.json
+npm run components -- --verify   # fail if catalog/sidecars are stale vs yamls
 
 npm run feedback:report      # human-readable feedback digest
-npm run feedback:promote     # promote high-confidence feedback → new patterns
+npm run feedback:promote     # promote high-confidence feedback → new training data
 
 npm run ticket               # open ticket tracker
 npm run ticket:list          # list open tickets
@@ -199,36 +198,34 @@ Script inventory (`scripts/`):
 
 | Script                    | Purpose                                              |
 |---------------------------|------------------------------------------------------|
-| `extract.js`              | **Retired** — fed the exemplar→chunks pipeline (removed 2026-05-11). Kept on disk for archaeology only. |
-| `ingest.js`               | **Retired** — same vintage. Still imported by `run-pipeline.mjs` for back-compat; no-op without exemplars. |
-| `run-pipeline.mjs`        | Pipeline orchestrator — now mostly a no-op on the extract/ingest side; chunks come from `harvest-chunks.mjs` instead |
-| `build-pattern-index.mjs` | Computes the keyword inverted index (`patterns/_index.json`) |
 | `chunk-library.js`        | In-memory loader + keyword/semantic search over `chunks/` |
 | `feedback-report.js`      | Aggregates feedback JSONL into a readable digest     |
-| `feedback-promote.js`     | Moves high-confidence feedback into the pattern lib  |
+| `feedback-promote.js`     | Moves high-confidence feedback into training data    |
 | `ticket.mjs`              | Corpus/pipeline issue tracker                        |
 
+> **Retired in v0.4.7 §72** (with the `patterns/` + `compositions/` dirs):
+> `extract.js`, `ingest.js`, `run-pipeline.mjs`, `build-pattern-index.mjs`.
+> These fed the pattern-library retrieval surface (`pattern-library.js`,
+> retired v0.4.6 §64) and the exemplar→chunks pipeline (retired v0.4.4
+> §36). Pattern embeddings (`scripts/build/embeddings.mjs`) and the
+> grounded-corpus triage audit (`scripts/audit/grounded-corpus-triage.mjs`)
+> also retired in the same arc — the corpus is now one-format
+> (chunks-only) and harvester-driven.
+
 Repo-side build scripts (not in tarball; run from the workspace root):
 
 | Script                              | Purpose                                                  |
 |-------------------------------------|----------------------------------------------------------|
 | `npm run harvest:chunks`            | Walks site/pages/*, apps/*, playgrounds/*, catalog/*, harvests every `[data-chunk]` element, writes `chunks/<name>.json` + `_index.json` |
-| `npm run build:embeddings`          | Generates `pattern-embeddings.json` (113 patterns × 1536d) |
-| `npm run build:embeddings:chunks`   | Generates `chunk-embeddings.json` (329 chunks × 1536d)   |
-| `npm run build:embeddings:all`      | Both embedding indices in sequence                       |
+| `npm run build:embeddings:chunks`   | Generates `chunk-embeddings.json` (~190 chunks × 1536d)  |
 
 ## Exports
 
 ```javascript
-// Catalog — the aggregated read-target for engines
+// Catalog — the aggregated read-target for engines.
+// Carries per-component aliases under `components[name].x-adiaui.synonyms.tags`.
 import catalog from '@adia-ai/a2ui-corpus';
 
-// Alias map + taxonomy
-import aliases from '@adia-ai/a2ui-corpus/patterns';
-
-// Held-out eval fixture
-import fixture from '@adia-ai/a2ui-corpus/evals/held-out.jsonl?raw';
-
 // Chunk corpus (since 0.0.3 / 0.0.4)
 import chunkIndex from '@adia-ai/a2ui-corpus/chunks';   // _index.json (metadata only)
 import { searchChunks, searchChunksAsync, getChunk }
@@ -255,8 +252,7 @@ When adding coverage for a new intent:
    hold coverage ≥ 83%, avgScore ≥ 88 (per the regression floors in
    `AGENTS.md`).
 
-See `data-flow.md` for the full pipeline (chunks → fragments →
-compositions → feedback).
+See `data-flow.md` for the full pipeline (chunks → feedback).
 
 ## Regression floors