@adia-ai/adia-ui-forge 0.1.3 → 0.8.0

package/skills/adia-ui-a2ui/references/eval-diagnostics.md

@@ -1,127 +0,0 @@
-# Reference: Eval diagnostics — gap diagnosis + regression triage
-
-**Source:** Absorbed from the former `a2ui-pipeline` skill (§Eval Gap Diagnosis + §Eval Suite + §Verification) — Phase 3 rollup. **Used by:** mode 6 of `adia-ui-a2ui` (diagnose eval gap or regression). **Companion:** `strategy-engines.md`, `zettel-calibration.md`.
-
----
-
-## Eval Gap Diagnosis
-
-**Mandatory first step — run diagnostics before touching code.**
-
-The #1 anti-pattern: "tweak and hope" — changing a prompt or adding a chunk without knowing why the current score is low. This wastes eval runs (real LLM cost per run).
-
-## Phase 0: Author a diagnostic script
-
-For each failing intent, capture:
-
-1. **Search ranking** — what `searchChunks()` returns for the intent query
-2. **Composition output** — what the pipeline actually emitted (HTML or plan)
-3. **Tag inventory** — every custom-element-like tag in emitted HTML
-4. **Coverage delta** — expected_components vs. found_components
-
-Diagnostic must be runnable in **stub mode** (no LLM cost).
-
-Template:
-
-```js
-async function diagnose(intent) {
-  const search = searchChunks(intent.intent, { limit: 10 });
-  const comp = await composeFromIntent({ intent: intent.intent, llmAdapter: null });
-  const tags = [...comp.html.matchAll(/<([a-z]+-[a-z-]+)[\s>]/gi)]
-    .map(m => m[1]).filter((v,i,a) => a.indexOf(v)===i);
-  const found = intent.expected_components.filter(tag =>
-    new RegExp(`<${kebab(tag)}-ui[\\s>]`).test(comp.html));
-  return { search, tags, found, missing: intent.expected_components.filter(t => !found.includes(t)) };
-}
-```
-
-## Phase 1: Classify failures
-
-| Bucket | Symptom | Root cause | Speed |
-| --- | --- | --- | --- |
-| A. Holdout misalignment | Top-1 retrieved ≠ expected_chunk | Holdout expected_chunk drifted from corpus | Minutes |
-| B. Coverage gap | Retrieved chunk exists but HTML lacks expected tags | Chunk HTML doesn't contain those components | Hours (new chunks) |
-| C. Wrong shell | Retrieval OK but LLM picks wrong page shell | Prompt ambiguity or missing domain→shell mapping | Hours (prompt tuning) |
-| D. Broken render | HTML emitted but console errors / blank | Missing component registrations or bad markup | Hours (harvester bug) |
-| E. Measurement bug | Composition looks correct but score low | Scoring code has regex/casing/substring bug | Minutes (fix scorer) |
-| F. Embedding drift | Async search returns different top-1 than sync | Embeddings non-deterministically boost wrong chunks | Minutes (use sync for fast path) |
-
-**Fix order:** A → E → F → C → B → D
-
-- Fix measurement before fixing content (otherwise you can't trust scores)
-- Fix determinism before adding content (otherwise evals fluctuate)
-
-## Phase 2: Specific fixes
-
-### A. Holdout alignment
-
-```bash
-npm run eval:diagnose
-# Or directly: node packages/a2ui/mcp/scripts/eval-fix.mjs --verbose
-```
-
-Map each intent's `expected_chunk` to the actual top-1 retrieved chunk. Update `holdout-compose-from-chunks.jsonl`.
-
-### E. Measurement bugs (common traps)
-
-1. **PascalCase → kebab-case** — `AgentTrace` must become `agent-trace` (not `agenttrace`)
-2. **Substring match** — `pane` matching `panel` gives false positives
-3. **Regex escaping** — `textarea-ui` contains `text-ui` as substring; need word boundaries
-4. **Case sensitivity** — HTML may be uppercase; regex needs `/i` flag
-
-### F. Embedding drift
-
-Symptom: `searchChunksAsync` returns different top-1 than `searchChunks`. Root cause: small cosine boosts (0.1–0.3) flip rankings unpredictably. Fix: prefer sync keyword search for deterministic paths (fast-retrieval tier). Keep async for synthesis-tier only (where creativity is desired).
-
-### B. Coverage gap (adding block chunks)
-
-When the LLM needs components not in corpus:
-
-1. Create `catalog/ui-patterns/app/<name>/<name>.contents.html`
-2. Use `data-chunk="<name>"` + `data-chunk-kind="block"` markers
-3. Include actual component tags (`<input-ui>`, `<button-ui>`, etc.) so coverage scoring matches
-4. Re-harvest: `npm run harvest:chunks`
-
-### C. Wrong shell (prompt tuning)
-
-When the LLM consistently picks the wrong page shell:
-
-1. Check `SYSTEM_PROMPT` domain→shell mapping table
-2. Add explicit examples for the failing domain
-3. Add negative constraint: "NEVER default to dashboard-admin-page for non-dashboard intents"
-
-### D. Broken render
-
-When HTML is structurally valid but render score is low:
-
-1. Check `render-fidelity.mjs` output: console errors, blank viewport, undefined custom elements
-2. Verify component registrations in `packages/web-components/index.js`
-3. Check harvester didn't strip `data-chunk-slot` from page shells
-
-## Verification
-
-After all fixes, run:
-
-```bash
-# Stub first (fast, free)
-npm run eval:compose-from-chunks
-
-# Then real-LLM if stub improved
-npm run eval:compose-from-chunks -- --real-llm --report-file
-```
-
-Stop only when all intents pass and avg is stable across 3 runs.
-
-## Eval Suite
-
-`packages/a2ui/mcp/scripts/test-evals.mjs` scores generated output on 5 weighted dimensions:
-
-- structural_validity (30%): schema validation score
-- intent_alignment (25%): F1 of required vs present components
-- component_coverage (20%): absence of forbidden patterns
-- card_model_compliance (15%): header/section/footer structure
-- anti_pattern_count (10%): text without variant, header children without slot
-
-Baseline regression: `--save-baseline` stores scores, subsequent runs flag any dimension dropping
-
-> 5 points or aggregate dropping >3 points. Exit code 2 = regression.

package/skills/adia-ui-a2ui/references/fragment-graph.md

@@ -1,91 +0,0 @@
-# Reference: Fragment graph — leverage rule, extraction, $fragment refs
-
-**Source:** Absorbed from the former `a2ui-pipeline` skill (§Fragment Extraction) and `zettel-internals` (§How fragment-graph composer resolves `$fragment` refs) — Phase 3 rollup. **Used by:** mode 4 of `adia-ui-a2ui` (extract a fragment). **Companion:** `strategy-engines.md`, `chunk-authoring.md`.
-
----
-
-## Fragment Extraction
-
-Use when converting a monolithic pattern into a composition that references one or more zettel fragments, OR when authoring a new fragment from shared substructure across multiple patterns.
-
-## The leverage rule
-
-**Do not extract a fragment unless it has leverage ≥ 3** — i.e., at least 3 compositions would use it. Two exceptions:
-
-1. Singleton fragment that closes a well-defined semantic gap (e.g., `kbd-shortcut-row` even at leverage 1, because "a keyboard shortcut row" is a distinct domain primitive).
-
-2. **Intra-composition multi-use.** A single composition that instantiates the same fragment N times (N ≥ ~10) also justifies extraction — reuse ratio is `fragment_refs / composition_nodes`, not `fragments / comps`. Example: `calendar-day-cell` is used 35× inside one composition (`calendar-month-view`), and this alone lifted corpus reuse from 26.4% → 33.5%. Treat a repeating subtree of 10+ instances as a fragment even if no other composition references it yet.
-
-Sub-leverage fragments bloat the library, slow retrieval, and make maintenance harder without improving reuse.
-
-## Steps
-
-1. **Candidate identification**
-   - Look for repeated subtrees across ≥ 3 patterns
-   - Common candidates: card headers, key-value rows, icon+text rows, labeled progress bars, stat displays, notification rows
-   - Check for existing fragment first: `ls packages/a2ui/compose/fragments/` or call the `zettel_stats` MCP tool to see the current corpus
-
-2. **Fragment authoring**
-   - File: `packages/a2ui/compose/fragments/<name>.json`
-   - Schema: `{ name, description, keywords, slots, template }`
-   - Keywords should be rich — retrieval depends on them
-   - Slots use `{ $slot: 'name' }` placeholders in the template
-
-3. **Composition refactor**
-   - Replace the inlined subtree with `{ $fragment: 'name', bindings: {...} }`
-   - Composer at `packages/a2ui/compose/strategies/zettel/composer.js` handles namespaced id rewriting and slot expansion
-
-4. **KEYWORD PRESERVATION (critical)**
-   - Extraction SHRINKS the composition's own keyword surface
-   - Before: the composition contained all the subtree's text
-   - After: the composition only references the fragment
-   - **Fix:** enrich the composition's `keywords` field with the semantic tokens that left with the fragment
-   - Without this, retrieval degrades silently
-
-5. **Rebuild + verify**
-   - `node scripts/build/components.mjs`
-   - Run the full structural-gate sweep (see SKILL.md §Plan-Execute-Verify)
-   - Specifically check `eval:diff --engine zettel` for coverage + avgScore regressions
-
-## The extraction drift lesson
-
-During the zettel migration, extracting `card-header-with-description` from the `login-form` pattern dropped its retrieval score from 28 → 24. The cause: the composition lost all the "login", "sign in", "email password" tokens because those moved into the fragment slot bindings.
-
-Fix applied: added `keywords: ["login", "sign in", "email", "password", "authenticate"]` explicitly to the composition. Retrieval rebounded to 28.
-
-**Lesson:** every fragment-using composition must have richer keywords than its inlined predecessor.
-
-## Threshold calibration
-
-If a known-good intent starts scoring below the retrieval threshold after extraction, DO NOT lower the threshold first. First:
-
-1. Verify the composition's keywords were preserved
-2. Check the fragment's description doesn't cannibalize the composition's semantic space
-3. Only then consider threshold adjustment (current: 22; lowering below 20 starts producing false matches)
-
-## Current corpus state
-
-For the live fragment count, reuse ratio, and top-leverage fragments, run the `zettel_stats` MCP tool or `npm run pipeline:stats` — these are runtime-derived, not pinned in this reference (the numbers drift as the corpus grows).
-
----
-
-## How fragment-graph composer resolves `$fragment` refs
-
-`composer.js` algorithm (memorize before modifying):
-
-```text
-For each node in the composition template:
-  If node.$fragment:
-    1. Clone the fragment's template
-    2. Prefix every internal id with the composition-node id (avoid collisions)
-       e.g. fragment node "fi-in" under composition node "email" → "email--fi-in"
-    3. Apply slot bindings (set the attribute on the slot's targetId)
-       slots: [{ name: "label", targetId: "fi-field", attribute: "label", required: true }]
-       bindings: { label: "Email" } → cloned[0].label = "Email"
-    4. If node.children was provided, append those ids to the fragment root's children
-    5. Emit the fragment root under node.id, then the rest of the fragment nodes
-  Else:
-    Emit the node as-is
-```
-
-**Don't change ID prefixing without a corpus-wide search** — fragments that compose nest IDs predictably; consumers may pattern-match on the `{compNode}--{fragNode}` shape.

package/skills/adia-ui-a2ui/references/mcp-pipeline-ops.md

@@ -1,106 +0,0 @@
-# Reference: MCP pipeline operations — generate → validate → render → feedback
-
-**Source:** Absorbed from the former `adia-ui-training` skill (§Workflows + §Validation checks + §Feedback loop) — Phase 3 rollup. **Used by:** mode 1 of `adia-ui-a2ui` (operator-side MCP runs). **Companion:** `mcp-tool-reference.md`, `anti-patterns.md`, `eval-diagnostics.md`.
-
----
-
-## Workflows
-
-### Full pipeline — one command
-
-```bash
-node scripts/mcp-pipeline.cjs "dashboard with 4 stat cards and a revenue chart"
-```
-
-Pipes `intent → generate_ui → validate_schema → a2ui-to-html → check_anti_patterns` and prints the scores. Fastest way to confirm the whole stack is working after a change.
-
-### Step-by-step (when you need to inspect intermediates)
-
-```bash
-# 1. Generate
-node scripts/mcp-call.cjs generate_ui \
-  '{"intent":"dashboard with 4 stat cards","mode":"instant"}'
-
-# 2. Validate — paste the messages array from step 1
-node scripts/mcp-call.cjs validate_schema \
-  '{"messages":"<paste-messages-json>"}'
-
-# 3. Render — inspect the HTML before anti-pattern scanning
-echo '<paste-messages-json>' | node scripts/a2ui-to-html.cjs
-
-# 4. Anti-pattern check
-node scripts/mcp-call.cjs check_anti_patterns \
-  '{"html":"<paste-rendered-html>"}'
-```
-
-### Validation-only (existing A2UI doc)
-
-Useful for auditing training corpus entries. `validate_schema` is fast and deterministic — batch it over `packages/a2ui/corpus/patterns/**/*.json` to surface drift without re-running the generator.
-
-### Multi-turn generation (legacy executionId chain)
-
-Every `generate_ui` response includes an `executionId`. Pass it back on subsequent calls to keep the pattern-library context and the feedback record coherent — so a score submitted against that id attributes correctly to the original run.
-
-### Compose from the chunk corpus
-
-When the intent matches a known page-shape (auth flow, dashboard layout, error shell), prefer `compose_from_chunks` over `generate_ui`:
-
-```bash
-node scripts/mcp-call.cjs compose_from_chunks \
-  '{"intent":"sign-in card with email + password + OAuth"}'
-```
-
-The synthesizer first checks for a strong retrieval match (score ≥ 8); if no chunk dominates, it picks a `{page, slot_bindings}` plan from a pre-filtered ~30 catalog and materializes via the chunk composer. Validator enforces slot-name + chunk-kind contracts before HTML emission. Returns `{state_id, html, plan, candidates}` — pass `state_id` to `refine_composition` to mutate this surface in subsequent turns.
-
-### Multi-turn refinement (state_id chain)
-
-```bash
-# Turn 1: create the surface
-node scripts/mcp-call.cjs compose_from_chunks \
-  '{"intent":"sign-in card with email + password"}'
-# → returns state_id "abc123"
-
-# Turn 2: mutate by intent
-node scripts/mcp-call.cjs refine_composition \
-  '{"state_id":"abc123","intent":"add OAuth row with Google + GitHub"}'
-# → returns updateComponents A2UI messages + new state_id "def456"
-```
-
-The refiner runs two-pass synthesis:
-
-1. **Locator** — identifies which slots are affected by the intent
-2. **Modifier** — picks the op (`rebindSlot`/`appendToSlot`/`removeFromSlot`/`replacePage`) and target chunk
-
-Validator-driven retry loop (default `maxAttempts=2`). Each refinement chains through `parent_state_id` so you can `get_state` and walk back through the conversation history.
-
-### Reporting issues from the LLM side
-
-If you (the agent) determine that the engine produced something that breaks expectations, fire `report_issue`:
-
-```bash
-node scripts/mcp-call.cjs report_issue \
-  '{"state_id":"abc123","reporter":"llm","reason":"slot-binding produced empty section","trace":{...}}'
-```
-
-Lands as immutable JSON under `.brain/audit-history/issues/`. These become weekly-triage candidates for promotion to curated tickets when patterns emerge.
-
-## Validation checks
-
-`validate_schema` runs a weighted checklist (target aggregate ≥ 80). Common failures when training data drifts:
-
-- `hasRootComponent` — missing `id: "root"` on the surface root.
-- `cardContentModel` — section without `col-ui` wrapper, or heading inside section instead of header.
-- `headingHierarchy` — skipped levels (h1 → h3).
-- `flatAdjacency` — nested components instead of sibling id references.
-
-Full list + weights: `mcp-tool-reference.md`.
-
-## Feedback loop
-
-After a run, score the output with `submit_feedback` keyed on the `executionId`. The feedback analyzer (`packages/a2ui/retrieval/feedback/feedback-analyzer.js`) aggregates these into:
-
-- Per-intent quality trends.
-- Promotion candidates (runs scoring ≥ 95 with ≥ 4 rating across 3+ runs become named patterns via `npm run feedback:promote --apply`).
-- Gap registry — intents with no pattern match AND low scores land in `packages/a2ui/corpus/gaps/registry.json`.
-
-Running `npm run feedback:report` surfaces the current state.