kumi 0.0.33 → 0.0.36

data/docs/ARCHITECTURE.md

@@ -1,277 +1,141 @@
-# Kumi Documentation & IDE Architecture
+# Kumi Compiler Architecture
+
+Kumi compiles schemas through a series of explicit semantic layers. Each layer
+owns one kind of transformation, invariants are validated at layer boundaries,
+and the backends emit syntax from fully normalized IR — never meaning from
+partially-lowered IR.
+
+## Pipeline
+
+```text
+Schema source (.kumi text or Ruby DSL)
+  -> AST          parser output
+  -> NAST         normalized AST
+  -> SNAST        semantic AST with dimensional/type stamps
+  -> DFIR         dataflow graph; access paths, imports, structural cleanup
+  -> VecIR        axis-aware value semantics; every value stamped axes+dtype
+  -> LoopIR       explicit execution structure: loops, accumulators, reads
+  -> Emitters     Ruby / JavaScript serialization
+```
 
-## Overview
+The analyzer (`lib/kumi/analyzer.rb`) drives this as three pass groups:
 
-This document describes the automatic documentation generation system and IDE support infrastructure for Kumi functions and kernels.
+- `DEFAULT_PASSES` — name indexing, imports, input collection, validation,
+  dependency resolution, topo ordering, input access planning
+- `LOWERING_PASSES` — NAST → SNAST → DFIR → VecIR → LoopIR, with a validator
+  pass after each IR boundary
+- `TARGET_PASSES` — `Codegen::LoopRubyPass` and `Codegen::LoopJsPass`, which
+  serialize LoopIR
 
-## System Architecture
+## Layer Responsibilities
 
-```
-┌─────────────────────────────────────┐
-│  Function & Kernel Definitions      │
-├─────────────────────────────────────┤
-│  data/functions/*.yaml              │
-│  data/kernels/**/*.yaml             │
-└──────────────┬──────────────────────┘
-               │
-        ┌──────▼──────┐
-        │ bin/kumi-   │
-        │ doc-gen     │
-        └──────┬──────┘
-               │
-      ┌────────┴────────┐
-      │                 │
-   ┌──▼───┐      ┌─────▼──────┐
-   │ JSON │      │  Markdown  │
-   │ Data │      │  Reference │
-   └──┬───┘      └─────┬──────┘
-      │                │
-  ┌───▼──────────┐ ┌───▼─────────────────┐
-  │ IDE Tools    │ │ Developers/Docs     │
-  ├──────────────┤ ├─────────────────────┤
-  │ VSCode       │ │ docs/FUNCTIONS.md   │
-  │ Monaco       │ │ GitHub Reference    │
-  │ LSP Servers  │ │ API Documentation   │
-  └──────────────┘ └─────────────────────┘
-```
+### Frontend: AST, NAST, SNAST
 
-## Core Components
-
-### 1. Data Sources
-
-**Function Definitions** (`data/functions/*.yaml`)
-```yaml
-functions:
-  - id: agg.sum
-    kind: reduce
-    params: [{ name: source_value }]
-    dtype: { rule: same_as, param: source_value }
-    reduction_strategy: identity  # KEY: Identity-based reducer
-    aliases: ["sum"]
-
-  - id: agg.min
-    kind: reduce
-    params: [{ name: source_value }]
-    dtype: { rule: element_of, param: source_value }
-    reduction_strategy: first_element  # KEY: First-element reducer
-    aliases: ["min"]
-```
+Source-language normalization and semantic preparation.
 
-**Kernel Implementations** (`data/kernels/ruby/*.yaml`)
-```yaml
-kernels:
-  - id: agg.sum:ruby:v1
-    fn: agg.sum
-    inline: "+= $1"
-    impl: "(a,b)\n  a + b"
-    fold_inline: "= $0.sum"
-    identity:
-      float: 0.0
-      integer: 0
-```
+- **AST** — parser output, close to source shape
+- **NAST** — normalized form, removes frontend irregularity; constant folding
+- **SNAST** — carries dimensional and type metadata, explicit enough to lower
 
-### 2. Doc Generator Module
-
-**Location:** `lib/kumi/doc_generator/`
-
-#### Loader
-- Parses YAML files from `data/functions/` and `data/kernels/`
-- Returns raw function and kernel definitions
-- No transformation or filtering
-
-#### Merger
-- Combines function definitions with kernel implementations
-- Creates entries indexed by function aliases (so `sum`, `add`, `sub` all resolvable)
-- Extracts important metadata:
-  - `reduction_strategy` - How the reducer initializes
-  - `dtype` - Type inference rules
-  - `arity` - Parameter count
-  - `kernels` - Available implementations
-
-#### Formatters
-
-**Json Formatter**
-- Output: `docs/functions-reference.json`
-- Consumer: IDE plugins (VSCode, Monaco, etc.)
-- Data:
-  - Function ID and aliases
-  - Arity and parameter info
-  - Type information
-  - Kernel availability
-  - **Reduction strategy** (for reducer distinction)
-
-**Markdown Formatter**
-- Output: `docs/FUNCTIONS.md`
-- Consumer: Developers, documentation sites
-- Presentation:
-  - Human-readable function descriptions
-  - Inline operations (`$0 = accumulator, $1 = element`)
-  - Actual implementation code
-  - Fold strategies
-  - Identity values (when applicable)
-  - **Reduction semantics** (monoid vs first-element)
-
-### 3. VSCode Extension
-
-**Location:** `vscode-extension/`
-
-**Features:**
-- Autocomplete for functions when typing `fn(:`
-- Hover tooltips with signatures
-- Schema block context detection (Ruby files only)
-- Works with `.kumi` and `.rb` files
-
-**Components:**
-- `FunctionCompletionProvider` - Offers suggestions
-- `FunctionHoverProvider` - Shows detailed information
-- `isInSchemaBlock()` - Detects if inside `schema do...end` block (Ruby files)
-
-## Key Design Decisions
-
-### 1. Reduction Strategy Distinction
-
-**Problem:** Min/Max don't have identity values like Sum/Count do.
-
-**Solution:** Capture `reduction_strategy` from YAML:
-- `identity` → Monoid operation, can use identity element
-- `first_element` → First array element initializes accumulator
-
-**Display:**
-- Markdown shows: "Monoid operation with identity element" or "First element is initial value"
-- JSON includes: `"reduction_strategy": "identity" | "first_element"`
-
-### 2. Kernel Implementation Visibility
-
-**Decision:** Show actual kernel code inline in markdown.
-
-**Benefits:**
-- Developers see what the function actually does
-- Inline operations (`+= $1` vs `= $1 if $1 < $0`) show the pattern
-- Implementation code is actual Ruby/JavaScript
-
-**Format:**
-```markdown
-**Inline:** `+= $1` ($0 = accumulator, $1 = element)
-**Implementation:**
-```ruby
-(a,b)
-  a + b
-```
-**Fold:** `= $0.sum`
-**Identity:** float: 0.0, integer: 0
-```
+Unsatisfiable-constraint detection and the output/input form schemas are
+derived at this stage.
 
-### 3. Single Source of Truth
+### DFIR
 
-**Flow:**
-```
-YAML definitions
-    ↓
-bin/kumi-doc-gen (one command)
-    ↓
-    ├→ docs/FUNCTIONS.md (auto-generated)
-    ├→ docs/functions-reference.json (auto-generated)
-    └→ IDE/Tools consume JSON
-```
+The first graph-shaped semantic layer.
 
-Changes to function definitions automatically flow to:
-- IDE completions
-- Markdown reference
-- JSON API
-
-### 4. Context-Aware IDE Support
-
-**Ruby files:** Only offer completions inside `schema do...end` blocks
-- Prevents noise from unrelated `fn(:` calls
-- Tracks brace nesting to detect context
-
-**Kumi files:** Always available
-- Native language file type
-
-## Data Model
-
-### Function Entry (After Merge)
-```json
-{
-  "id": "agg.sum",
-  "kind": "reduce",
-  "arity": 1,
-  "params": [{ "name": "source_value" }],
-  "dtype": { "rule": "same_as", "param": "source_value" },
-  "aliases": ["sum"],
-  "reduction_strategy": "identity",
-  "kernels": {
-    "ruby": {
-      "id": "agg.sum:ruby:v1",
-      "inline": "+= $1",
-      "impl": "(a,b)\n  a + b",
-      "fold_inline": "= $0.sum",
-      "identity": { "float": 0.0, "integer": 0 }
-    }
-  }
-}
-```
+- lowers SNAST into per-declaration dataflow functions
+- normalizes input traversal and access paths against the input plans
+- inlines declaration references and **schema imports** (callee bodies are
+  spliced in; axis names and plan references are canonicalized to the
+  caller's input plans at this boundary)
+- canonicalizes tuples/objects, runs CSE, dedup, and broadcast cleanup
 
-## Usage Workflows
+Bugs about input access, import behavior, or graph structure belong here.
 
-### For End Users
+### VecIR
 
-**View function reference:**
-```bash
-# Markdown documentation
-open docs/FUNCTIONS.md
+Pure, declarative, axis-aware value semantics.
 
-# IDE support (VSCode)
-cd vscode-extension && npm install && npm run compile
-# Press F5 in VSCode
-```
+- every value is stamped with `axes` (named, ordered) and `dtype`
+- broadcasts, shifts, indices, maps, selects, and reductions are explicit ops
+- deterministic vector-level rewrites (GVN, canonicalization)
+- no execution, storage, or target concerns
 
-### For Developers
+See [lib/kumi/ir/vec_definition.md](../lib/kumi/ir/vec_definition.md).
 
-**Modify functions:**
-1. Update `data/functions/category/*.yaml`
-2. Run `bin/kumi-doc-gen`
-3. Commit both YAML and generated files
+### LoopIR
 
-**Add new reducer:**
-```yaml
-- id: agg.product
-  kind: reduce
-  params: [{ name: source_value }]
-  reduction_strategy: identity
-  aliases: ["product"]
-```
-Add kernel in `data/kernels/ruby/agg/numeric.yaml`, then regenerate docs.
+The execution layer. Lowering from VecIR materializes:
 
-## Extension Points
+- loop nests (`loop_start`/`loop_end`) over axis carrier arrays
+- reductions as `acc_init`/`acc_step`/`acc_load`
+- `axis_index` as loop index registers; `axis_shift` as policy-explicit
+  shifted reads
+- materialization (`array_init`/`array_push`/`index_read`) for values that
+  cross loop boundaries
 
-The architecture supports adding:
+After LoopIR, emitters never infer axes, rediscover broadcasts, or branch on
+rank. If a backend still needs vector semantics to emit code, LoopIR is not
+finished.
 
-1. **New formatters** (HTML, PDF, LSP protocol)
-2. **New generators** (TypeScript definitions, GraphQL schema)
-3. **New IDE support** (Neovim, Emacs plugins via LSP)
-4. **Validation** (against declared types/arity)
+See [lib/kumi/ir/loop_definition.md](../lib/kumi/ir/loop_definition.md).
 
-All through the same YAML source data.
+### BufIR
 
-## Testing
+A reserved boundary for storage concerns (allocation, lifetimes, layout).
+LoopIR currently stays a pure execution layer and emitters serialize it
+directly, so BufIR remains a stub. It becomes a real phase only if LoopIR
+starts absorbing storage policy or emitters need layout knowledge.
 
-- 8 comprehensive tests for doc generation
-- All 944 existing Kumi tests pass
-- No regression in core functionality
+### Emitters
 
-## Performance
+Mechanical serializers (`Codegen::Loop::Ruby::Emitter`,
+`Codegen::Loop::Js::Emitter`). Every LoopIR opcode maps to a fixed syntax
+shape; emitters own naming, literal formatting, and kernel
+inlining/helper emission — nothing semantic.
 
-- **Generation time:** <100ms for all functions
-- **File sizes:**
-  - FUNCTIONS.md: ~950 lines
-  - functions-reference.json: ~1800 lines
-- **IDE load time:** Instant (JSON loaded once on activation)
+The architecture is healthy when emitter code is boring.
+
+## Boundary Validators
+
+Each IR boundary ends with a validator pass — the executable form of the
+layer contract, not optional diagnostics:
+
+- **`DFValidatePass`** — legal op families, access-path rules, root-only
+  `load_input`, object/tuple sanity
+- **`VecValidatePass`** — axes/dtype present on every value, broadcast and
+  reduction shape rules
+- **`LoopValidatePass`** — only execution opcodes, balanced loops,
+  defs-before-use, defined returns
+
+Validator specs include negative cases for illegal states, not only golden
+snapshots of successful pipelines.
+
+## Ownership Rule
+
+When deciding where a fix belongs:
+
+- meaning, traversal, axes, broadcast, or reduction semantics → DFIR / VecIR /
+  LoopIR
+- materialization, layout, ownership, temporary storage → BufIR
+- emitted target syntax only → the emitter
+
+If a backend patch compensates for an earlier semantic bug, it is in the
+wrong place.
+
+## Verification
+
+Verification is staged. Each layer can be inspected and verified without
+running the later ones:
+
+```bash
+bundle exec bin/kumi pp <repr> <schema.kumi>          # print one layer
+bundle exec bin/kumi golden_v2 verify --repr <group>  # snapshot-check a layer
+bundle exec bin/kumi golden test                      # runtime ground truth
+```
 
-## Future Improvements
+Repr groups: `frontend`, `df`, `vec`, `loop`, `codegen`. Golden failures
+identify which layer regressed, not just that "something failed later".
 
-1. **LSP Server**: Standalone language server for any editor
-2. **Type validation**: Check function call arity at compile time
-3. **IDE diagnostics**: Show type mismatches as you type
-4. **Documentation linking**: Cross-reference related functions
-5. **Kernel visualization**: Show kernel implementations side-by-side
+See [GOLDEN_TESTS.md](GOLDEN_TESTS.md).

data/docs/DEVELOPMENT.md

@@ -79,6 +79,19 @@ bin/kumi golden test           # Run all tests
 bin/kumi golden update EXAMPLE # Regenerate expected files
 ```
 
+## Debugging the Compiler Pipeline
+
+The pipeline (SNAST → DFIR → VecIR → LoopIR → emitters) can be inspected one
+layer at a time — see [ARCHITECTURE.md](ARCHITECTURE.md):
+
+```bash
+bin/kumi pp loopir <schema.kumi>              # print one representation
+bin/kumi golden_v2 verify --repr df           # snapshot-check one layer
+bin/kumi golden_v2 diff --repr codegen <name> # unified diff for a schema
+```
+
+[AGENTS.md](AGENTS.md) collects more inspection recipes.
+
 ---
 
 *For more development documentation, see the other guides in this directory.*