@fraczak/k 6.3.2 → 6.4.0

package/DOCS/KIR_SPECIALIZATION_DESIGN.md

@@ -1,4 +1,4 @@
-# KIR: k Intermediate Representation, Retyping, and Conformance
+# KIR: k Intermediate Representation, Envelope Specialization, and Conformance
 
 The document defines an object IR and conformance platform for LLVM and
 WebAssembly backends.
@@ -6,9 +6,9 @@ WebAssembly backends.
 ## Goals
 
 - Define a portable polymorphic object IR.
-- Define retyping for concrete input envelope patterns.
+- Define envelope specialization for concrete input envelope patterns.
 - Define conformance fixtures for interpreter, object execution,
-  retyped execution, and future backends.
+  envelope-specialized execution, and future backends.
 - Keep `.ko` and `.klib` inspectable.
 - Record type-derivation convergence per relation.
 
@@ -32,7 +32,7 @@ KIR-P must:
 - preserve filters and pattern operations;
 - store relation input and output patterns;
 - be the canonical `.ko` / `.klib` relation format;
-- be the reference for retyping and backend tests.
+- be the reference for envelope specialization and backend tests.
 
 Alternative:
 
@@ -40,29 +40,31 @@ Alternative:
   but makes object behavior depend on the compiler version. Do not use this for
   KIR.
 
-### KIR-R: Retyped IR
+### Envelope-Specialized KIR-P
 
-KIR-R is produced from `KIR-P + input envelope pattern`.
+Envelope-Specialized KIR-P is produced from `KIR-P + input envelope pattern`.
 
-KIR-R must:
+Envelope Specialization must:
 
 - be equivalent to typing the entry expression `?P __main__`;
 - derive the output pattern for that invocation;
 - validate filters and pattern constraints before value execution;
 - preserve the same value-level partial function semantics;
-- support multiple pattern contexts for the same helper relation;
-- be cacheable by relation hash and input pattern hash.
+- support multiple pattern contexts for the same helper relation internally;
+- be cacheable by relation hash and input pattern hash without serializing that
+  cache key into KIR.
 
 Alternative:
 
 - Re-run full source compilation for each input pattern. This is acceptable for
   the first prototype, but the public API should still be
-  `retype(rel, inputPattern)`.
+  `specializeEnvelope(partialFunction, inputEnvelope)`. Current implementation identifiers
+  may still use `retype...` until they are renamed.
 
 ### KIR-M: Backend IR
 
-KIR-M is produced from KIR-R after call-site pattern contexts, layout, and ABI
-decisions are fixed.
+KIR-M is produced from envelope-specialized KIR-P after call-site pattern contexts, layout,
+and ABI decisions are fixed.
 
 KIR-M should:
 
@@ -73,16 +75,19 @@ KIR-M should:
 - require converged type derivation for every compiled relation;
 - target LLVM, Wasm, C, or another backend.
 
-Alternative:
+Current baseline:
 
-- Compile KIR-P directly. This is useful for a correctness experiment, but it
-  recreates the polymorphic interpreter and is not the final performance path.
+- Lower KIR-P directly to kVM. This gives a single backend-facing contract for
+  both polymorphic and envelope-specialized relations. Later KIR-M layout work can specialize
+  the same KIR-P input further instead of adding another serialized relation
+  shape.
 
 Relationship to kVM:
 
 - [KVM_EXECUTION_MODEL.md](KVM_EXECUTION_MODEL.md) sketches kVM as the concrete
   execution contract for KIR-M.
-- KIR-R remains the retyping contract.
+- envelope specialization remains a pass that emits ordinary KIR-P.
+- kVM lowering consumes KIR-P relation bodies directly.
 - kVM keeps product and union as structured regions so backends can choose
   sequential execution first, then safe parallel scheduling later.
 
@@ -124,7 +129,7 @@ Implementation:
 
 Alternative:
 
-- Change the stored object JSON immediately. This is less code, but risks
+- Change the stored object payload immediately. This is less code, but risks
   mixing backend contracts with current JS evaluator internals.
 
 ### Step 1a: Record Type-Derivation Status
@@ -231,9 +236,9 @@ Alternative:
 
 - Store union-find state as debug metadata. Do not use it as backend input.
 
-## Retyping
+## Envelope Specialization
 
-### Step 5: Add `retype(rel, inputPattern)`
+### Step 5: Add Envelope Specialization
 
 Input:
 
@@ -244,9 +249,9 @@ Input:
 
 Output:
 
-- KIR-R relation;
+- envelope-specialized KIR-P relation;
 - output pattern;
-- call-site pattern summary.
+- compiler-local call-site worklist.
 
 Algorithm:
 
@@ -261,13 +266,13 @@ Alternative:
 - Implement this inside `run.mjs` first. This may be useful for bootstrapping,
   but the final API should be separate from the evaluator.
 
-### Step 6: Execution After Retyping
+### Step 6: Execution After Envelope Specialization
 
 Safe baseline:
 
 - keep current envelope-carrying execution;
-- use KIR-R only to derive the top-level output pattern;
-- treat filters as already validated by retyping.
+- use envelope-specialized KIR-P only to derive the top-level output pattern;
+- treat filters as already validated by envelope specialization.
 
 Optimized mode:
 
@@ -281,7 +286,7 @@ Alternative:
 - Never remove envelopes in the JS evaluator. This is simplest and remains the
   reference path. Envelope-free execution can be left to KIR-M.
 
-### Step 7: Call-Site Retyping
+### Step 7: Call-Site Envelope Specialization
 
 One relation may be called under multiple patterns:
 
@@ -290,7 +295,7 @@ f@P1
 f@P2
 ```
 
-KIR-R should model these as separate instances:
+The envelope specialization cache should model these as separate instances:
 
 ```text
 relation hash + input pattern hash
@@ -303,7 +308,7 @@ Alternative:
 - Use one annotation per relation. This is easier, but too coarse for precise
   output patterns and unsafe for envelope-free execution.
 
-### Step 8: Cache Retyping Results
+### Step 8: Cache Envelope Specialization Results
 
 Cache key:
 
@@ -313,15 +318,15 @@ relation-hash + input-pattern-hash
 
 Cache value:
 
-- KIR-R relation;
+- envelope-specialized KIR-P relation;
 - output pattern;
-- call-site pattern summary;
+- optional compiler-local call-site worklist;
 - optional backend artifact.
 
 Alternative:
 
-- Cache by pattern JSON first. Replace with pattern hashes once pattern hashing
-  is stable.
+- Cache by a stable pattern encoding first. Replace with pattern hashes once
+  pattern hashing is stable.
 
 ## Conformance
 
@@ -345,7 +350,7 @@ Example:
   "program": "program.k",
   "input": "input.kv",
   "expected": "expected.kv",
-  "modes": ["source", "object", "retyped"],
+  "modes": ["source", "object", "envelope-specialized"],
   "tags": ["runtime", "projection"]
 }
 ```
@@ -362,7 +367,7 @@ Modes:
 
 - `source`: compile source and run the JS evaluator;
 - `object`: compile `.ko`, load it, and run object evaluation;
-- `retyped`: retype using the input wire pattern and run the retyped evaluator;
+- `envelope-specialized`: specialize using the input wire pattern and run the envelope-specialized evaluator;
 - future `wasm` / `llvm`: run backend artifacts.
 
 Comparison:
@@ -413,12 +418,12 @@ Codecs:
 - Unicode labels;
 - explicit pattern vs derived witness pattern.
 
-Retyping:
+Envelope Specialization:
 
 - output pattern derived from `?P __main__`;
 - helper called under two different patterns;
-- recursive helper retyping convergence;
-- retyping cache reuse.
+- recursive helper envelope specialization convergence;
+- envelope specialization cache reuse.
 
 Backend eligibility:
 
@@ -436,32 +441,32 @@ Alternative:
 - `k-inspect-object`: print object sections, KIR-P, aliases, metadata, and
   pattern summaries.
 - `k-validate-object`: validate `.ko` and `.klib` schemas and references.
-- `k-retype-object`: precompute KIR-R for one relation and input pattern.
-- Retyping heat counters in the JS runtime.
+- `k-specialize-envelope`: precompute envelope-specialized KIR-P for one relation and input pattern.
+- Envelope Specialization heat counters in the JS runtime.
 - Pattern-hash backend artifact cache.
 - Optional binary executable object payload after KIR stabilizes.
 - Minimal C backend before LLVM.
-- Wasm KIR-R interpreter before native LLVM.
+- Wasm envelope-specialized KIR interpreter before native LLVM.
 
 ## Implementation Order
 
 1. Write `DOCS/KIR_V1.md`.
-2. Treat current `.ko` / `.klib` storage as the initial KIR-P input format.
+2. Treat current `.ko` / `.klib` storage as the initial KIR-P input artifact.
 3. Add `kir.mjs` to export KIR-P from current objects.
 4. Add `objects/inspect.mjs --kir`.
 5. Add `objects/validate.mjs`.
 6. Add conformance fixture format and runner.
-7. Add entry retyping for `?P __main__`.
-8. Add a KIR-R JS evaluator or retyped execution wrapper.
-9. Compare KIR-R execution against `run.mjs`.
-10. Add call-site retyping for helper relations.
-11. Add retyping cache.
+7. Add entry envelope specialization for `?P __main__`.
+8. Add an envelope-specialized KIR JS evaluator or envelope-specialized execution wrapper.
+9. Compare envelope-specialized KIR execution against `run.mjs`.
+10. Add call-site envelope specialization for helper relations.
+11. Add envelope specialization cache.
 12. Reject non-converged relations in LLVM/Wasm compilation.
 13. Prototype a small C or Wasm backend through kVM.
 
 Rule:
 
 - KIR-P is the portable semantic contract.
-- KIR-R is the retyping contract.
+- envelope specialization emits KIR-P and may use local cache keys/worklists.
 - KIR-M is backend material; kVM is the proposed concrete execution model for
   that material.

package/DOCS/KIR_V1.md

@@ -0,0 +1,221 @@
+# KIR v1
+
+KIR v1 is the first backend-facing artifact contract for compiled k objects. It
+is an inspection and export contract, not a replacement for the current `.ko`
+and `.klib` containers.
+
+The first layer is **KIR-P**, the portable polymorphic relation format. KIR-P
+keeps the same pattern-carrying semantics as current object execution, but
+normalizes names and graph IDs so backends can consume object contents without
+depending on parser-shaped fields, current object storage, or
+`TypePatternGraph` internals.
+
+## Scope
+
+KIR v1 defines:
+
+- a serialization-ready contract over current `.ko` and `.klib` payloads;
+- relation input and output pattern roots;
+- a closed expression opcode vocabulary;
+- dense per-relation pattern graph IDs;
+- relation type-derivation status;
+- object aliases, metadata, compile statistics, and code table pass-through.
+
+KIR v1 does not define:
+
+- KIR-M layout and ABI decisions;
+- LLVM, Wasm, C, or native runtime layout;
+- a new stored object format;
+- a required physical encoding.
+
+## Top-Level Shape
+
+The current textual export uses this field shape:
+
+```json
+{
+  "format": "k-ir",
+  "version": 1,
+  "layer": "KIR-P",
+  "sourceFormat": "k-object",
+  "kind": "executable",
+  "main": "__main__",
+  "codes": {},
+  "rels": {},
+  "relAlias": {},
+  "compileStats": {},
+  "meta": {}
+}
+```
+
+Rules:
+
+- `kind` is `executable` when `main` names an entry relation.
+- `kind` is `library` when `main` is `null`.
+- `codes`, `relAlias`, `compileStats`, and `meta` preserve the current object
+  payload content, sorted for stable inspection.
+- KIR consumers must not infer backend eligibility from the top level alone.
+  Each relation has its own `typeDerivation.status`.
+
+## Relations
+
+Each relation entry has this shape:
+
+```json
+{
+  "inputPattern": 0,
+  "outputPattern": 1,
+  "typeDerivation": { "status": "converged" },
+  "patternGraph": {
+    "nodes": [],
+    "sourceNodeMap": {}
+  },
+  "body": {}
+}
+```
+
+Rules:
+
+- `inputPattern` and `outputPattern` are node IDs in the relation's
+  `patternGraph`.
+- Valid type-derivation statuses are `converged`, `not-converged`, and
+  `unknown`.
+- LLVM and optimized Wasm backends should reject relations whose status is not
+  `converged`, unless they intentionally implement an envelope-aware reference
+  mode.
+- `sourceNodeMap` maps original object pattern graph representative IDs to KIR
+  node IDs. It exists for inspection and debugging, not as a semantic contract.
+
+## Pattern Graphs
+
+KIR-P pattern graph node IDs are dense and local to each relation.
+
+```json
+{
+  "id": 0,
+  "kind": "open-product",
+  "edges": [{ "label": "x", "target": 1 }]
+}
+```
+
+Valid `kind` values:
+
+- `any`
+- `open-product`
+- `closed-product`
+- `open-union`
+- `closed-union`
+- `type`
+
+Type nodes carry a type-code hash:
+
+```json
+{ "id": 2, "kind": "type", "code": "@..." }
+```
+
+Rules:
+
+- Edge labels are sorted lexically.
+- Parallel edges are represented as repeated edge records with the same label
+  and different targets.
+- Edge targets always name nodes in the same relation pattern graph.
+- The graph is already union-find-normalized; KIR consumers should use KIR node
+  IDs directly.
+
+## Expression Opcodes
+
+KIR-P relation bodies use this closed opcode set:
+
+```json
+{ "op": "identity", "patterns": [0, 0] }
+{ "op": "empty", "patterns": [0, 1] }
+{ "op": "dot", "label": "x", "patterns": [0, 1] }
+{ "op": "div", "tag": "some", "patterns": [0, 1] }
+{ "op": "vid", "tag": "some", "patterns": [0, 1] }
+{ "op": "code", "code": "@...", "patterns": [0, 1] }
+{ "op": "filter", "filter": {}, "patterns": [0, 1] }
+{ "op": "ref", "ref": "@...", "patterns": [0, 1] }
+{ "op": "comp", "items": [], "patterns": [0, 1] }
+{ "op": "union", "items": [], "patterns": [0, 1] }
+{ "op": "product", "fields": [{ "label": "x", "expr": {} }], "patterns": [0, 1] }
+```
+
+Rules:
+
+- `patterns` entries are KIR pattern graph node IDs.
+- `comp.items` preserve composition order.
+- `union.items` preserve source branch priority.
+- `product.fields` preserve the relation body field list. Backends may sort or
+  lay out fields later only after preserving observable semantics.
+- `filter.filter` currently preserves the stored filter tree. A later KIR
+  revision may normalize filter internals.
+
+## Current Tooling
+
+Use `k-inspect-object --kir` to print the current KIR-P export:
+
+```sh
+k-compile program.k program.ko
+k-inspect-object --kir program.ko
+```
+
+Inside a checkout:
+
+```sh
+./objects/compile.mjs '()' /tmp/id.ko
+./objects/inspect.mjs --kir /tmp/id.ko
+```
+
+This command decodes the current object format, hydrates relation pattern
+graphs through the normal object loader, and emits the current KIR-P textual
+encoding.
+
+Use the conformance runner to compare source and object execution and validate
+KIR-P export for deterministic fixtures:
+
+```sh
+npm run conformance
+```
+
+## Backend Path
+
+The intended backend pipeline remains:
+
+```text
+k source
+  -> .ko / .klib object
+  -> KIR-P
+  -> envelope-specialized KIR-P for an input envelope
+  -> KIR-M / kVM
+  -> LLVM / Wasm / C / other backend
+```
+
+KIR-P is the shared semantic object contract. Envelope Specialization produces another KIR-P
+object whose entry relation is specialized by an input envelope. KIR-M remains a
+separate backend contract and should not be encoded by overloading KIR-P fields.
+The current kVM lowerer consumes KIR-P relation bodies directly.
+
+## Envelope Specialization
+
+The current API `retypeObjectRelation(object, relationName, inputPattern)`
+exports envelope-specialized KIR-P. The `retype...` spelling is legacy/current
+implementation naming; new public names should use envelope specialization. The
+pass re-runs existing type derivation as if the entry program were:
+
+```k
+?inputPattern relationName
+```
+
+The result is a normal KIR-P executable object:
+
+- `layer: "KIR-P"`;
+- `kind: "executable"`;
+- `main` naming the specialized entry relation;
+- relation `inputPattern` and `outputPattern` roots in the entry relation's
+  `patternGraph`;
+- ordinary KIR relation bodies in `rels`.
+
+This is the safe baseline: it preserves envelope-aware execution semantics while
+making the specialized entry pattern explicit through ordinary KIR-P relation
+typing. Relation/input-pattern cache keys and call-site worklists are local
+compiler data; they are not part of the serialized KIR artifact.

package/DOCS/KVM_EXECUTION_MODEL.md

@@ -13,23 +13,32 @@ rules for partial failure, products, unions, calls, and safe scheduling.
 The current design already points at three layers:
 
 - KIR-P: portable, polymorphic object IR.
-- KIR-R: KIR-P retyped for a concrete input envelope pattern.
-- KIR-M: backend material after layout and ABI decisions.
+- Polymorphic kVM Template: unspecialized kVM instruction streams with principal pattern signatures.
+- Envelope-Specialized kVM: kVM functions instantiated for a concrete input envelope, with dead branches pruned and redundant guards eliminated.
+- Target Lowering: direct emission from specialized kVM to WebAssembly or native machine code.
 
-kVM should make KIR-M concrete.
+kVM bridges high-level relational expressions and concrete execution targets:
 
 ```text
+[AOT Compilation Phase]
 k source
   -> AST
-  -> type derivation
-  -> KIR-P object relation
-  -> KIR-R relation instance for an input pattern
-  -> kVM function
-  -> LLVM / Wasm / C / JS kVM interpreter
+  -> principal type derivation
+  -> polymorphic KIR-P object relations
+  -> polymorphic kVM template (.kvm)
+
+[Request / JIT Phase: Enveloped Value V = (v, Pv) arrives]
+  kVM template + input envelope Pv
+  -> boundary type compatibility check (validates Pv against Pin; raises Type Error on contradiction)
+  -> fast kVM specialization (specializeKVM: guard folding, dead branch pruning, output pattern derivation)
+  -> specialized kVM function
+  -> Tier 0: direct interpretation in kVM runtime (< 1 ms cold start)
+  -> Tier 1: single-pass lowering to WebAssembly (kvm2wasm) or native machine code (~17 ms peak speed)
 ```
 
-KIR-P remains the portable semantic object format.  KIR-R remains the retyping
-contract.  kVM is the executable middle form consumed by code generators.
+KIR-P remains the portable semantic object format. The kVM lowerer compiles KIR-P into
+polymorphic kVM templates ahead of time. When concrete input envelopes arrive at runtime,
+`specializeKVM` instantiates and optimizes the kVM instruction stream in microseconds.
 
 ## Design Center
 
@@ -79,7 +88,7 @@ There are two valid execution modes:
 
 - **Envelope-aware mode**: each `KRef` may carry a runtime pattern, matching the
   current interpreter and codec model.
-- **Envelope-free mode**: retyping has proven the input and output patterns for
+- **Envelope-free mode**: envelope specialization has proven the input and output patterns for
   every call site, so inner operations use static layouts and attach the derived
   output pattern only at the boundary.
 
@@ -135,7 +144,7 @@ return          %value                     -> ok(%value)
 Composition is ordinary control flow: run the next instruction only after the
 previous one has returned `ok`.
 
-Filters lower either to `guard_pattern` or disappear after retyping proves that
+Filters lower either to `guard_pattern` or disappear after envelope specialization proves that
 the guard is redundant.  Type/code expressions lower to guards.  Variant
 introduction lowers to `make_variant`.  Product construction lowers to a
 product region plus `make_product`.
@@ -235,7 +244,7 @@ failure that an earlier unresolved branch could have prevented.  If the
 contract is only result equivalence for terminating computations, product
 fail-fast and broader cancellation are valid optimizations.
 
-## Lowering From KIR-R
+## Lowering From Envelope-Specialized KIR-P
 
 Suggested lowering rules:
 
@@ -256,10 +265,38 @@ Suggested lowering rules:
 The lowerer should preserve labels and tags as table ids, not strings in hot
 instructions.  Debug metadata may retain source labels.
 
+## Polymorphic kVM and JIT Specialization
+
+A `polymorphic kVM function` is an instruction stream derived from a polymorphic
+KIR-P relation before a concrete input envelope is known. Its input and output
+patterns contain pattern variables (open products, open unions, or type
+variables).
+
+### Specialization Contract (`specializeKVM`)
+
+When an `input enveloped value` $(v, P_V)$ arrives at runtime, `specializeKVM`:
+
+1. **Validates Type Compatibility**:
+   Checks whether $P_V$ conforms to the function's principal input pattern.
+   If $P_V$ is disjoint or contradicts the input pattern, execution immediately
+   signals a **Type Error** (never partial failure) without compiling.
+2. **Substitutes Pattern Variables**:
+   Binds the open pattern variables of the principal input pattern to the concrete
+   subpatterns carried in $P_V$, deriving the concrete output envelope $P_D$.
+3. **Optimizes Guards and Eliminates Dead Code**:
+   - `guard_pattern` instructions proven satisfied by $P_V$ are rewritten to `id`
+     or erased.
+   - `guard_pattern` instructions proven unsatisfiable by $P_V$ are rewritten to `fail`.
+   - In `union` regions, branches that unconditionally fail are pruned. If an
+     earlier branch is proven to always succeed, later branches are discarded.
+4. **Binds Concrete Layouts**:
+   Product field labels and variant tags are mapped to concrete memory offsets
+   matching $P_V$, enabling envelope-free execution in lower tiers.
+
 ## Layout Tables
 
 kVM should not hard-code one memory layout.  It should refer to layout tables
-generated from retyped patterns and canonical codes.
+generated from envelope-specialized patterns and canonical codes.
 
 Useful tables:
 
@@ -385,8 +422,8 @@ only after `f` has failed.
 
 ## Open Questions
 
-- Should kVM be serialized as JSON first, or as a compact binary section inside
-  `.ko` only after the schema stabilizes?
+- Should the first persisted kVM encoding be textual for inspection, compact
+  binary for embedding, or both?
 - Should product fail-fast be the only release behavior, or should there be a
   standard diagnostic mode that collects all failed fields?
 - How should effectful debug intrinsics be represented so they cannot silently
@@ -399,10 +436,10 @@ only after `f` has failed.
 ## Implementation Order
 
 1. Keep KIR-P as the current object-level relation format.
-2. Define a JSON kVM schema for specialized relation instances.
-3. Lower current relation ops to kVM in envelope-aware mode.
+2. Define a kVM artifact schema for specialized relation instances.
+3. Lower KIR-P relation ops to kVM in envelope-aware mode.
 4. Add a sequential JS kVM interpreter and compare it against `run.mjs`.
-5. Lower retyped/converged relation instances to envelope-free kVM.
+5. Lower envelope-specialized and converged relation instances to envelope-free kVM.
 6. Add kVM validation and conformance fixtures.
 7. Add a minimal C or Wasm backend for sequential kVM.
 8. Add LLVM lowering.

package/DOCS/OBJECT_FILE_AND_PATTERN.md

@@ -145,7 +145,8 @@ Standalone helpers live in `objects/`:
 ```sh
 k-compile path/to/program.k path/to/program.ko
 k-compile path/to/library.k path/to/library.klib
-k-compile path/to/program.k path/to/program.kvm
+k-compile --input-pattern '[["closed-product",[]]]' path/to/program.k path/to/program.kvm
+k-compile --input-type '$ bit = <{} off, {} on>; $bit' path/to/program.k path/to/program.kvm
 k-decompile path/to/program.ko path/to/program.decompiled.k
 k-extract-aliases path/to/library.klib path/to/aliases.k
 ```