@fraczak/k 6.0.3 → 6.0.4

package/DOCS/KIR_SPECIALIZATION_DESIGN.md

@@ -0,0 +1,458 @@
+# KIR: k Intermediate Representation, Retyping, and Conformance
+
+The document defines an object IR and conformance platform for LLVM and
+WebAssembly backends.
+
+## Goals
+
+- Define a portable polymorphic object IR.
+- Define retyping for concrete input envelope patterns.
+- Define conformance fixtures for interpreter, object execution,
+  retyped execution, and future backends.
+- Keep `.ko` and `.klib` inspectable.
+- Record type-derivation convergence per relation.
+
+## Out of Scope
+
+- LLVM backend design.
+- WebAssembly backend design.
+- New language syntax.
+- Replacing the current JavaScript evaluator.
+
+## Execution Layers
+
+### KIR-P: Polymorphic IR
+
+KIR-P is the portable representation of a k relation. It keeps the current
+pattern-carrying execution model.
+
+KIR-P must:
+
+- run any valid k program;
+- preserve filters and pattern operations;
+- store relation input and output patterns;
+- be the canonical `.ko` / `.klib` relation format;
+- be the reference for retyping and backend tests.
+
+Alternative:
+
+- Store source and re-run type derivation on load. This gives smaller objects,
+  but makes object behavior depend on the compiler version. Do not use this for
+  KIR.
+
+### KIR-R: Retyped IR
+
+KIR-R is produced from `KIR-P + input envelope pattern`.
+
+KIR-R must:
+
+- be equivalent to typing the entry expression `?P __main__`;
+- derive the output pattern for that invocation;
+- validate filters and pattern constraints before value execution;
+- preserve the same value-level partial function semantics;
+- support multiple pattern contexts for the same helper relation;
+- be cacheable by relation hash and input pattern hash.
+
+Alternative:
+
+- Re-run full source compilation for each input pattern. This is acceptable for
+  the first prototype, but the public API should still be
+  `retype(rel, inputPattern)`.
+
+### KIR-M: Backend IR
+
+KIR-M is produced from KIR-R after call-site pattern contexts, layout, and ABI
+decisions are fixed.
+
+KIR-M should:
+
+- use static product field offsets;
+- use static union tag dispatch;
+- represent partial failure explicitly;
+- avoid envelope propagation only where call-site pattern contexts prove it safe;
+- require converged type derivation for every compiled relation;
+- target LLVM, Wasm, C, or another backend.
+
+Alternative:
+
+- Compile KIR-P directly. This is useful for a correctness experiment, but it
+  recreates the polymorphic interpreter and is not the final performance path.
+
+## Object IR
+
+### Step 1: Define KIR-P
+
+Add `DOCS/KIR_V1.md`.
+
+Define:
+
+- top-level object fields: `format`, `codes`, `rels`, `relAlias`,
+  `compileStats`, `meta`, and `main`;
+- canonical code table;
+- polymorphic relation table;
+- relation aliases;
+- metadata;
+- per-relation type-derivation status;
+- optional origin source range;
+- `.ko` main relation;
+- `.klib` null main.
+
+Current storage rules to preserve:
+
+- `.klib` is plain UTF-8 JSON with no binary header;
+- `.ko` is a `KOBJ\n` binary container around the JSON payload;
+- no payload version field is stored;
+- relation `typeDerivation` stores only `status`;
+- code entries do not carry `typeDerivation`;
+- relation bodies do not store generated input/output boundary filters;
+- source ranges live only in metadata origin entries;
+- origin entries do not have `kind`; the metadata entry has `type`.
+
+Implementation:
+
+- Add `kir.mjs`.
+- Add a converter from current object payloads to KIR-P.
+- Keep current object storage until KIR-P is validated.
+
+Alternative:
+
+- Change the stored object JSON immediately. This is less code, but risks
+  mixing backend contracts with current JS evaluator internals.
+
+### Step 1a: Record Type-Derivation Status
+
+Each relation definition should carry:
+
+```json
+{
+  "typeDerivation": {
+    "status": "converged"
+  }
+}
+```
+
+Rules:
+
+- valid statuses are `converged`, `not-converged`, and `unknown`;
+- LLVM and Wasm compilation require `converged`;
+- codes/types do not carry type-derivation status;
+- `.ko` and `.klib` may still store non-converged relations;
+- object execution may still use the JS evaluator for non-converged relations.
+
+Alternative:
+
+- Store one object-level `backendEligible` flag. This hides which relation
+  blocks backend compilation and is not enough for libraries.
+
+### Step 2: Normalize Opcodes
+
+KIR-P should use a closed opcode set:
+
+```json
+{ "op": "identity" }
+{ "op": "empty" }
+{ "op": "dot", "label": "x" }
+{ "op": "div", "tag": "some" }
+{ "op": "vid", "tag": "some" }
+{ "op": "code", "code": "@..." }
+{ "op": "filter", "filter": {} }
+{ "op": "ref", "ref": "@..." }
+{ "op": "comp", "items": [] }
+{ "op": "union", "items": [] }
+{ "op": "product", "fields": [{ "label": "x", "expr": {} }] }
+```
+
+Alternative:
+
+- Keep current AST field names. This is faster now, but backend code should not
+  depend on parser-shaped fields.
+
+### Step 3: Normalize Codes
+
+Represent codes as canonical product or union nodes:
+
+```json
+{
+  "@hash": {
+    "kind": "product",
+    "fields": [{ "label": "x", "code": "@..." }],
+    "canonical": "$C0={...};"
+  }
+}
+```
+
+Rules:
+
+- Sort labels and tags canonically.
+- Store references as canonical hashes.
+- Validate that each hash matches its canonical form.
+
+Alternative:
+
+- Omit `canonical` and recompute it. This reduces payload size. Keeping it helps
+  debugging and validation during the POC.
+
+### Step 4: Normalize Pattern Graphs
+
+Do not expose `TypePatternGraph` internals as the backend contract.
+
+Use:
+
+```json
+{
+  "root": 0,
+  "nodes": [
+    {
+      "id": 0,
+      "kind": "open-product",
+      "edges": [{ "label": "x", "target": 1 }]
+    }
+  ]
+}
+```
+
+Rules:
+
+- Valid kinds: `any`, `open-product`, `closed-product`, `open-union`,
+  `closed-union`, `type`.
+- Type nodes carry a code hash.
+- Edge targets must exist.
+- Edge order must be canonical.
+
+Alternative:
+
+- Store union-find state as debug metadata. Do not use it as backend input.
+
+## Retyping
+
+### Step 5: Add `retype(rel, inputPattern)`
+
+Input:
+
+- KIR-P relation;
+- concrete canonical input pattern;
+- code table;
+- relation table.
+
+Output:
+
+- KIR-R relation;
+- output pattern;
+- call-site pattern summary.
+
+Algorithm:
+
+1. Type the entry as if the program were `?inputPattern __main__`.
+2. Re-run pattern derivation over the reachable relation graph.
+3. Preserve value-level partial operations.
+4. Record the derived output pattern.
+5. Record call-site pattern contexts for helper calls.
+
+Alternative:
+
+- Implement this inside `run.mjs` first. This may be useful for bootstrapping,
+  but the final API should be separate from the evaluator.
+
+### Step 6: Execution After Retyping
+
+Safe baseline:
+
+- keep current envelope-carrying execution;
+- use KIR-R only to derive the top-level output pattern;
+- treat filters as already validated by retyping.
+
+Optimized mode:
+
+- remove envelope propagation only when every call site has a known stable
+  pattern context;
+- keep value-level partiality for `.field`, `/tag`, merge, and composition;
+- attach the derived output pattern after execution.
+
+Alternative:
+
+- Never remove envelopes in the JS evaluator. This is simplest and remains the
+  reference path. Envelope-free execution can be left to KIR-M.
+
+### Step 7: Call-Site Retyping
+
+One relation may be called under multiple patterns:
+
+```text
+f@P1
+f@P2
+```
+
+KIR-R should model these as separate instances:
+
+```text
+relation hash + input pattern hash
+```
+
+Recursive definitions require fixed-point iteration over these instances.
+
+Alternative:
+
+- Use one annotation per relation. This is easier, but too coarse for precise
+  output patterns and unsafe for envelope-free execution.
+
+### Step 8: Cache Retyping Results
+
+Cache key:
+
+```text
+relation-hash + input-pattern-hash
+```
+
+Cache value:
+
+- KIR-R relation;
+- output pattern;
+- call-site pattern summary;
+- optional backend artifact.
+
+Alternative:
+
+- Cache by pattern JSON first. Replace with pattern hashes once pattern hashing
+  is stable.
+
+## Conformance
+
+### Step 9: Fixture Format
+
+Use one directory per case:
+
+```text
+conformance/projection-field/
+  case.json
+  program.k
+  input.kv
+  expected.kv
+```
+
+Example:
+
+```json
+{
+  "name": "project field",
+  "program": "program.k",
+  "input": "input.kv",
+  "expected": "expected.kv",
+  "modes": ["source", "object", "retyped"],
+  "tags": ["runtime", "projection"]
+}
+```
+
+Alternative:
+
+- Store all fixtures in one JSON file. This is compact but harder to debug.
+
+### Step 10: Runner
+
+Add `conformance/run.mjs`.
+
+Modes:
+
+- `source`: compile source and run the JS evaluator;
+- `object`: compile `.ko`, load it, and run object evaluation;
+- `retyped`: retype using the input wire pattern and run the retyped evaluator;
+- future `wasm` / `llvm`: run backend artifacts.
+
+Comparison:
+
+- Parse `input.kv` through `k-parse`.
+- Run one mode.
+- Print through `k-print`.
+- Compare normalized textual output.
+
+Alternative:
+
+- Compare JS values directly. This is faster, but fixtures should not depend on
+  JS object representation.
+
+### Step 11: Initial Fixtures
+
+Runtime:
+
+- identity;
+- product projection;
+- union projection;
+- variant construction;
+- composition;
+- merge fallback;
+- product construction.
+
+Typing and patterns:
+
+- `$type` filter;
+- open product filter;
+- open union filter;
+- recursive list/tree filter;
+- incompatible filter error.
+
+Objects and libraries:
+
+- compile `.klib`;
+- compile `.ko` with `--lib`;
+- load aliases from metadata;
+- `:load --no-alias`;
+- reload `extract-aliases` output.
+
+Codecs:
+
+- unit;
+- nested product;
+- nested variant;
+- Unicode labels;
+- explicit pattern vs derived witness pattern.
+
+Retyping:
+
+- output pattern derived from `?P __main__`;
+- helper called under two different patterns;
+- recursive helper retyping convergence;
+- retyping cache reuse.
+
+Backend eligibility:
+
+- converged relation compiles to KIR-M;
+- non-converged relation is rejected by LLVM/Wasm compiler;
+- `.klib` with mixed relations reports the exact rejected aliases.
+
+Alternative:
+
+- Start with fuzzing. Do this later. First fixtures should be deterministic and
+  readable.
+
+## Tooling Ideas
+
+- `k-inspect-object`: print object sections, KIR-P, aliases, metadata, and
+  pattern summaries.
+- `k-validate-object`: validate `.ko` and `.klib` schemas and references.
+- `k-retype-object`: precompute KIR-R for one relation and input pattern.
+- Retyping heat counters in the JS runtime.
+- Pattern-hash backend artifact cache.
+- Optional binary executable object payload after KIR stabilizes.
+- Minimal C backend before LLVM.
+- Wasm KIR-R interpreter before native LLVM.
+
+## Implementation Order
+
+1. Write `DOCS/KIR_V1.md`.
+2. Treat current `.ko` / `.klib` storage as the initial KIR-P input format.
+3. Add `kir.mjs` to export KIR-P from current objects.
+4. Add `objects/inspect.mjs --kir`.
+5. Add `objects/validate.mjs`.
+6. Add conformance fixture format and runner.
+7. Add entry retyping for `?P __main__`.
+8. Add a KIR-R JS evaluator or retyped execution wrapper.
+9. Compare KIR-R execution against `run.mjs`.
+10. Add call-site retyping for helper relations.
+11. Add retyping cache.
+12. Reject non-converged relations in LLVM/Wasm compilation.
+13. Prototype a small C or Wasm backend for KIR-R.
+
+Rule:
+
+- KIR-P is the portable semantic contract.
+- KIR-R is the retyping contract.
+- KIR-M is backend material.

package/DOCS/OBJECT_FILE_AND_PATTERN.md

@@ -82,29 +82,53 @@ type still needs design work because the concrete input value is typed by the
 decoded pattern. The important boundary is that object files should carry
 ordinary k values and patterns, not a JSON codec object.
 
-Later versions can add relation IR, constant tables, symbol dictionaries,
-relocation/linking records, optimization metadata, and optional indexes.
+The format can add relation IR, constant tables, symbol dictionaries,
+relocation/linking records, optimization metadata, and optional indexes as the
+implementation changes.
 
-## Executable Object Container v1
+## Object And Library Containers
 
-The current executable object format is a small binary container:
+The current executable `.ko` format is a small binary container:
 
 ```text
-"KOBJ" 0x00 0x01 0x0a
+"KOBJ" 0x0a
 uint32-be payload-byte-length
 utf8-json-payload
 ```
 
+Library files (`.klib`) are plain UTF-8 JSON with no binary header.
+
+Both containers use the same JSON payload shape. Executable `.ko` files set
+`main` to the entry relation name. Library `.klib` files set `main` to `null`.
+There is no payload version field.
+
 The payload stores:
 
 - the normalized code repository needed by the object,
-- the compiled relation IR,
+- the compiled relation IR without generated boundary filters,
 - each relation's type-pattern graph state,
+- each relation's `typeDerivation.status`,
+- relation aliases in `relAlias`,
+- compiler convergence details in top-level `compileStats`,
+- metadata, including aliases and origin source locations,
 - the main relation name.
 
+Relation `def` values do not store source `start` / `end` markers. Source
+locations live on `meta[hash].origins[]` entries. Each metadata entry has
+`type: "code"` or `type: "rel"`. Origin entries hold `source`, `name`,
+`compiledAt`, and optional `start` / `end`; origin entries do not have `kind`.
+
+Relation `def` values also do not store the generated input/output filters.
+Those filters are derived from `def.patterns` and `typePatternGraph` when
+printing or enforcing relation boundaries.
+
+Codes/types do not carry `typeDerivation`. Type derivation is a relation
+property, and the persisted `typeDerivation` object currently contains only
+`status`.
+
 Loading an object hydrates the type-pattern graphs back into runtime
-`TypePatternGraph` instances and executes them through `run.mjs`. This avoids
-source parsing and type derivation on the execution path.
+`TypePatternGraph` instances and executes relation boundaries through
+`run.mjs`. This avoids source parsing and type derivation on the execution path.
 
 Object execution through `k.mjs`:
 
@@ -120,10 +144,12 @@ Standalone helpers live in `objects/`:
 
 ```sh
 ./objects/compile.mjs path/to/program.k path/to/program.ko
+./objects/compile-lib.mjs path/to/library.k path/to/library.klib
 ./objects/decompile.mjs path/to/program.ko path/to/program.decompiled.k
+./objects/extract-aliases.mjs path/to/library.klib path/to/aliases.k
 ```
 
-With no arguments, both helpers read from stdin and write to stdout:
+With no arguments, the helpers read from stdin and write to stdout:
 
 ```sh
 cat path/to/program.k | ./objects/compile.mjs > path/to/program.ko
@@ -153,8 +179,8 @@ compile it with:
 The first bytes of `id.ko` are the binary container header:
 
 ```text
-4b 4f 42 4a 00 01 0a 00 00 03 0b ...
-K  O  B  J  \0 v1 \n payload-length
+4b 4f 42 4a 0a 00 00 03 0b ...
+K  O  B  J  \n payload-length
 ```
 
 After the header, the payload is UTF-8 JSON. Pretty-printed and shortened, the
@@ -163,59 +189,119 @@ payload has this shape:
 ```json
 {
   "format": "k-object",
-  "version": 1,
-  "codes": {
-    "@NiDZqYggx3VZ6b8quBZKTfkgJztWctkesuX4CrhTxM5c": {
-      "code": "product",
-      "product": {},
-      "def": "$C0={};"
+  "codes": {},
+  "main": "__main__",
+  "rels": {
+    "__main__": {
+      "def": { "op": "identity", "patterns": [0, 0] },
+      "typeDerivation": {
+        "status": "converged"
+      },
+      "typePatternGraph": {
+        "patterns": {
+          "nodes": [{ "pattern": "(...)", "fields": [] }],
+          "parent": []
+        },
+        "edges": [{}],
+        "codeId": {}
+      }
     }
   },
-  "main": "__main__",
-  "defs": {
-    "rels": {
-      "__main__": {
-        "def": {
-          "op": "filter",
-          "filter": { "type": null, "open": true, "fields": {}, "name": "X0" },
-          "patterns": [0, 0],
+  "relAlias": { "__main__": "@QfyLpmn56wuppuGvrrUJz8LKtgaXfbMtad7RnzBcFk2S" },
+  "compileStats": { "sccs": [...], "sccCount": 1 },
+  "meta": {
+    "@QfyLpmn56wuppuGvrrUJz8LKtgaXfbMtad7RnzBcFk2S": {
+      "type": "rel",
+      "origins": [
+        {
+          "source": "id.k",
+          "name": "__main__",
+          "compiledAt": "2026-05-20T00:00:00.000Z",
           "start": { "line": 1, "column": 1 },
           "end": { "line": 1, "column": 3 }
-        },
-        "typePatternGraph": {
-          "patterns": {
-            "nodes": [{ "pattern": "(...)", "fields": [] }],
-            "parent": []
-          },
-          "edges": [{}],
-          "codeId": {}
         }
-      }
-    },
-    "representatives": {},
-    "relAlias": { "__main__": "..." },
-    "compileStats": { "sccs": [...], "sccCount": 1 }
+      ]
+    }
   }
 }
 ```
 
 The `codes` field is the type repository snapshot needed by this object. The
-`defs.rels.__main__.def` field is the executable relation IR. The
+`rels.__main__.def` field is the executable relation body. The
 `typePatternGraph` field is the compiled pattern graph used by `run.mjs` for
-input-envelope refinement. Loading the object reconstructs live
+relation-boundary refinement. Loading the object reconstructs live
 `TypePatternGraph` instances from this JSON state and then evaluates the input
-value directly against the compiled IR.
+value against the compiled IR.
 
 Decompiling this object produces canonical source with generated prefix names:
 
 ```k
 ----- codes -----
-$ NiDZ = {};
 ----- rels -----
-QfyL = ?X0;
+QfyL = ?(...) () ?(...);
 ----- main -----
 ?(...) QfyL ?(...)
 ```
 
 In the `rels` section, relation definitions are printed in the same SCC order
 recorded during compilation. Non-empty SCC groups are separated by a blank line.
+
+## Small Library Example
+
+For a definitions-only source file:
+
+```k
+-- defs.k
+$ nat = <{} zero, nat succ>;
+succ = |succ;
+```
+
+compile it with:
+
+```sh
+./objects/compile-lib.mjs defs.k defs.klib
+```
+
+The output file is plain JSON and starts with `{`, not a binary prefix. Its
+shape is the same object payload shape with `main: null`:
+
+```json
+{
+  "format": "k-object",
+  "codes": {
+    "@...": {
+      "code": "union",
+      "union": { "zero": "@...", "succ": "@..." },
+      "def": "$C0=<C1\"succ\",C2\"zero\">;..."
+    }
+  },
+  "rels": {
+    "@...": {
+      "def": { "op": "vid", "vid": "succ", "patterns": [0, 1] },
+      "typeDerivation": { "status": "converged" },
+      "typePatternGraph": { "patterns": "...", "edges": "...", "codeId": "..." }
+    }
+  },
+  "relAlias": { "succ": "@..." },
+  "compileStats": { "sccs": [...], "sccCount": 1 },
+  "meta": {
+    "@...": {
+      "type": "rel",
+      "origins": [
+        {
+          "source": "defs.k",
+          "name": "succ",
+          "compiledAt": "2026-05-20T00:00:00.000Z",
+          "start": { "line": 3, "column": 8 },
+          "end": { "line": 3, "column": 13 }
+        }
+      ]
+    }
+  },
+  "main": null
+}
+```
+
+When a library is compiled with `--lib`, the output is the merged library
+closure. Imported codes, relations, aliases, and metadata are preserved, and the
+new source adds its own origins.