@fraczak/k 7.0.2 → 7.1.0

package/CHANGELOG.md

@@ -5,6 +5,48 @@ All notable changes to this project will be documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [7.1.0] — 2026-10-16 — Polymorphic K IR (PKIR) & CLI Standardization
+
+### Features & Architecture Improvements
+
+- **Polymorphic K IR (PKIR) & Access Table Engine**:
+  - Implemented `pkir.mjs` providing an intermediate representation
+    tailored for polymorphic relations with variant dispatch, projection
+    tables, and partial function failure semantics.
+  - Implemented `access-table.mjs` (`buildAccessTable`) for resolving
+    field offsets and variant selectors across arbitrary schemas.
+  - Added PKIR specialization (`specializePKIR`) eliminating dead
+    branches and baking in direct physical offsets.
+  - Supported PKIR execution and standalone executable generation across
+    both WebAssembly and LLVM backends.
+  - Added dedicated test suites: `tests/test-pkir.mjs`,
+    `tests/test-access-table.mjs`, and `tests/test-polymorphic-tdd.mjs`.
+  - Added specification and documentation in `DOCS/POLYMORPHIC_IR.md`
+    and updated `DOCS/FILE_FORMATS.md`.
+
+- **Web REPL (`repl.html`) & Wasm Integration**:
+  - Enabled `.pkir` generation and lowering to WebAssembly in
+    `repl.mjs` and `repl.html`.
+  - Promoted Web REPL visibility in `README.md` (top banner,
+    zero-install section, further reading).
+
+- **Multi-Backend CLI Standardization & Help Completeness**:
+  - Standardized CLI syntax, argument parsing, options, and help text
+    (`-h` / `--help`) across tools in `backends/*/bin/` and
+    `objects/`.
+  - Consolidated `backends/llvm/bin/` (`k-llvm.mjs`, `k-llvm-build.mjs`,
+    `k-llvm-compile.mjs`) to mirror `backends/wasm/bin/`.
+  - Aligned `--lib` and `--export` options across LLVM, Wasm, and object
+    tools.
+  - Documented explicit defaults (such as cache directories, Clang
+    optimization levels, and runtime modes) in CLI help texts.
+
+- **Concurrent Multi-Backend Performance Benchmarks**:
+  - Added concurrent `llvm` and `llvm-pkir` execution lanes across all
+    performance suites (`perf:ieee`, `perf:int`, `perf:poly`,
+    `perf:mult`).
+  - Documented benchmark findings and throughput in `JOURNAL.md`.
+
 ## [7.0.2] — 2026-10-13 — LLVM Squaring Scaling & Backend Streamlining
 
 ### Features & Performance Improvements

package/DOCS/FILE_FORMATS.md

@@ -1,20 +1,30 @@
 # File Formats in the K Universe
 
-The `k` toolchain operates across four primary file formats: `.k`,
-`.ko`, `.klib`, and `.kvm`. Each format occupies a distinct position
-in the pipeline from human-authored source to ahead-of-time (AOT)
-type-checked artifacts and JIT/native execution.
+The `k` toolchain operates across five primary file formats: `.k`,
+`.ko`, `.klib`, `.kvm`, and `.pkir`. Each format occupies a distinct
+position in the pipeline from human-authored source to ahead-of-time
+(AOT) type-checked artifacts, intermediate representations, and
+JIT/native execution.
 
 ---
 
 ## Overview & Comparison
 
-| Extension  | Role                               | Internal Storage                                | Entrypoint (`main`)            | Generated By                      | Consumed By                                             |
-|:----------:|:-----------------------------------|:------------------------------------------------|:-------------------------------|:----------------------------------|:--------------------------------------------------------|
-| **`.k`**   | Human-readable source code         | UTF-8 plain text                                | Optional (trailing expression) | Editor, `k-decompile`             | `k.mjs`, `k-repl`, `k-compile`, backends                 |
-| **`.ko`**  | Executable compiled object         | Binary (`KOBJ\n` + JSON payload)                | **Required** (`"__main__"`)    | `k-compile`, REPL (`:ko`)         | `k.mjs`, `k-wasm-compile`, `k-llvm-build`, inspect tools  |
-| **`.klib`**| Reusable compiled library          | UTF-8 JSON payload                              | **None** (`null`)              | `k-compile --lib`, REPL (`:klib`) | REPL (`:load`), CLI linkers (`--lib`)                    |
-| **`.kvm`** | Polymorphic kVM bytecode template  | JSON / CBOR (`format: "k-vm", layer: "KVM-P"`)  | Per-relation functions         | `k-compile --format kvm`, kvm.mjs | `kvm.mjs`, `k-wasm-compile`, `k-llvm-build`              |
+| Extension | Role | Format / Magic | Entrypoint |
+|:---:|:---|:---|:---|
+| **`.k`** | Human source code | UTF-8 plain text | Optional |
+| **`.ko`** | Executable object | Binary (`KOBJ\n`) | Required |
+| **`.klib`** | Reusable library | UTF-8 JSON | None |
+| **`.kvm`** | Polymorphic kVM IR | `k-vm` (KVM-P) | Relations |
+| **`.pkir`** | Polymorphic IR | `pk-ir` (v1) | Relations |
+
+| Extension | Generated By | Consumed By |
+|:---:|:---|:---|
+| **`.k`** | Editor, `k-decompile` | `k.mjs`, `k-compile`, backends |
+| **`.ko`** | `k-compile`, REPL (`:ko`) | `k.mjs`, backends, tools |
+| **`.klib`** | `k-compile --lib` | REPL, linkers (`--lib`) |
+| **`.kvm`** | `k-compile --format kvm` | `kvm.mjs`, backends |
+| **`.pkir`** | `compileRelationToPKIR` | `executePKIR`, specializer |
 
 ### Format Transformation Workflow
 
@@ -30,20 +40,22 @@ flowchart TD
         KO[".ko (Executable Object)"]
     end
 
-    subgraph IR_Artifacts ["Register IR"]
-        KVM[".kvm (Polymorphic Template)"]
+    subgraph IR_Artifacts ["Intermediate Representations"]
+        KVM[".kvm (kVM Bytecode)"]
+        PKIR[".pkir (Polymorphic IR)"]
     end
 
     subgraph Targets ["Execution Targets"]
         WASM[".wasm (WebAssembly)"]
         NATIVE["Native Binary"]
-        INTERP["Interpreter (k.mjs / kvm.mjs)"]
+        INTERP["Interpreter / kVM / PKIR"]
     end
 
     %% Transformations from .k
     K -->|"k-compile"| KO
     K -->|"k-compile --lib"| KLIB
     K -->|"k-compile --format kvm"| KVM
+    K -->|"compileRelationToPKIR"| PKIR
     K -->|":load"| REPL
 
     %% REPL exports
@@ -60,11 +72,22 @@ flowchart TD
     KO -->|"k-llvm-build"| NATIVE
     KO -->|"k.mjs"| INTERP
     KO -->|"k-decompile"| K
+    KO -->|"compileRelationToPKIR"| PKIR
 
     %% Lowering from .kvm
     KVM -->|"k-wasm-compile"| WASM
     KVM -->|"k-llvm-build"| NATIVE
     KVM -->|"kvm.mjs / specializeKVM"| INTERP
+
+    %% Execution and Specialization from .pkir
+    PKIR -->|"executePKIR + AccessTable"| INTERP
+    PKIR -->|"specializePKIR"| PKIR_SPEC["Specialized PKIR"]
+    PKIR -->|"k-wasm-compile"| WASM
+    PKIR -->|"k-llvm-compile"| LLVM[".ll (LLVM IR)"]
+    PKIR -->|"k-llvm-build"| NATIVE
+    PKIR_SPEC -->|"executePKIR"| INTERP
+    PKIR_SPEC -->|"backend lowering"| WASM
+    PKIR_SPEC -->|"backend lowering"| NATIVE
 ```
 
 ---
@@ -365,6 +388,204 @@ k-compile add5.ko add5.kvm
 
 ---
 
+## 5. `.pkir` — Polymorphic K Intermediate Representation File
+
+### What It Is
+`.pkir` is a state-indexed, slot-addressed intermediate representation
+artifact (`format: "pk-ir"`, `version: 1`).
+
+While `.kvm` models imperative register control-flow using symbolic
+field strings (`project_field`, `inst.label`), `.pkir` addresses the
+fundamental data-layout problem for polymorphic functions. It unifies
+streaming bitstream processing and random-access memory tables:
+
+1. **Slot Addressing & State Indexing:**
+   Eliminates all runtime string lookups (`val.product[label]`). Every
+   field access is compiled to an integer slot relative to an abstract
+   pattern state (`project_slot`, `slot`, `state`). Every variant test
+   is compiled to a branch index (`match_variant`, `branchId`, `state`).
+
+2. **Duality of Transducers & Witnesses:**
+   - **Streaming Pushdown Transducers:** Navigates prefix-free serialized
+     wire streams using static skip counts (`skipCount`). Erasing and
+     linear operations execute in $O(\text{depth})$ stack memory with zero
+     node materialization.
+   - **Layout Witnesses (`AccessTable`):** Operates on random-access
+     memory arenas. Given any concrete input schema $S$ and function
+     access profile $P$, `buildAccessTable(S, P)` resolves physical byte
+     offsets $\Omega(S, P)$ in a single array load.
+
+3. **Hybrid `KVal` Fat-Pointer Representation:**
+   Values remain compact serialized wire slices by default. When
+   non-linear access occurs (e.g. repeated reads or field reordering),
+   demand-driven **Spine Promotion** materializes only the outer product
+   node into an arena record while leaving child payloads as untouched
+   slices. Opaque payloads are never re-encoded or reconstructed.
+
+### Payload Structure
+
+A `.pkir` artifact is structured as follows:
+
+```json
+{
+  "format": "pk-ir",
+  "version": 1,
+  "name": "extract_id",
+  "inputReg": "%v0",
+  "outputReg": "%v1",
+  "inputPattern": 3,
+  "outputPattern": 4,
+  "patternGraph": { ... },
+  "accessProfile": {
+    "0": { "fields": ["id"], "variants": [] },
+    "3": { "fields": [], "variants": ["click", "sensor", "ping"] }
+  },
+  "instructions": [
+    {
+      "op": "union",
+      "dest": "%v1",
+      "src": "%v0",
+      "branches": [ ... ]
+    }
+  ]
+}
+```
+
+#### Instruction Set Summary
+- `id`: Register assignment (`{ op: "id", dest, src }`).
+- `project_slot`: State-indexed slot projection
+  (`{ op: "project_slot", dest, src, state, slot, label }`).
+- `match_variant`: State-indexed variant match
+  (`{ op: "match_variant", dest, src, state, tag, branchId }`).
+- `make_variant`: Variant construction
+  (`{ op: "make_variant", dest, src, tag }`).
+- `product`: Product construction with sub-instruction streams.
+- `union`: Multi-branch selection with failure backtracking.
+- `call_rel`: Relation call / recursion
+  (`{ op: "call_rel", dest, ref, src }`).
+- `fail`: Unconditional branch failure (`{ op: "fail" }`).
+- **Specialized Instructions:**
+  - `project_offset`: Direct memory offset projection:
+    `{ op: "project_offset", dest, src, offset, skipCount, childState }`.
+  - `match_tag_imm`: Immediate discriminator test
+    (`{ op: "match_tag_imm", dest, src, tagId, branchId, childState }`).
+
+### Why `.pkir` Exists (What You Gain)
+
+1. **Elimination of the Envelope Tax (26x Speedup):**
+   Previous polymorphic execution required wrapping values in dynamic
+   envelopes $(v, P_v)$, suffering a 26x performance penalty during
+   envelope packing/unpacking (`scripts/perf-poly.mjs`). PKIR navigates
+   heterogeneous records directly via layout witnesses without envelopes.
+
+2. **Sub-30µs Specialization Latency:**
+   `specializePKIR` bakes static byte offsets into instructions and prunes
+   dead union branches in ~27 microseconds (compared to 100–1500 ms for
+   traditional AOT/Wasm compilation), enabling instant JIT tiering.
+
+3. **Zero Payload Re-Encoding:**
+   Preserves arbitrary opaque payloads (`data: payload_i`) as byte slices
+   without traversal or re-serialization during recursive list
+   processing (`search`, `poly_rev`).
+
+4. **1:1 Machine Code Lowering:**
+   Specialized PKIR instructions map directly to WebAssembly (`i64.load
+   offset=C`, `i32.eq`) and LLVM machine instructions without dynamic
+   runtime fallbacks (`@k_product_get_n`).
+
+### How to Generate
+
+In JavaScript / Node.js API:
+```javascript
+import { compileRelationToPKIR } from "@fraczak/k/backend-api.mjs";
+
+const pkir = compileRelationToPKIR(relDef, "extract_id");
+```
+
+### How to Consume
+
+- **Direct Execution with AccessTable:**
+  ```javascript
+  import {
+    executePKIR,
+    buildAccessTable,
+    valueToKVal,
+    kvalToValue
+  } from "@fraczak/k/backend-api.mjs";
+
+  const table = buildAccessTable(schemaS, patternP);
+  const kval = valueToKVal(inputVal, table.rootState);
+  const resKval = executePKIR(pkir, kval, table);
+  const result = kvalToValue(resKval);
+  ```
+
+- **Monomorphic Specialization (`specializePKIR`):**
+  ```javascript
+  import {
+    specializePKIR,
+    executePKIR
+  } from "@fraczak/k/backend-api.mjs";
+
+  // Inlines static byte offsets and prunes dead union branches:
+  const specialized = specializePKIR(pkir, table);
+
+  // Executes with zero table overhead:
+  const result = executePKIR(specialized, kval, null);
+  ```
+
+- **Lowering to WebAssembly (`compileWasmArtifactFromPKIR`):**
+  ```javascript
+  import {
+    compileWasmArtifactFromPKIR
+  } from "@fraczak/k/backend-api.mjs";
+
+  const wasmBytes = await compileWasmArtifactFromPKIR(pkir);
+  ```
+  CLI commands:
+  ```bash
+  # Compile .pkir directly to WebAssembly binary:
+  node backends/wasm/bin/k-wasm-compile.mjs input.pkir -o prog.wasm
+
+  # Lower .k source via PKIR to WebAssembly:
+  node backends/wasm/bin/k-wasm-compile.mjs --pkir prog.k -o prog.wasm
+  ```
+
+- **Lowering to LLVM IR & Native Binaries:**
+  ```javascript
+  import {
+    compileLLVMArtifactFromPKIR
+  } from "@fraczak/k/backend-api.mjs";
+  import {
+    compilePKIRToExecutable,
+    compilePKIRAndRun
+  } from "@fraczak/k/backends/llvm/src/executable.mjs";
+
+  // Compile PKIR directly to LLVM IR string:
+  const llvmIR = compileLLVMArtifactFromPKIR(pkir);
+
+  // Compile PKIR directly to a native executable:
+  compilePKIRToExecutable(pkir, "./prog.exe");
+
+  // Compile and execute in one step:
+  const res = compilePKIRAndRun(pkir, { input: inputVal });
+  ```
+  CLI commands:
+  ```bash
+  # Compile .pkir directly to LLVM IR:
+  node backends/llvm/bin/k-llvm-compile.mjs input.pkir -o prog.ll
+
+  # Lower .k source via PKIR to LLVM IR:
+  node backends/llvm/bin/k-llvm-compile.mjs --pkir prog.k -o prog.ll
+
+  # Compile .pkir directly to native executable:
+  node backends/llvm/bin/k-llvm-build.mjs input.pkir -o prog.exe
+
+  # Lower .k source via PKIR to native executable:
+  node backends/llvm/bin/k-llvm-build.mjs --pkir prog.k -o prog.exe
+  ```
+
+---
+
 ## Practical Rules & Common Gotchas
 
 ### 1. Library Aliases Require Explicit `--export`
@@ -398,9 +619,11 @@ In `k`, there are no primitive numerical types or built-in integer
 literals. Identifiers such as `5` or `10` are not language keywords;
 they are defined relations (aliases) in `Examples/arithmetics.k`:
 - `5` and `10` are nullary relations (constants) of type `{} -> bits`,
-  defined as binary digit trees over `bits? = ?< {} _, bits? 0, bits? 1 > = bits?;`.
+  defined as binary digit trees over:
+  `bits? = ?< {} _, bits? 0, bits? 1 > = bits?;`.
 - Arithmetic operations (`plus`, `minus`, `times`, `inc`) operate on
-  signed integers (`int? = ?< <bits 0, bits 1, {} _>=bits "+", bits "-" >`).
+  signed integers:
+  `int? = ?< <bits 0, bits 1, {} _>=bits "+", bits "-" >`.
 - Passing a bit-representation like `5` to an integer relation requires
   applying the `int` constructor (`5 int`), producing a constant
   relation of type `{} -> int`.

package/DOCS/POLYMORPHIC_IR.md

@@ -0,0 +1,104 @@
+# Polymorphic K Intermediate Representation (PKIR)
+
+## 1. Overview
+
+PKIR (Polymorphic K Intermediate Representation) is the target execution
+contract for polymorphic `k` relations.
+
+Unlike the high-level relational AST or traditional bytecode that
+relies on runtime string lookups for field names and variant tags, PKIR
+is a **state-indexed, slot-addressed intermediate representation**.
+
+PKIR operates in two cooperating execution modes:
+1. **Generic Polymorphic Mode**: Instructions execute against a
+   precomputed `AccessTable` (Layout Witness), accessing fields and
+   variants via static slot numbers and array lookups with zero string
+   comparisons.
+2. **Specialized Monomorphic Mode**: Given an `AccessTable`, a fast linear
+   pass (`specializePKIR`) inlines offsets directly into immediate machine
+   instructions, pruning unreachable union branches for peak LLVM/Wasm
+   performance.
+
+---
+
+## 2. Core Concepts
+
+### 2.1 The Structural Access Profile
+
+Every PKIR relation is accompanied by an **Access Profile** derived from
+its pattern graph:
+- For every product pattern node $u$: a list of accessed field labels
+  defining dense, zero-based **logical slots**:
+  $$\text{fields}(u) = [ f_0, f_1, \dots, f_{k-1} ]$$
+  where slot $i$ corresponds to field $f_i$.
+- For every union pattern node $u$: a list of matched variant tags
+  defining dense **branch targets**:
+  $$\text{variants}(u) = [ v_0, v_1, \dots, v_{m-1} ]$$
+  where branch $j$ corresponds to variant $v_j$.
+
+### 2.2 Dual-Form Values (`KVal`)
+
+Values in PKIR conform to the hybrid representation:
+- **Serialized (Wire Slice)**: A pointer into an immutable prefix-free
+  bitstream, carrying a bit offset and schema state.
+- **Materialized (Arena Node)**: An allocated record in memory containing
+  word-aligned pointers to child values.
+
+---
+
+## 3. Instruction Set
+
+PKIR instructions operate over a register file (`%v0`, `%v1`, ...):
+
+### 3.1 Field & Variant Operations
+
+- **`project_slot dest, src, state, slot`**:
+  Projects field at logical `slot` of pattern `state` from `src`.
+  At runtime with `AccessTable`:
+  ```c
+  dest.ptr = src.ptr + table[src.state].slotToByteOffset[slot];
+  dest.state = table[src.state].slotToChildState[slot];
+  ```
+- **`match_tag src, state, branches`**:
+  Evaluates variant discriminator of `src` against pattern `state`.
+  At runtime with `AccessTable`:
+  ```c
+  branch_idx = table[src.state].branchMap[src.tag];
+  dest.state = table[src.state].childStateMap[src.tag];
+  jump branches[branch_idx];
+  ```
+- **`make_variant dest, src, tag_id`**:
+  Wraps `src` into a tagged union with discriminator `tag_id`.
+- **`make_product dest, [src0, src1, ...]`**:
+  Allocates a product node whose fields are registers `[src0, src1, ...]`.
+
+### 3.2 Memory & Control Operations
+
+- **`promote_spine dest, src`**:
+  If `src` is in serialized wire form, materializes only its immediate
+  outer record in the arena; children remain serialized wire slices.
+  If `src` is already materialized, acts as identity.
+- **`dup dest1, dest2, src`**:
+  Zero-copy duplication of a value handle.
+- **`call_rel dest, rel_name, src`**:
+  Invokes relation `rel_name` with input `src`.
+- **`id dest, src`**:
+  Identity move.
+- **`fail`**:
+  Signals partial function failure (backtracking or branch exit).
+
+---
+
+## 4. Execution Pipeline
+
+```text
+[AOT Phase]
+k source -> AST -> Type Pattern Graph -> PKIR Relation
+
+[Runtime Invocation with (Schema S, Payload V)]
+  1. AccessTable = buildAccessTable(S, PKIR.patternGraph)
+  2. Choose execution tier:
+     - Tier 0 (Interpreter): executePKIR(PKIR, V, AccessTable)
+     - Tier 1 (Specialized AOT/JIT): specializePKIR(PKIR, AccessTable)
+       -> direct emission to LLVM / WebAssembly
+```