@fraczak/k 6.10.3 → 6.10.5

package/CHANGELOG.md

@@ -5,6 +5,84 @@ All notable changes to this project will be documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [6.10.5] — 2026-10-08 — Adaptive Division and VFS Codecs
+
+### Arithmetic & Algorithms
+
+- **Adaptive Hybrid Division (`Examples/arithmetics.k`)**:
+  - Implemented hybrid integer division that dynamically chooses between
+    down-shift division ($|x| < 2|y|$) and restoring long division
+    ($|x| \ge 2|y|$).
+  - Down-shift division scales divisor $y$ up to match length of $x$,
+    performing trial subtraction and down-shifting in $O(1)$ per step.
+  - Crossover condition is evaluated structurally in $O(\min(|y|, |x|/2))$
+    without prior length computations.
+  - Added native number-theory operations: `mod`, Euclidean `gcd`, and
+    `lcm`.
+  - Refactored `arithmetics.k` to focus purely on arithmetic operations.
+
+### Polymorphic Lists
+
+- **Polymorphic List Library (`Examples/poly.k`)**:
+  - Encapsulated list constructors and destructors (`list?`, `nil`,
+    `cons`, `car`, `cdr`, `singleton`, `nil?`) directly in `poly.k`.
+  - Updated benchmark suites (`harness/suites/poly.mjs`,
+    `scripts/perf-poly.mjs`) to import only arithmetic symbols from
+    `arithmetics.k`.
+
+### Web REPL & Codecs
+
+- **VFS Codec Subfolder Organization**:
+  - Reorganized all codec files in the Virtual File System under the
+    `codecs/` subfolder (`codecs/int.mjs`, `codecs/json.mjs`, etc.).
+  - Added backward-compatible resolution in `repl-codecs.mjs`.
+  - Added `karatsuba-mult.k` to the VFS and example picker in `repl.html`.
+
+---
+
+## [6.10.4] — 2026-10-06 — Karatsuba Multiplication, Multi-Backend Harness, and CLI Help
+
+### Arithmetic & Algorithms
+
+- **Karatsuba Multiplication (`Examples/karatsuba-mult.k`)**:
+  - Implemented standalone, minimal Karatsuba integer multiplication with
+    asymptotic time complexity $O(n^{\log_2 3}) \approx O(n^{1.585})$.
+  - Utilizes unary counter (`$unat`) splitting with a 512-bit threshold
+    switching to grade-school multiplication (`times_bits`) for small
+    inputs.
+  - Implements direct bit-level arithmetic (`plus_bits`, `minus_bits`,
+    `times_bits`) and direct $2m$ bit shifting (`shift2_bits_u`).
+  - Added dedicated test suite in `tests/test-karatsuba.mjs` verifying
+    squaring cases and mixed-sign pair multiplications against BigInt
+    oracle.
+
+### Performance & Benchmarking
+
+- **Multiplication Performance Harness**
+  (`scripts/perf-multiplication.mjs`):
+  - Added multi-backend performance runner comparing `times` and
+    `karatsuba` across LLVM, Wasm, kVM, and JS engines.
+  - Supports `--mode squaring` (doubling bit width per step) and
+    `--mode sizes` (random operands of fixed bit lengths).
+  - Includes per-step timeout protection and process RSS memory guards.
+  - Added `npm run perf:mult` script in `package.json`.
+
+### CLI Usability & Backend Enhancements
+
+- **Unified `-h` and `--help` CLI Documentation**:
+  - Added comprehensive option and environment variable documentation
+    across all benchmark, performance, and test scripts in `scripts/`
+    and `backends/*/scripts/`.
+- **LLVM Direct `.kvm` Bytecode Compilation**:
+  - Added `compileKVMToExecutable` in `backends/llvm/src/executable.mjs`,
+    enabling direct binary compilation from lowered `.kvm` IR.
+- **Runtime Mark Compaction Fixes**:
+  - Updated LLVM runtime compaction tracking (`%k_rt_tail_mark`) to
+    accurately preserve base and last compaction arena marks in
+    tail loops.
+
+---
+
 ## [6.10.3] — 2026-10-04 — Unified File Formats Specification and Documentation Modernization
 
 ### Documentation & Core Architecture

package/DOCS/FILE_FORMATS.md

@@ -14,7 +14,7 @@ type-checked artifacts and JIT/native execution.
 | **`.k`**   | Human-readable source code         | UTF-8 plain text                                | Optional (trailing expression) | Editor, `k-decompile`             | `k.mjs`, `k-repl`, `k-compile`, backends                 |
 | **`.ko`**  | Executable compiled object         | Binary (`KOBJ\n` + JSON payload)                | **Required** (`"__main__"`)    | `k-compile`, REPL (`:ko`)         | `k.mjs`, `k-wasm-compile`, `k-llvm-build`, inspect tools  |
 | **`.klib`**| Reusable compiled library          | UTF-8 JSON payload                              | **None** (`null`)              | `k-compile --lib`, REPL (`:klib`) | REPL (`:load`), CLI linkers (`--lib`)                    |
-| **`.kvm`** | Polymorphic kVM bytecode template  | JSON / CBOR (`format: "k-vm", layer: "KVM-P"`)  | Per-relation functions         | `k-compile --format kvm`, kvm.mjs | kVM runtime, JIT specializer, `k-wasm-compile`           |
+| **`.kvm`** | Polymorphic kVM bytecode template  | JSON / CBOR (`format: "k-vm", layer: "KVM-P"`)  | Per-relation functions         | `k-compile --format kvm`, kvm.mjs | `kvm.mjs`, `k-wasm-compile`, `k-llvm-build`              |
 
 ### Format Transformation Workflow
 
@@ -63,6 +63,7 @@ flowchart TD
 
     %% Lowering from .kvm
     KVM -->|"k-wasm-compile"| WASM
+    KVM -->|"k-llvm-build"| NATIVE
     KVM -->|"kvm.mjs / specializeKVM"| INTERP
 ```
 
@@ -276,21 +277,38 @@ Export all currently loaded relations and types into a reusable library:
 `.kvm` is a polymorphic register-IR template artifact
 (`format: "k-vm"`, `layer: "KVM-P"`, `isPolymorphic: true`).
 
-#### How `.kvm` Differs from `.ko` and `.klib`:
-- **Abstraction Level:** `.ko` and `.klib` store high-level relational
-  expressions (KIR) and structural type graphs. In contrast, `.kvm` is
-  lowered into explicit register-based virtual machine instructions
-  (`alloc`, `make_variant`, `project_variant`, `union`, `call`, and
-  tail loops).
-- **Polymorphic Template vs. Fixed Object:**
-  - `.ko` is a concrete executable bound to an entrypoint (`main`).
-  - `.klib` is a library of high-level relation definitions for linking.
-  - `.kvm` stores compiled bytecode functions paired with principal
-    pattern graphs ($P_{in}, P_{out}$).
-- **Instant Specialization:** Instead of running the compiler's full
-  constraint-derivation engine, the runtime specializer (`specializeKVM`)
-  prunes dead union branches and folds redundant guards on-the-fly
-  against concrete input envelopes $(v, P_v)$.
+#### Why `.kvm` Exists (What You Gain):
+
+1. **Relational AST vs. Linear Register Machine:**
+   `.ko` and `.klib` store high-level relational combinators (compositions
+   `(f g)`, products `{a x, b y}`, unions `<f, g>`) and structural type
+   graphs. They contain no virtual registers, instruction sequences, or
+   control-flow branches. `.kvm` transforms this functional tree into a
+   linear 3-address virtual register machine IR (`alloc`, `project_field`,
+   `make_variant`, `project_variant`, `jump_if_fail`, `call`, and loops).
+
+2. **$1 \times N$ Backend Decoupling:**
+   Rather than having every code generator (LLVM, WebAssembly, ARM64,
+   and the JIT interpreter) independently lower the relational tree and
+   re-implement type-guided control flow, relational-to-imperative
+   lowering occurs once in `kvm.mjs`. All backends consume the exact same
+   ~10 linear kVM instructions (`kvm2llvm.mjs`, `kvm2wasm.mjs`,
+   `kvm2arm64.mjs`, `kvm.mjs`).
+
+3. **Microsecond Polymorphic Specialization (`layer: "KVM-P"`):**
+   In polymorphic programs, specializing `.ko` requires re-running the
+   compiler's full structural type derivation and constraint convergence
+   engine. A `.kvm` template stores pre-compiled register bytecode with
+   principal pattern graphs. The runtime specializer (`specializeKVM`)
+   prunes dead branches and folds redundant guards against concrete
+   runtime input envelopes $(v, P_v)$ in microseconds with zero AST
+   recompilation overhead.
+
+4. **Inspectable & Cacheable Register IR:**
+   `.kvm` produces a human- and machine-readable JSON or CBOR format.
+   You can inspect virtual register assignments, verify dead-branch
+   elimination, diff compiler output across runs, or cache the lowered IR
+   to bypass frontend compilation when building native or WASM targets.
 
 ### How to Generate
 
@@ -309,7 +327,24 @@ k-compile add5.ko add5.kvm
 
 ### How to Consume
 
+- **Direct lowering from `.kvm` to Native Binaries (`k-llvm-build`):**
+  Compile a `.kvm` template directly into a native executable:
+  ```bash
+  k-llvm-build add5.kvm add5-bin
+  echo "10" | ./codecs/int.mjs --parse | \
+    ./add5-bin | \
+    ./codecs/int.mjs --print
+  # Output: 15
+  ```
+
+- **Direct lowering from `.kvm` to LLVM IR (`k-llvm-compile`):**
+  Emit LLVM assembly directly from the register IR:
+  ```bash
+  k-llvm-compile add5.kvm add5.ll
+  ```
+
 - **Direct lowering from `.kvm` to WebAssembly (`k-wasm-compile`):**
+  Compile a `.kvm` template directly into a standalone `.wasm` module:
   ```bash
   k-wasm-compile add5.kvm add5.wasm
   echo "10" | ./codecs/int.mjs --parse | \

package/Examples/arithmetics.k

@@ -4,14 +4,6 @@ swap_yes_no = < /yes |no, /no |yes > $yes_no;
 yes = {}|yes $yes_no;
 no = yes swap_yes_no;
 
-list? = ?< {} nil, {X car, Y cdr} cons > = Y;
-nil = {}|nil list?;
-singleton = {() car, nil cdr} |cons list?;
-cons = {.car car, .cdr cdr} |cons list?;
-nil? = list? / nil nil;
-car = list? /cons .car;
-cdr = list? /cons .cdr;
-
 $ bits = < {} _, bits 0, bits 1 >;
 
 0b = {} | _;
@@ -70,29 +62,10 @@ inOrder_ = ${bits x, bits y, yes_no inOrder}
 
 GT = {inOrder /no if, () then} . then;
 
-EQ = GT {.x y, .y x} GT;
-
-toBitList = {inv x, nil r} _toBitList; 
-_toBitList = < 
-  {.x empty? if, .r then}.then, 
-  <
-    {.x /0 x, {0 car, .r cdr} cons r}, 
-    {.x /1 x, {1 car, .r cdr} cons r}
-  > _toBitList 
->;
-fromBitList = {() l, 0b r} _fromBitList inv $bits;
-_fromBitList =
-  <
-    {.l nil? if, .r then} .then,
-    <
-      {.l car /0 if, {.l cdr l, .r |0 r} then} .then,
-      {.l car /1 if, {.l cdr l, .r |1 r} then} .then
-    > _fromBitList
-  >;
+EQ = {inOrder /yes if, {.x y, .y x} inOrder /yes and, () then} .then;
 
 --------------------------------------
 $ int = < bits '+', bits '-' >;
-$ ints = <{int car, ints cdr} cons, {} nil>;
 
 remove_leading_zeros = <
   /0 remove_leading_zeros, 
@@ -102,7 +75,7 @@ nat = remove_leading_zeros <empty? 0, ()>;
 int = nat | '+' $int;
 abs = $int </'+',/'-'>;
 
-pos? = $int {dec /'+' if, () then} .then;
+pos? = $int {/'+' remove_leading_zeros /1 if, () then} .then;
 neg? = $int {/'-' if, () then} .then; 
 zero_int? = $int {/'+' /0 empty? if, () then} .then;
 minus_one? = $int {/'-' /1 empty? if, () then} .then;
@@ -125,12 +98,12 @@ dec = $int <
   /'-' {inv x, 0b y} _inc nat |'-'
 > $int;
 
-_dec = ?{X x, X y, ...} <
+_dec = ${bits x, bits y} <
     {.x /1 x, .y |0  y} concat_, 
     {.x /0 x, .y |1 y} _dec
-> ?X;
+>;
 
-$carry = <{} 0, {} 1>;
+$ carry = <{} 0, {} 1>;
 c0 = {}|0 $carry;
 c1 = {}|1 $carry;
 
@@ -218,7 +191,7 @@ times = ${int x, int y}
       sign_minus
     > sign,
     {
-      0 result,
+      0b result,
       .x abs inv _x,
       .y abs y
     } _times result
@@ -249,105 +222,244 @@ _times =
 ;
 
 
-mod =
-  ${bits x, bits y}
-  -- _log!
-  < 
-    -- if y > x then (x mod y) = x 
-    {.y x, .x y} GT .y,
-    -- else, i.e.
-    -- if x >= y then (x mod y) = ((x-y mod 2y) mod y)
-    { 
-      {  
-        {.x int x, .y int y} minus /"+" x, 
-        .y inv |0  inv y
-      } mod x, 
-      .y y 
-    } mod
-  >
-; 
 
 
-// gratest common divisor
-gcd = ${bits x, bits y}
-  <
-    {.y x, .x y} GT, 
-    ()
-  > _gcd
-;
 
-_gcd = <
-  { .y /0 if, .x then } .then,
-  { .y x, mod y } _gcd
->;
+// ----------------------------------------------------------------------
+// Integer Division: Adaptive Hybrid Algorithm
+//
+// Combines two complementary strategies based on the bit-length
+// comparison between dividend x and divisor y (|x| vs |y|):
+//
+// 1. Down-Shift Division (|x| < 2|y|, y is large relative to x):
+//    - Scales y up to y_max = y * 2^k to match length(x) via 'y inv'
+//      and prepending k zeros (|0).
+//    - In each step: trial subtraction x = x - y_scaled, then down-shifts
+//      y_scaled by 1 bit in O(1) by popping the outer zero (/_0).
+//    - Runs in exactly (|x| - |y| + 1) iterations. When y ~= x, this
+//      terminates in 1-2 steps rather than looping through all bits of x.
+//
+// 2. Restoring Division (|x| >= 2|y|, y is small relative to x):
+//    - Consumes bits of x from MSB to LSB into remainder accumulator r.
+//    - Remainder r stays small (r <= 2y), so trial subtractions operate
+//      on tiny numbers of length |y| rather than full-length |x|.
+//
+// The crossover condition |x| < 2|y| is evaluated structurally in
+// O(min(|y|, |x|/2)) by consuming 2 bits of x for every 1 bit of y.
+// ----------------------------------------------------------------------
+
+// Helpers for Restoring Division (shift-and-subtract)
+shift_in_0 = inv |0 inv nat;
+shift_in_1 = inv |1 inv nat;
+sub_bits = ${bits x, bits y} {.x inv x, .y inv y, c0 c} _minus inv nat;
+
+_div_restoring =
+  ${bits x, bits y, bits r, bits q}
+  <
+    {.x empty? if, {.q inv int div, .r rem} then} .then,
+    <
+      {.x /0 x, .r shift_in_0 r, .y y, .q q},
+      {.x /1 x, .r shift_in_1 r, .y y, .q q}
+    >
+    {
+      .x x,
+      .y y,
+      .q q,
+      .r r,
+      {.y x, .r y} inOrder test
+    }
+    <
+      {.test /yes if, {.x x, .y y, .q |1 q, {.r x, .y y} sub_bits r} then} .then,
+      {.test /no  if, {.x x, .y y, .q |0 q, .r r} then} .then
+    >
+    _div_restoring
+  >;
 
+// Helpers for Down-Shift Division (align and shift divisor down)
+// Phase 1: pad _y with |0 to match length of x, counting shift steps
+_pad_down = ${bits x, bits _y, bits steps}
+  <
+    {.x empty? if, {._y _y, .steps steps} then} .then,
+    {
+      .x </0, /1> x,
+      ._y |0 _y,
+      .steps |0 steps
+    } _pad_down
+  >;
 
+_align_down = ${bits x, bits y, bits _y}
+  <
+    // Both x and y have bits remaining: advance both
+    {
+      .x </0, /1> x,
+      .y </0, /1> y,
+      ._y _y
+    } _align_down,
 
-// div
+    // y is empty while x remains: pad _y with shift zeros
+    {
+      .y empty? if,
+      {.x x, ._y _y, 0b steps} _pad_down then
+    } .then,
 
-log2_floor =
-  ${bits x, bits step} 
--- finds the largest k, such that 'step * 2^k <= x',
--- returns k1_, i.e., 0 if x < step, and k1_ otherwise.
-  {.x x, .step step, 0 log} log2_floor_;
+    // Both x and y are empty: lengths equal (0 shift steps)
+    {
+      .x empty? if,
+      .y empty? and,
+      {._y _y, 0b steps} then
+    } .then
+  >;
 
-log2_floor_ = <
-  {{.step x,.x y} GT if, .log then} .then,
-  {.x x, {.step x,0 y} GT {.x x,2 y} {.x int x, .y int y} times abs step, .log int inc abs log} log2_floor_
->;
+// Phase 2: down-shift trial subtraction
+_div_down = ${bits x, bits _y, bits steps, bits q}
+  {
+    .x x,
+    ._y _y,
+    .steps steps,
+    .q q,
+    ._y inv nat y_scaled
+  }
+  {
+    .x x,
+    ._y _y,
+    .steps steps,
+    .q q,
+    .y_scaled y_scaled,
+    {.y_scaled x, .x y} inOrder test
+  }
+  <
+    // If x >= y_scaled, subtract and emit quotient bit 1
+    {
+      .test /yes if,
+      {
+        {.x int x, .y_scaled int y} minus /+ x,
+        .q |1 q,
+        ._y _y,
+        .steps steps
+      } then
+    } .then,
+    // If x < y_scaled, keep x and emit quotient bit 0
+    {
+      .test /no if,
+      {.x x, .q |0 q, ._y _y, .steps steps} then
+    } .then
+  >
+  <
+    // If no more shift steps remaining, division is complete
+    {
+      .steps empty? if,
+      {.q inv int div, .x rem} then
+    } .then,
+    // Otherwise down-shift: remove the last 0 from _y and steps
+    {
+      .x x,
+      ._y /0 _y,
+      .steps /0 steps,
+      .q q
+    } _div_down
+  >;
 
+$ choose_div = < {} down_shift, {} restore >;
 
-x_times_2_to_k =
-  ${bits x, bits k}
-  {.x inv _x, .k int k} _x_times_2_to_the_power_of_k inv
-;
+// Evaluates whether |x| < 2|y| by advancing 2 bits of x per 1 bit of y
+_check_2y = ${bits x, bits y}
+  <
+    // x is empty -> |x| < 2|y| (choose down_shift)
+    {.x empty? if, {} | down_shift $choose_div then} .then,
+    // y is empty while x remains -> |x| >= 2|y| (choose restore)
+    {.y empty? if, {} | restore $choose_div then} .then,
+    // x has only 1 bit left while y remains -> |x| < 2|y|
+    {.x </0, /1> empty? if, {} | down_shift $choose_div then} .then,
+    // Advance 2 bits from x and 1 bit from y
+    {
+      .x </0, /1> </0, /1> x,
+      .y </0, /1> y
+    } _check_2y
+  >;
 
-_x_times_2_to_the_power_of_k = <
-  {.k zero_int? if, ._x then} .then,
-  {._x |0  _x, .k dec k} _x_times_2_to_the_power_of_k
->
-;
- 
-div = 
-  ${int x, int y} 
-  // remember sign
+// Public division interface
+div = ${int x, int y}
   {
     <
       { .x/'+' if, .y get_sign then } .then,
-      { .x/'-' if, .y  swap_sign get_sign then } .then
+      { .x/'-' if, .y swap_sign get_sign then } .then
     > sign,
-    {.x abs x, .y abs step, 0 int res} _div res_wo_sign
+    .x abs remove_leading_zeros x_norm,
+    .y abs remove_leading_zeros y_norm
   }
-  < 
-    {.sign/'+' if, {.res_wo_sign.div div, .res_wo_sign.rem rem} then} .then,
-    { .res_wo_sign.div swap_sign div, .res_wo_sign.rem rem }
-  > 
-  ;
-
-_div =
-  {
-    .x x,
-    .step step,
-    .res res,
-    {.x x, .step step} log2_floor log
-  }
-  < 
-    { .log int zero_int? if, {.res div, .x rem} then } .then,
-    { 
-      {1 x, .log int dec abs k} x_times_2_to_k factor,
-      () i
-    }
+  <
+    // Case 1: x == 0 -> quotient 0, remainder 0
+    {
+      .x_norm empty? if,
+      { 0 int div, 0 rem } then
+    } .then,
+    // Case 2: |x| < |y| -> x < y, quotient 0, remainder x
+    {
+      {.y_norm x, .x_norm y} GT if,
+      { 0 int div, .x_norm rem } then
+    } .then,
+    // Case 3: general division via adaptive hybrid choice
     {
-      {.i.x int x, {.i.step int x, .factor int y} times y} minus abs x, 
-      .i.step step, 
-      {.i.res x, .factor int y} plus res
-    } _div
+      .sign sign,
+      {
+        .x_norm x,
+        .y_norm y,
+        {.x_norm x, .y_norm y} _check_2y choice
+      }
+      <
+        // If |x| < 2|y|, use down-shift division
+        {
+          .choice /down_shift if,
+          {
+            .x x,
+            {.x x, .y y, .y inv _y} _align_down aligned
+          }
+          {
+            .x x,
+            .aligned._y _y,
+            .aligned.steps steps,
+            0b q
+          } _div_down then
+        } .then,
+        // If |x| >= 2|y|, use restoring long division
+        {
+          .choice /restore if,
+          {.x x, .y y, 0 r, 0b q} _div_restoring then
+        } .then
+      > res
+    }
+    <
+      {.sign/'+' if, {.res.div div, .res.rem rem} then} .then,
+      { .res.div swap_sign div, .res.rem rem }
+    >
   >;
 
 
+// ----------------------------------------------------------------------
+// Number Theory: Modulo, Greatest Common Divisor, Least Common Multiple
+// ----------------------------------------------------------------------
+
+// Modulo: remainder of x divided by y in $bits, delegated to div
+mod = ${bits x, bits y}
+  {{.x int x, .y int y} div .rem rem} .rem;
+
+// Greatest common divisor (Euclidean algorithm)
+gcd = ${bits x, bits y}
+  <
+    {.y x, .x y} GT, 
+    ()
+  > _gcd
+;
+
+_gcd = <
+  { .y remove_leading_zeros empty? if, .x nat then } .then,
+  { .y x, mod y } _gcd
+>;
+
+// Least common multiple: (x / gcd(x, y)) * y
 lcm = 
   ${bits x, bits y}
-  {{.x int x, .y int y} times x, gcd int y} div .div abs
+  {{.x int x, gcd int y} div .div x, .y int y} times abs
 ;
 
 factorial = {()n, 1 int acc} factorial_;