@fraczak/k 6.10.2 → 6.10.4

package/CHANGELOG.md

@@ -5,6 +5,79 @@ All notable changes to this project will be documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [6.10.4] — 2026-10-06 — Karatsuba Multiplication, Multi-Backend Harness, and CLI Help
+
+### Arithmetic & Algorithms
+
+- **Karatsuba Multiplication (`Examples/karatsuba-mult.k`)**:
+  - Implemented standalone, minimal Karatsuba integer multiplication with
+    asymptotic time complexity $O(n^{\log_2 3}) \approx O(n^{1.585})$.
+  - Utilizes unary counter (`$unat`) splitting with a 512-bit threshold
+    switching to grade-school multiplication (`times_bits`) for small
+    inputs.
+  - Implements direct bit-level arithmetic (`plus_bits`, `minus_bits`,
+    `times_bits`) and direct $2m$ bit shifting (`shift2_bits_u`).
+  - Added dedicated test suite in `tests/test-karatsuba.mjs` verifying
+    squaring cases and mixed-sign pair multiplications against BigInt
+    oracle.
+
+### Performance & Benchmarking
+
+- **Multiplication Performance Harness**
+  (`scripts/perf-multiplication.mjs`):
+  - Added multi-backend performance runner comparing `times` and
+    `karatsuba` across LLVM, Wasm, kVM, and JS engines.
+  - Supports `--mode squaring` (doubling bit width per step) and
+    `--mode sizes` (random operands of fixed bit lengths).
+  - Includes per-step timeout protection and process RSS memory guards.
+  - Added `npm run perf:mult` script in `package.json`.
+
+### CLI Usability & Backend Enhancements
+
+- **Unified `-h` and `--help` CLI Documentation**:
+  - Added comprehensive option and environment variable documentation
+    across all benchmark, performance, and test scripts in `scripts/`
+    and `backends/*/scripts/`.
+- **LLVM Direct `.kvm` Bytecode Compilation**:
+  - Added `compileKVMToExecutable` in `backends/llvm/src/executable.mjs`,
+    enabling direct binary compilation from lowered `.kvm` IR.
+- **Runtime Mark Compaction Fixes**:
+  - Updated LLVM runtime compaction tracking (`%k_rt_tail_mark`) to
+    accurately preserve base and last compaction arena marks in
+    tail loops.
+
+---
+
+## [6.10.3] — 2026-10-04 — Unified File Formats Specification and Documentation Modernization
+
+### Documentation & Core Architecture
+
+- **Unified File Formats Specification (`DOCS/FILE_FORMATS.md`)**:
+  - Added comprehensive specification for all primary file formats: `.k`
+    (source), `.ko` (executable object), `.klib` (library object), and
+    `.kvm` (polymorphic register-IR template), plus target artifacts
+    (`.wasm`, native binary).
+  - Added Mermaid transformation flowchart mapping compilation, linking,
+    specialization, and execution workflows across CLI tools.
+- **AOT Type Checking & Canonical Content-Addressing**:
+  - Formally specified that AOT type checking performs relation expansion,
+    structural type derivation, and constraint convergence in advance.
+  - Clarified canonical content-addressed hashes (`@hash`) vs. local aliases,
+    confirming there is no global symbol namespace, and documented explicit
+    `--export` semantics for `--lib`.
+- **Relations as Values & Absence of Built-in Numbers**:
+  - Documented that numerical literals do not exist in `k`; names like `5`
+    or `10` are nullary relations mapping `${} -> $bits`. Arithmetic relations
+    operate on signed integer variants (`$int`), converting via `5 int`.
+  - Clarified stream transformers (open domain) vs. closed constant relations
+    (unit input domain `${}`).
+- **Documentation Reformatting**:
+  - Systematically updated and wrapped prose across `README.md` and all
+    documents in `DOCS/`, `codecs/`, `objects/`, and `tests/` to <= 74
+    characters per line.
+
+---
+
 ## [6.10.2] — 2026-10-03 — Stack-Safe Value Serializer for REPL and Web REPL
 
 ### REPL & Codecs

package/DOCS/BIG_PICTURE.md

@@ -1,28 +1,32 @@
 # Big Picture
 
-`Kernel-code` is an experiment aiming to define a "perfect programming language".
+`Kernel-code` is an experiment aiming to define a "perfect programming
+language".
 
 ## Basics
 
-The language describes "partial first-order functions" on "printable" values.
-The printable values can be seen as rooted, edge-labeled trees, intuitively similar to an
-XML or a JSON document. The values are always typed; in the current codec-aware
-runtime, a value may carry a pattern describing the family of types compatible
-with that tree.
+The language describes "partial first-order functions" on "printable"
+values. The printable values can be seen as rooted, edge-labeled trees,
+intuitively similar to an XML or a JSON document. The values are always
+typed; in the current codec-aware runtime, a value may carry a pattern
+describing the family of types compatible with that tree.
 
 ### Types (codes)
 
-Formally, a _type_ is a finite tree automaton. The type defines a set of values,
-which corresponds to the set of trees accepted by the automaton.
-Therefore, every type admits a unique normal form in terms of its minimal deterministic tree automaton.
+Formally, a _type_ is a finite tree automaton. The type defines a set of
+values, which corresponds to the set of trees accepted by the
+automaton. Therefore, every type admits a unique normal form in terms of
+its minimal deterministic tree automaton.
 
 Types are defined by two constructs:
 
 1. Product
 2. Union (also called tagged or disjoint union)
 
-> Notation: We use native k-like notation as canonical — products `{ T l, ... }` and unions `< T t, ... >`.
-> For convenience, JSON-like forms `{ l: T, ... }` and `< t: T, ... >` are also supported and equivalent in meaning.
+> Notation: We use native k-like notation as canonical — products
+> `{ T l, ... }` and unions `< T t, ... >`. For convenience, JSON-like
+> forms `{ l: T, ... }` and `< t: T, ... >` are also supported and
+> equivalent in meaning.
 
 ```bnf
 type ::= name | '{' type_label_list '}' | '<' type_label_list '>'
@@ -31,18 +35,23 @@ name ::= IDENTIFIER
 label ::= STRING
 ```
 
-Given a finite (possibly empty) set of types `t1, t2, ..., tn` and a set of pairwise different tags (labels) `l1, l2, ..., ln`,
+Given a finite (possibly empty) set of types `t1, t2, ..., tn` and a
+set of pairwise different tags (labels) `l1, l2, ..., ln`,
 
-- by `{ t1 l1, t2 l2, ..., tn ln }`, we define a _product type_ with exactly `n`
-  _projection_ functions: `.l1`, `.l2`, `...`, `.ln`.
-  Each projection function maps a value of type `{t1 l1, t2 l2, ..., tn ln}`
-  into a value of type `t1`, `t2`, `...`, `tn`, respectively.
-- by `< t1 l1, t2 l2, ..., tn ln >`, we define a _union type_ with exactly `n` projection functions `/l1`, `/l2`, `...`, `/ln`,
-  mapping a value of type `<t1 l1, t2 l2, ..., tn ln>` into a value of type `t1`, `t2`, `...`, `tn`, respectively.
+- by `{ t1 l1, t2 l2, ..., tn ln }`, we define a _product type_ with
+  exactly `n` _projection_ functions: `.l1`, `.l2`, `...`, `.ln`.
+  Each projection function maps a value of type
+  `{t1 l1, t2 l2, ..., tn ln}` into a value of type `t1`, `t2`,
+  `...`, `tn`, respectively.
+- by `< t1 l1, t2 l2, ..., tn ln >`, we define a _union type_ with
+  exactly `n` projection functions `/l1`, `/l2`, `...`, `/ln`,
+  mapping a value of type `<t1 l1, t2 l2, ..., tn ln>` into a value
+  of type `t1`, `t2`, `...`, `tn`, respectively.
 
-Intuitively, the projection functions on a product type are all total (meaning, they are defined for every value of the
-input product type), whereas in the case of a union type, exactly one projection function is defined
-for a given value of the union type.
+Intuitively, the projection functions on a product type are all total
+(meaning, they are defined for every value of the input product type),
+whereas in the case of a union type, exactly one projection function
+is defined for a given value of the union type.
 
 Some properties of these types:
 
@@ -51,26 +60,32 @@ Some properties of these types:
 
 #### Variant (union) value literal convention
 
-- Variant values are written using "|" (pipe) followed by the variant tag.
+- Variant values are written using "|" (pipe) followed by the variant
+  tag.
   - Unit variant `` `{}` `` | tag (e.g., `{} | nil`, `{} | zero`)
   - Variant with payload `v` at label `tag`: ` v | tag `
 - Example (list): with `$list = < {} nil, {X car, list cdr} cons >`:
   - Empty list []: `{}|nil`
   - Singleton `[v]`: `{ v car, {}|nil cdr } | cons`
-- Angle brackets are for type definitions and merge expressions; use "`| tag`" for a variant construction.
+- Angle brackets are for type definitions and merge expressions; use
+  "`| tag`" for a variant construction.
 
 #### Equivalence of types (bisimilarity)
 
-Two (possibly recursive) codes are equivalent iff they are bisimilar over their definition graphs. Concretely, there exists a relation B such that `(t1, t2) ∈ B` if and only if:
+Two (possibly recursive) codes are equivalent iff they are bisimilar
+over their definition graphs. Concretely, there exists a relation B
+such that `(t1, t2) ∈ B` if and only if:
 - t1 and t2 have exactly the same set of labels/tags;
 - t1 and t2 are simultaneously products or simultaneously unions;
-- for each label/tag `ℓ` in the set, the subcodes under `ℓ` are again related by B.
+- for each label/tag `ℓ` in the set, the subcodes under `ℓ` are again
+  related by B.
 
 ### Filters (patterns)
 
-A _filter_ is a way of representing a _pattern_, i.e., a set of types sharing some structure.
-For example, if I wanted to refer to all product types that have a field `f` of type `T`,
-I would write: `?{ T f, ... }`. In general, a filter is defined by:
+A _filter_ is a way of representing a _pattern_, i.e., a set of types
+sharing some structure. For example, if I wanted to refer to all product
+types that have a field `f` of type `T`, I would write:
+`?{ T f, ... }`. In general, a filter is defined by:
 
 ```bnf
 filter ::= name | '$' type_expr | '{' filter_label_list '}'
@@ -92,29 +107,36 @@ Examples:
 
 ### Partial functions
 
-A _partial function_ is a function that is not defined for every value of its domain.
+A _partial function_ is a function that is not defined for every value
+of its domain.
 
 There are three ways of combining partial functions:
 
-1. __Composition__: we write a composition of `f` and `g` as `(f  g)`,
-   meaning that if `f` is defined on value `x` producing `y`,
-   and `g` is defined on value `y` producing `z`, then `(f g)` is defined on value `x` producing `z`.
-2. __Union__: we write `< f1, f2 >` for a union of two functions `f1` and `f2`, meaning that
-   if `f1` is defined on value `x` producing `y`, then `< f1, f2 >`
-   is defined on value `x` producing `y`. Otherwise, i.e., when `f1` is not defined for `x`, `< f1, f2 >` acts as `f2`.
-3. __Product__: we write `{f1 lab1, f2 lab2 }` for a product of two functions `f1` and `f2`,
-   meaning that if `f1` and `f2` are both defined on value `x` producing `y1` and `y2`,
-   respectively, then `{f1 lab1, f2 lab2 }` is defined on value `x` producing
-   `{y1 lab1, y2 lab2}`. 
-   Otherwise, `{f1 lab1, f2 lab2 }` is not defined for `x`.
-
-Actually, composition, union, and product combine any number of partial functions, not only two. For example,
-empty composition, `()`, defines the identity function.
+1. __Composition__: we write a composition of `f` and `g` as `(f g)`,
+   meaning that if `f` is defined on value `x` producing `y`, and `g`
+   is defined on value `y` producing `z`, then `(f g)` is defined on
+   value `x` producing `z`.
+2. __Union__: we write `< f1, f2 >` for a union of two functions `f1`
+   and `f2`, meaning that if `f1` is defined on value `x` producing
+   `y`, then `< f1, f2 >` is defined on value `x` producing `y`.
+   Otherwise, i.e., when `f1` is not defined for `x`, `< f1, f2 >`
+   acts as `f2`.
+3. __Product__: we write `{f1 lab1, f2 lab2 }` for a product of two
+   functions `f1` and `f2`, meaning that if `f1` and `f2` are both
+   defined on value `x` producing `y1` and `y2`, respectively, then
+   `{f1 lab1, f2 lab2 }` is defined on value `x` producing
+   `{y1 lab1, y2 lab2}`. Otherwise, `{f1 lab1, f2 lab2 }` is not
+   defined for `x`.
+
+Actually, composition, union, and product combine any number of
+partial functions, not only two. For example, empty composition,
+`()`, defines the identity function.
 
 ### Program
 
-A `kernel-core` program is a set of definitions of types and partial functions, followed by an _expression_,
-which is the partial function the program defines.
+A `kernel-core` program is a set of definitions of types and partial
+functions, followed by an _expression_, which is the partial function
+the program defines.
 
 The language syntax is defined by:
 
@@ -132,47 +154,65 @@ expr_label_list ::= /* empty */
    | (expression label_name ',')* expression label
 ```
 
-We assume that in a `kernel-core` program the names (for types, functions, and labels) are identifiers (strings) such that:
+We assume that in a `kernel-core` program the names (for types,
+functions, and labels) are identifiers (strings) such that:
 
 1. All defined type names are distinct.
 2. All defined function names are distinct.
-3. All label names within a type definition or a `product` expression are locally distinct.
+3. All label names within a type definition or a `product` expression
+   are locally distinct.
 
-Filters and type expressions act as a type annotation, i.e., an identity
-function defined only for the values of the corresponding types.
+Filters and type expressions act as a type annotation, i.e., an
+identity function defined only for the values of the corresponding
+types.
 
 That's it.
 
-No `builtin` types, no `if` statement, no `loop`, no `throw`, no _closure_, no _annotations_, and no macros.
-Just _types_, _filters_, and _functions_.
+No `builtin` types, no `if` statement, no `loop`, no `throw`, no
+_closure_, no _annotations_, and no macros. Just _types_, _filters_,
+and _functions_.
 
-### Normalization
+### Normalization and Content-Addressing
 
-All type names are replaced by the hash of their canonical representation.
+All type names are replaced by the canonical hash (`@hash`) of their
+minimal deterministic tree automaton representation.
 
-We replace the function name by the hash of the normalized definition of the function as provided by the program defining the function.
-Obviously, a function may have many different definitions so the hash-based names are not canonical. By using hash-based function names we avoid the problem of name clashes when bringing together many `kernel-core` programs.
+Every relation is identified canonically by the content-addressed hash
+of its normalized definition AST.
 
-This approach solves the problem of modules, imports, etc., and opens the door to a universal
-registry of types (schemas) and functions.
+There is no global symbol namespace. Human-readable names (such as
+`plus`, `int`, or `5`) are local aliases recorded in object metadata
+(`relAlias`, `typeAliases`). When libraries are loaded with `--lib`,
+definitions are identified by their canonical `@hash`, but local aliases
+are not implicitly exported into the importing scope unless explicitly
+mapped via `--export`.
+
+This approach completely solves the problem of modules, name collisions,
+and unversioned imports, opening the door to a universal registry of
+types (schemas) and relations.
 
 ## Typing and Polymorphism (codes and patterns)
 
-The abstract syntax tree of a program consists of a dictionary of _type definitions_,
-a dictionary of _function definitions_, and the final _main expression_.
+The abstract syntax tree of a program consists of a dictionary of
+_type definitions_, a dictionary of _function definitions_, and the
+final _main expression_.
 
-In principle, the initial form of the Abstract Syntax Tree, _raw_ AST, is enough to evaluate
-the program on an input.
+In principle, the initial form of the Abstract Syntax Tree, _raw_ AST,
+is enough to evaluate the program on an input.
 
-Our objective, however, is to compute canonical representations for types and normalized
-representations for functions so they can be reused.
+Our objective, however, is to compute canonical representations for
+types and normalized representations for functions so they can be
+reused and compiled ahead-of-time.
 
 The normalization steps are:
 
-1. Build the _type graph_ from all types used by the program, i.e., used in _type definitions_
-   as well as type expressions used in _function definitions_ and the _main expression_.
-2. Annotate the expressions with filters; every node of the AST is annotated with a pair of filters.
-3. Turn the _singleton_ filters into types and add them to the _type graph_. Go to (2) unless no change.
+1. Build the _type graph_ from all types used by the program, i.e.,
+   used in _type definitions_ as well as type expressions used in
+   _function definitions_ and the _main expression_.
+2. Annotate the expressions with filters; every node of the AST is
+   annotated with a pair of filters.
+3. Turn the _singleton_ filters into types and add them to the _type
+   graph_. Go to (2) unless no change.
 
 ### Examples
 
@@ -213,21 +253,25 @@ inc3
 In the above program, we define type `bit` as union of two unit types,
 and type `byte` as product, for simplicity, of four `bit`s.
 
-Functions `bit0`, `bit1`, and `zero` are "constant polymorphic functions", meaning:
+Functions `bit0`, `bit1`, and `zero` are "constant polymorphic
+functions", meaning:
 
 - constant: if defined, they always return exactly the same value;
-- polymorphic: they are defined for more than one pair of input and output types: functions
-  `bit0` and `bit1` are of type `?X -> $bit`, and `zero` is of type `?X -> $byte`, where `?X`
-  denotes an unconstrained type pattern, also denoted as `?(...)`.
+- polymorphic: they are defined for more than one pair of input and
+  output types: functions `bit0` and `bit1` are of type `?X -> $bit`,
+  and `zero` is of type `?X -> $byte`, where `?X` denotes an
+  unconstrained type pattern, also denoted as `?(...)`.
 
 Functions `inc` and `inc3` are (non-polymorphic) functions of types
 `$byte -> ${ bit overflown, byte byte }` and
-`${ bit overflown, byte byte } -> ${ bit overflown, byte byte }`, respectively.
+`${ bit overflown, byte byte } -> ${ bit overflown, byte byte }`,
+respectively.
 
-Function `inc_o` is a polymorphic function of type patterns: `?X -> ${ byte byte, bit overflown }`
-with the following constraints:
+Function `inc_o` is a polymorphic function of type patterns:
+`?X -> ${ byte byte, bit overflown }` with the following constraints:
 
-- `?X` is a product type with at least two fields: `byte` and `overflown`, denoted by:
+- `?X` is a product type with at least two fields: `byte` and
+  `overflown`, denoted by:
    > `?{ $byte byte, Z overflown, ...}`;
 - `?Z` is a union type with field `i` denoted by:
    > `?< V i, ...>`;
@@ -235,8 +279,9 @@ with the following constraints:
 
 We can write it as: `?{$byte byte, <(...) i, ...> overflown, ...}`.
 
-The target type `${ byte byte, bit overflown }` corresponds to pattern `?{ $byte byte, $bit overflown }`.
-Such a pattern is called _singleton pattern_, as only one type fits the pattern.
+The target type `${ byte byte, bit overflown }` corresponds to pattern
+`?{ $byte byte, $bit overflown }`. Such a pattern is called _singleton
+pattern_, as only one type fits the pattern.
 
 #### Example 2
 
@@ -261,43 +306,61 @@ car = list? /cons .car;  -- @...
 
 ## Universal Schema Registry
 
-Since the normalization process for types is fast and deterministic, we can build a universal
-schema registry that will store all invented types.
-The registry will be a key-value store, where the key is the hash of the normalized type,
-and the value is the normalized type itself.
+Since the normalization process for types is fast and deterministic,
+we can build a universal schema registry that will store all invented
+types. The registry will be a key-value store, where the key is the hash
+of the normalized type, and the value is the normalized type itself.
 
-All functions (non-polymorphic and polymorphic) will be renamed by the hash of their normalized
-definition and stored in a similar key-value store.
+All functions (non-polymorphic and polymorphic) are named by the hash
+of their normalized definition and stored in a similar key-value store.
 
-Non-polymorphic functions can be easily indexed by the hashes of their input and output types
-so that we can quickly find the function we need.
+Non-polymorphic functions can be easily indexed by the hashes of their
+input and output types so that we can quickly find the function we need.
 Indexing polymorphic functions seems complicated (TODO).
 
 ## Serialization and Compilation
 
-The language is designed to transform _codes_, i.e., serialized typed values, i.e., trees
-accepted by a tree automaton, into other _codes_.
-The transformation is done by partial functions as defined in the language.
-The non-recursive and tail-recursive (and even some non-tail-recursive) functions can
-be compiled into deterministic finite (pushdown) transducers.
+The language is designed to transform _codes_, i.e., serialized typed
+values, i.e., trees accepted by a tree automaton, into other _codes_.
+The transformation is done by partial functions as defined in the
+language. The non-recursive and tail-recursive (and even some
+non-tail-recursive) functions can be compiled into deterministic finite
+(pushdown) transducers.
+
+Ahead-Of-Time (AOT) compilation produces four primary file formats (see
+[`DOCS/FILE_FORMATS.md`](./FILE_FORMATS.md)):
+- `.k`: Plain source text.
+- `.ko`: AOT type-checked executable object container (`KOBJ\n` + JSON).
+- `.klib`: AOT type-checked library container (JSON).
+- `.kvm`: Polymorphic register-IR template (`layer: "KVM-P"`).
+
+Downstream backends compile or specialize these into WebAssembly
+(`.wasm`) and native binaries.
 
 ### AND-OR graphs for encoding and decoding codes
 
-This could be a generic and compact way to encode and decode typed values.
-The idea is to use _prefix codes_ (a class of languages such that no word is a prefix of
-another word) to encode the values.
+This could be a generic and compact way to encode and decode typed
+values. The idea is to use _prefix codes_ (a class of languages such
+that no word is a prefix of another word) to encode the values.
 
 ### Rust data structures
 
-A type can be translated into a Rust data structure, and a function can be translated into a Rust function.
+A type can be translated into a Rust data structure, and a function
+can be translated into a Rust function.
 
 ## Linking with Other Languages
 
 ## Streams and Arrays (Map-Reduce)
 
-## Bits
+## Bits and Absence of Built-in Numbers
+
+In `k`, there are no primitive numbers or built-in integer literals.
+Numerical identifiers such as `5` or `10` are nullary relations
+(constants) mapping the unit `${}` to bit trees of type `${} -> $bits`.
+Arithmetic relations operate on signed integers (`$int`), requiring
+conversion via `5 int`.
 
-Extending `@string` into `@bits`. 
+Bit trees are defined algebraically:
 
 ```text
 $ bits = < {} _, bits 0, bits 1 >;
@@ -318,17 +381,28 @@ $ bits = < {} _, bits 0, bits 1 >;
 
 #### Eat `/`
 
-- `$@bits / @bits` --- division, e.g., `0b1011 / 0b10` = `0b11`, `"abc" / "a"` = `"bc"`
+- `$@bits / @bits` --- division, e.g., `0b1011 / 0b10` = `0b11`,
+  `"abc" / "a"` = `"bc"`
 
 #### Prepend `\`
 
-- `$@bits \ @bits` --- multiplication, e.g., `0b11 \ 0b10` = `0b1011`, `"bc" \ "a"` = `"abc"`
+- `$@bits \ @bits` --- multiplication, e.g., `0b11 \ 0b10` = `0b1011`,
+  `"bc" \ "a"` = `"abc"`
 
-Empty `bits` can be checked by
+Empty `bits` can be checked by:
 
 ```k
-
 kind = $@bits 
   < {/ 0b0 \ 0b0 starts_with_0}, {/ 0b1 \ 0b1 starts_with_1}, {() empty} > 
   $< @bits starts_with_0, @bits starts_with_1, @bits empty >;
 ```
+
+## Further Reading
+
+- [DOCS/FILE_FORMATS.md](./FILE_FORMATS.md) — guide to
+  .k, .ko, .klib, and .kvm formats
+- [DOCS/DICTIONARY.md](./DICTIONARY.md) — concept names and terminology
+- [DOCS/TYPE_DERIVATION.md](./TYPE_DERIVATION.md) — type derivation
+  algorithm
+- [DOCS/OBJECT_FILE_AND_PATTERN.md](./OBJECT_FILE_AND_PATTERN.md) —
+  object format and pattern encoding