@fraczak/k 6.0.1 → 6.0.2

package/DOCS/book/14-canonical-serialization.md

@@ -0,0 +1,176 @@
+# Chapter 14 — Canonical Serialization
+
+## **14.1  Purpose**
+
+Serialization is the process of converting a value in memory into a compact sequence of bits or bytes that can be stored or transmitted.
+For the k language, serialization serves an additional goal: it defines a **canonical representation** of each value that is unique and independent of the machine on which it was produced.
+
+A canonical encoding allows values to be compared, hashed, or stored in a registry in a reproducible way.
+Two values that are structurally identical will always produce the same bit sequence.
+
+---
+
+## **14.2  Canonical type information**
+
+Serialization always depends on structural information. In the monomorphic case
+this is the **canonical form of the type**. In the polymorphic codec this is a
+pattern graph carried with the value.
+A type in canonical form is a finite tree automaton (Chapter 3) that describes the structure of all its values.
+Because this form is unique, every possible value of the type can be encoded deterministically.
+
+Before any value is serialized, the compiler or runtime must know:
+
+* the list of states (C0, C1, …),
+* for each state, whether it is a product or a union,
+* the number of possible transitions, and
+* for unions, the order of their variants.
+
+This information acts as the grammar from which all bit sequences are derived.
+For a pattern-framed value, the same role is played by the decoded leading
+pattern; singleton patterns are exactly the monomorphic type case.
+
+---
+
+## **14.3  Encoding principle**
+
+Each node in a value tree corresponds to one of the type’s states.
+
+* **Product node** – emits no bits; the encoder simply serializes each child in canonical field order.
+* **Union node** – emits a small binary code that identifies which variant is used, then serializes its single child.
+* **Unit node** (empty product) – emits nothing; it has no children.
+* **Empty union** – cannot be encoded because it has no possible values.
+
+Thus, the sequence of bits records exactly which union branches were taken while descending the tree.
+
+---
+
+## **14.4  Fixed-length codes per state**
+
+For simplicity, each union state uses fixed-length binary codes of equal size.
+
+If a union has *m* variants, then
+
+```
+k = ceil(log2(m))
+```
+
+bits are needed.
+Each variant is numbered in canonical order from 0 to m − 1 and represented by the binary form of its index.
+
+Products and units use k = 0 and emit no bits.
+
+This rule allows the decoder to know, from the type alone, how many bits to read at each step.
+
+---
+
+## **14.5  Example: natural numbers**
+
+For the type
+
+```
+$bnat = < bnat 0, bnat 1, {} _ >;
+```
+
+the canonical form has three transitions from state C0.
+Therefore, `k = ceil(log2(3)) = 2` bits are required per union node.
+
+Assigning codes in order:
+
+| Rule           | Code |
+| -------------- | ---- |
+| C0 → {C0 "0"}  | `00` |
+| C0 → {C0 "1"}  | `01` |
+| C0 → {C1 "_" } | `10` |
+
+The value `{ {{ {} _ } 1 } 0 }` is encoded by concatenating the codes of the transitions chosen during a leftmost derivation:
+
+```
+00 → first C0
+01 → next C0
+10 → final C0
+```
+
+Resulting bit sequence: `000110`.
+
+---
+
+## **14.6  Decoding**
+
+Decoding is deterministic and mirrors the encoding process:
+
+1. Start at the root state (C0).
+2. If the current state is a union, read k bits to choose a rule.
+3. Create the corresponding node, then recursively decode its child or children.
+4. For products, decode all children in order; for units, stop.
+
+The decoder stops when the entire tree has been reconstructed and all bits have been consumed.
+
+Because every union code is of fixed length, the decoding process requires no backtracking and can be implemented as a simple loop.
+
+---
+
+## **14.7  Folding repeated subtrees**
+
+Many values contain repeated substructures.
+To avoid serializing the same tree several times, identical subtrees can be *folded* into a **directed acyclic graph (DAG)** before encoding.
+
+Each unique node is assigned an identifier.
+When the encoder encounters a repeated node, it emits a reference to its identifier instead of encoding it again.
+
+During decoding, the identifier is resolved to the corresponding subtree, reconstructing the shared structure.
+
+Folding is optional; it saves space without changing the logical value.
+
+---
+
+## **14.8  Stream format**
+
+A minimal canonical bitstream consists of:
+
+1. **Header** – type hash (e.g., 8 bytes) identifying the canonical type.
+2. **Node count** – integer *N* if DAG encoding is used.
+3. **Encoded data** – concatenation of all fixed-length union codes and, if folded, node references.
+
+If two systems share the same canonical type table, the bitstream alone is enough to reconstruct the original value.
+
+---
+
+## **14.9  Implementation sketch**
+
+A compact encoder can be written as follows:
+
+```
+procedure encode(node):
+    if node.arity == 0:
+        return
+    if node.arity == 1:
+        write_bits(code_for(node.state, node.tag))
+        encode(node.child[0])
+    else:
+        for each child in node.children:
+            encode(child)
+```
+
+The corresponding decoder reverses the process.
+Both procedures require only the canonical type tables and simple bit operations.
+
+---
+
+## **14.10  Determinism and equality**
+
+Because encoding is fully determined by the canonical type and field order, two equal values always produce identical bit sequences.
+Conversely, decoding the same bit sequence always yields the same tree.
+
+This property allows equality comparison by direct byte comparison of serialized forms, without traversing the trees in memory.
+
+---
+
+## **14.11  Summary**
+
+* Every type’s canonical automaton defines a unique bit-level grammar for its values.
+* Products emit no bits; unions emit fixed-length variant codes.
+* The number of bits required for each state depends only on the number of variants in that state.
+* Optional DAG folding eliminates repeated subtrees.
+* The resulting bitstream is compact, deterministic, and machine-independent.
+
+---

package/DOCS/book/appendix-b-incorporating-external-predefined-types-and-functions.md

@@ -0,0 +1,155 @@
+# Appendix B — Incorporating External Predefined Types and Functions
+## **B.1  Purpose**
+
+The k language is intentionally minimal.
+Nevertheless, real systems often need to interact with data types and operations defined outside of k—for example, numeric values, text, or platform-specific constants.
+This appendix explains how external predefined types and functions can be introduced into a k compiler without changing the language itself.
+
+---
+
+## **B.2  External types**
+
+An **external type** is a canonical type that is *not* expressed through k's own product-and-union syntax but is known to the compiler through registration.
+
+Each external type has:
+
+| Field                 | Description                                                                  |
+| --------------------- | ---------------------------------------------------------------------------- |
+| **Name**              | symbolic name (e.g., `$int`, `$string`)                                      |
+| **Hash**              | unique stable identifier, just like canonical types                          |
+| **Runtime kind**      | how the value is stored in memory (pointer, immediate integer, etc.)         |
+| **Adapter functions** | conversion between the external representation and the standard `KNode` tree |
+
+Example registry entry:
+
+```json
+{
+  "hash": "E001",
+  "kind": "type",
+  "external": true,
+  "name": "$int",
+  "runtime_kind": "primitive_i64"
+}
+```
+
+The compiler treats such a type as an atomic node, represented internally as a single `KNode` with `arity = 0` but tagged as *external*.
+
+---
+
+## **B.3  External constructors**
+
+External types may provide special constructors or constants.
+For example, `$int` might include predefined constant functions:
+
+```
+zero  = {} $int;
+one   = {} $int;
+```
+
+At compile time these appear as constant functions returning fixed external nodes provided by the runtime.
+
+---
+
+## **B.4  External functions**
+
+External functions are partial functions implemented outside of k but declared inside a k program.
+They have known input and output types, possibly external.
+
+Declaration syntax:
+
+```
+extern add : $int $int → $int;
+extern less : $int $int → $bool;
+```
+
+At compile time the compiler records the signature and associates it with a runtime symbol name such as `k_ext_add`.
+
+When generating code, each external call becomes a standard function call following the ABI:
+
+```llvm
+declare %KOpt @k_ext_add(%KVal %a, %KVal %b)
+```
+
+The runtime library provides these implementations in a native language (C, C++, Rust, etc.).
+
+---
+
+## **B.5  Integration with canonical types**
+
+External types are integrated into the canonical system as *leaf states*.
+They can appear as components in products or unions just like built-in ones.
+
+Example:
+
+```
+$point = { $int x, $int y };
+```
+
+Canonical form:
+
+```
+$C0 = { C1 "x", C1 "y" };
+$C1 = external $int;
+```
+
+This allows normal serialization and type comparison; the external leaf is treated as opaque but identified by its hash.
+
+---
+
+## **B.6  Serialization and external values**
+
+During serialization, an external node is encoded by:
+
+1. Its type hash (to indicate which external kind it represents).
+2. A binary blob produced by a runtime-supplied encoder for that type.
+
+Deserialization reverses the process using the corresponding decoder.
+All external codecs must be deterministic and version-stable to preserve canonical equality.
+
+---
+
+## **B.7  Example**
+
+Suppose we introduce `$int` and external function `add`:
+
+```
+$pair_int = { $int a, $int b };
+sum = { .a x, .b y } (add x y);
+```
+
+The compiler generates code that:
+
+* projects fields `.a` and `.b`,
+* passes them as `KVal` pointers to `@k_ext_add`,
+* and returns the result as a new `$int` node.
+
+All logic outside arithmetic remains standard k code.
+
+---
+
+## **B.8  Practical implementation**
+
+In a Python prototype:
+
+```python
+EXTERNAL_TYPES = {
+    "$int": {"hash": "E001", "kind": "external", "runtime_kind": "primitive_i64"}
+}
+
+EXTERNAL_FUNCS = {
+    "add":  {"inputs": ["$int", "$int"], "output": "$int", "symbol": "k_ext_add"},
+    "less": {"inputs": ["$int", "$int"], "output": "$bool", "symbol": "k_ext_less"}
+}
+```
+
+During code generation, when a call refers to an external function, the compiler emits a call to the symbol given in the table.
+
+---
+
+## **B.9  Summary**
+
+* External predefined types extend k without altering its core semantics.
+* They are registered with hashes and treated as atomic nodes.
+* External functions follow the same calling convention as normal ones.
+* Serialization of external values uses type-specific codecs.
+* This mechanism allows integration of native computations (numbers, strings, system data) while preserving the purity and determinism of the k language.

package/DOCS/book/overview.txt

@@ -0,0 +1,68 @@
+Numbers are an extremely smart encoding.
+Kernel/core can describe that encoding — and many others.
+
+Most programming languages take numbers as a starting point. Integers, booleans, characters, and strings appear as given, primitive entities, and computation is then defined in terms of control flow operating over those primitives.
+
+Kernel/core deliberately inverts this perspective.
+
+Kernel/core starts from the observation that numbers are not fundamental objects of computation. They are encodings: highly successful, extremely smart encodings, but encodings nonetheless. What is truly fundamental is the ability to describe structured values and to define relations between them.
+
+Kernel/core is a language built around this idea.
+
+Kernel/core is based on exactly two concepts: encoding and relation.
+
+Encoding describes how values are represented. Relation describes how values are transformed.
+
+Everything else found in conventional programming languages—control flow, loops, exceptions, built-in types, higher-order functions—is intentionally absent. Not because kernel/core is low-level, but because those constructs are derivable, incidental, or encoding-specific.
+
+Kernel/core is a high-level language precisely because it is minimal.
+
+An encoding describes the shape of values. Encodings are algebraic data types built from two constructors: products and unions.
+
+Encodings are explicit, first-class, and user-defined. Kernel/core has no built-in integers, booleans, strings, or characters. If such values are needed, they are described as encodings within the language itself.
+
+This has several consequences. Every value has a precise, inspectable structure. Serialization and deserialization are intrinsic, not an afterthought. Different encodings of the same concept, such as unary, binary, or decimal numbers, coexist naturally.
+
+Numbers, in particular, appear not as primitives but as one possible encoding among many.
+
+A kernel/core program defines a relation: a partial, first-order function from encoded values to encoded values.
+
+Relations are partial: they may be undefined for some inputs. They are first-order: relations do not take other relations as values. They are pure: there is no implicit state and no hidden effects.
+
+Kernel/core deliberately avoids if, while, throw, and similar constructs. Instead, relations are combined using three forms of composition.
+
+Sequential composition chains transformations. Parallel composition executes multiple relations independently on the same input and produces a product value. Concurrent composition defines ordered alternatives, where the first defined result is selected.
+
+Control flow emerges from partiality and composition, not from dedicated syntax.
+
+Kernel/core bootstraps computation without primitives by relying on projections induced by encodings.
+
+Every product or union encoding introduces canonical projection relations. These projections are the only predefined relations in the language. All other behavior is built by composition.
+
+There are no built-in constants in the traditional sense. Even constants are relations: partial functions that ignore their input and return a fixed encoded value.
+
+This makes the language uniform: everything is a relation.
+
+Kernel/core supports polymorphic relations through patterns.
+
+A pattern is a relaxed encoding specification: a filter that constrains only part of a value’s structure. Patterns allow relations to be defined over classes of encodings, not just single concrete ones.
+
+For example, a relation may require any product type with at least a given field of a given encoding, without fixing the rest of the structure.
+
+This form of polymorphism is structural, explicit, and decidable. It does not rely on subtyping hierarchies, inheritance, or ad-hoc overloading.
+
+Kernel/core is not a scripting language, not an object-oriented language, not a numeric language, and not a control-flow language.
+
+Kernel/core is a language for describing encodings, a language for defining relations over structured data, a foundation for serialization, transformation, and compilation, and a minimal core on top of which richer systems can be built.
+
+Kernel/core does not claim that numbers are wrong.
+
+On the contrary: numbers are one of the most successful encodings ever invented.
+
+Kernel/core simply makes that fact explicit.
+
+By separating what computation is from how values are encoded, kernel/core offers a different foundation for programming—one where arithmetic becomes a special case, not the starting point.
+
+Numbers are an extremely smart encoding.
+
+Kernel/core is the language that lets you describe that encoding — and any other one you can imagine.

package/README.md

@@ -28,12 +28,12 @@ k-repl               # start the interactive REPL  (or: node repl2.mjs)
 | Binary | Source | Purpose |
 |--------|--------|---------|
 | `k` | `k.mjs` | Execute a `.k` script; reads binary pattern+value stream from stdin |
-| `k-repl` / `k-repl2` | `repl2.mjs` | Interactive REPL |
-| `k-parse` | `codecs/k-parse.mjs` | Encode a JSON value to the binary pattern+value wire format |
-| `k-print` | `codecs/k-print.mjs` | Decode binary pattern+value stream back to JSON |
-| `k-pattern` | `codecs/k-pattern.mjs` | Print the pattern of a binary-encoded value |
-| `k-compile-object` | `objects/compile.mjs` | Compile a `.k` source to a `.kobj` object file |
-| `k-decompile-object` | `objects/decompile.mjs` | Decompile a `.kobj` file back to k source |
+| `k-repl` / `k-repl2` | `repl2.mjs` | Interactive interpreter |
+| `k-parse` | `codecs/k-parse.mjs` | Encode textual k values to the binary pattern+value wire format |
+| `k-print` | `codecs/k-print.mjs` | Decode binary pattern+value stream to JSON-like textual k values |
+| `k-pattern` | `patterns/from-k.mjs` | Extract the canonical root pattern from a k script |
+| `k-compile-object` | `objects/compile.mjs` | Compile a `.k` source to an executable `.ko` object |
+| `k-decompile-object` | `objects/decompile.mjs` | Decompile a `.ko` or `.klib` file back to k source |
 
 ---
 
@@ -48,6 +48,9 @@ Codes are algebraic data types built from two constructors:
 
 Native k-like notation is canonical; JSON-like is supported as syntactic sugar.
 
+For the JSON-like textual notation used by `k-parse` and `k-print`, see
+[DOCS/TEXTUAL_VALUES.md](DOCS/TEXTUAL_VALUES.md).
+
 ### Defining codes (`$`)
 
 ```k
@@ -135,38 +138,60 @@ A partial function may be undefined on some inputs. Core primitives:
 
 ---
 
-## REPL
+## Interpreter
 
 Start with `k-repl` (or `node repl2.mjs`). The prompt is `> `.
 
-### Defining names and codes
+`repl2.mjs` keeps a live `.klib`-style state in memory. You can:
 
-```
-> $ nat = < {} zero, nat succ >;
-> succ = |succ;
-> zero = {} |zero $nat;
-```
+- define types and relations incrementally
+- run expressions against the current value
+- load `.k` or `.klib` files into the session
+- export the session as `.klib`
+- export an executable `.ko` from an expression in the current context
 
-### REPL commands
+### Main commands
 
 | Command | Effect |
 |---------|--------|
-| `--C name` | Show canonical (hash-addressed) form of a code |
-| `--R name` | Show the type derivation (pattern graph) for a function |
-| `--help` | List available commands |
+| `:type name = <...>` | Define a type alias |
+| `:rel name = expr` | Define a relation alias |
+| `:run expr` | Evaluate an expression on the current value |
+| `:t name` | Show relation input/output filters |
+| `:d name` | Show relation definition |
+| `:C name` | Show canonical code definition |
+| `:codes` / `:rels` | List type or relation aliases |
+| `:load file` | Load `.k` or `.klib` |
+| `:klib file` | Export current state as `.klib` |
+| `:ko file expr` | Export executable `.ko` |
+| `:val` | Print current value |
+| `:reset` | Clear state |
+| `:help` | Show command summary |
+
+### Raw snippets
+
+Raw k input is also accepted. It compiles on top of the current interpreter
+state.
 
-**Example session:**
-
-```
-> $ nat = < {} zero, nat succ >;
-> $ pair = { nat x, nat y };
-> --C pair
- $ @wq9VvJgYt8sASQfevD23... = { @w8iSHe... x, @w8iSHe... y };
-> toto = .toto;
-> --R toto
-  toto : ?{ X0 toto, ... }  -->  ?X0
+```text
+> $ bool = <{} true, {} false>;
+> not = $bool </true | false, {} | true>;
+> {} |true not
+{false: {}} ?<{} true, {} false>
 ```
 
+Definitions-only snippets extend the session silently. Snippets with a terminal
+expression are evaluated immediately.
+
+Tab completion covers:
+
+- command names
+- file paths for `:load`, `:klib`, and `:ko`
+- aliases
+- canonical names beginning with `@`
+
+See [DOCS/REPL.md](DOCS/REPL.md) for the exact buffering and evaluation rules.
+
 ---
 
 ## Code Derivation and Patterns
@@ -269,6 +294,25 @@ echo '{"cons":{"car":{"1":{}},"cdr":{"nil":{}}}}' \
 
 See [`codecs/README.md`](codecs/README.md) for codec internals and format details.
 
+For the textual boundary notation used by `k-parse` and `k-print`, see
+[DOCS/TEXTUAL_VALUES.md](DOCS/TEXTUAL_VALUES.md). For canonical exported
+patterns and `k-pattern`, see [DOCS/PATTERNS.md](DOCS/PATTERNS.md).
+
+## Object and Library Files
+
+The object toolchain is documented in [objects/README.md](objects/README.md).
+
+- `.klib` stores compiled library state: codes, relations, aliases, metadata
+- `.ko` stores an executable object with a main expression
+
+Typical CLI usage:
+
+```bash
+./objects/compile-lib.mjs Examples/ieee.k ieee.klib
+./objects/compile.mjs --lib ieee.klib Examples/nat.k nat.ko
+./objects/decompile.mjs nat.ko
+```
+
 ---
 
 ## Examples

package/codecs/README.md

@@ -70,6 +70,11 @@ The runtime boundary is therefore:
 binary pattern+value stream <-> Value(pattern, tree)
 ```
 
+At the command-line boundary, `k-parse` and `k-print` use a JSON-like textual
+notation for k values. That notation is documented in
+[`../DOCS/TEXTUAL_VALUES.md`](../DOCS/TEXTUAL_VALUES.md). It is a readable
+notation for k values, not a separate JSON container format.
+
 `k.mjs` is only the command-line adapter for that boundary. The operational
 runtime sees and preserves the pattern on the `Value` itself.
 
@@ -115,6 +120,9 @@ Canonical rules:
 This JSON-like graph is only a readable notation. The wire representation is
 the ordinary k `$pattern` value from `core.k`.
 
+For user-facing pattern export, see [`../DOCS/PATTERNS.md`](../DOCS/PATTERNS.md)
+and the `k-pattern` CLI.
+
 ## Witness-Derived Patterns
 
 When `k-parse` is used without an explicit input pattern or type, the wire

package/codecs/pandoc.css

@@ -0,0 +1,71 @@
+html {
+  line-height: 1.5;
+  font-family: Georgia, "Times New Roman", serif;
+  color: #1f2328;
+  background: #ffffff;
+}
+
+body {
+  max-width: 980px;
+  margin: 2rem auto 4rem;
+  padding: 0 1.25rem;
+}
+
+h1, h2, h3, h4 {
+  line-height: 1.2;
+  margin-top: 1.8rem;
+}
+
+h1 {
+  border-bottom: 1px solid #d0d7de;
+  padding-bottom: 0.3rem;
+}
+
+p, li {
+  max-width: 78ch;
+}
+
+pre, code {
+  font-family: "Iosevka Term", "SFMono-Regular", Consolas, monospace;
+}
+
+code {
+  background: #f6f8fa;
+  padding: 0.1rem 0.25rem;
+  border-radius: 4px;
+}
+
+pre {
+  background: #f6f8fa;
+  border: 1px solid #d0d7de;
+  border-radius: 8px;
+  padding: 0.9rem 1rem;
+  overflow-x: auto;
+}
+
+pre code {
+  background: transparent;
+  padding: 0;
+}
+
+blockquote {
+  margin-left: 0;
+  padding-left: 1rem;
+  border-left: 4px solid #d0d7de;
+  color: #57606a;
+}
+
+table {
+  border-collapse: collapse;
+  margin: 1rem 0;
+}
+
+th, td {
+  border: 1px solid #d0d7de;
+  padding: 0.4rem 0.7rem;
+  vertical-align: top;
+}
+
+th {
+  background: #f6f8fa;
+}