@fraczak/k 5.1.2 → 6.0.0

package/codecs/BINARY_FORMAT.md

@@ -0,0 +1,53 @@
+# Binary Packaging Notes
+
+This document no longer defines the active codec semantics.
+
+The active format is the self-hosted pattern-plus-prefix-tree stream described
+in [`POLYMORPHIC_BINARY_FORMAT.md`](./POLYMORPHIC_BINARY_FORMAT.md):
+
+```text
+encode($pattern_value : $pattern) encode(value : decoded_pattern)
+```
+
+## Purpose Of This File
+
+This file records constraints for any future optimized container layered above
+the canonical stream.
+
+The design sequence is:
+
+1. define the abstract pattern graph,
+2. define prefix-free value-tree encoding relative to that graph,
+3. encode the pattern itself as a k `$pattern` value,
+4. optionally add higher-level indexes or compression as separate layers.
+
+## Expected Container Shape
+
+The canonical stream contains:
+
+```text
+encoded-$pattern + value-bitstream
+```
+
+where:
+
+- `encoded-$pattern` is the k value representation defined in `core.k`,
+- the value bitstream is exactly the same prefix-free payload described by the
+  semantic codec.
+
+## Non-Goals
+
+This future binary package should not redefine the semantic codec.
+
+In particular, it should not introduce as part of the base semantics:
+
+- byte-oriented node tables,
+- mandatory DAG encoding,
+- transport-driven structural compromises.
+
+Those are explicitly outside the target architecture of the rewrite.
+
+## Current Format
+
+The working concrete format is the canonical stream above. JSON property-list
+patterns are debug notation only, not a transport container.

package/codecs/POLYMORPHIC_BINARY_FORMAT.md

@@ -0,0 +1,303 @@
+# K Prefix-Free Pattern Codec
+
+## Purpose
+
+This document defines the new target semantics for binary codecs in `k`.
+
+The object to be serialized is:
+
+```text
+(P, v)
+```
+
+where:
+
+- `P` is a rooted pattern graph,
+- `v` is a value tree compatible with `P`.
+
+This is both the transport semantics and the runtime value semantics. In the
+current JavaScript runtime, the materialized `Value` stores the pattern as
+`value.pattern`; the tree is still represented by `Product` and `Variant`
+nodes.
+
+The codec is designed around:
+
+1. compactness,
+2. efficient structural navigation,
+3. a clean separation between semantic encoding and optional acceleration
+   structures.
+
+This document defines the semantic codec and the canonical wire stream.
+
+## Design Principles
+
+### 1. Pattern is the structural authority
+
+The pattern determines:
+
+- whether a node is a product or union,
+- whether it is open or closed,
+- what field names or tag names exist,
+- how recursive positions are connected.
+
+The value payload is interpreted relative to the pattern.
+During execution, the same pattern context is propagated with materialized
+values, so structural operations do not have to reconstruct framing information
+after the fact.
+
+### 2. Values are trees in the base codec
+
+The base codec serializes value trees, not value DAGs.
+
+DAG compression is not part of the primitive codec. If wanted, it should happen
+as a separate transform above the base value encoding.
+
+### 3. Prefix-free value encoding
+
+The value payload is a prefix-free bitstream.
+
+Products rely on structural concatenation.
+Unions emit a choice code, then the payload of the selected branch.
+
+### 4. One canonical stream
+
+The pattern graph is embedded as an ordinary k `$pattern` value. The canonical
+stream is that encoded pattern followed by the value payload interpreted under
+the decoded pattern.
+
+## Pattern Graph
+
+### Semantic node kinds
+
+There are exactly five node kinds:
+
+- `any`
+- `open-product`
+- `open-union`
+- `closed-product`
+- `closed-union`
+
+Their informal meanings are:
+
+- `any`
+  unconstrained position,
+- `open-product`
+  explicit fields plus possibly more fields in the matched value,
+- `open-union`
+  explicit tags plus possibly more tags in the matched value,
+- `closed-product`
+  exactly the listed fields,
+- `closed-union`
+  exactly the listed tags.
+
+In the base codec, `any` is not directly value-carrying.
+
+### Graph structure
+
+A pattern graph node is:
+
+```text
+[kind, edges]
+```
+
+where `edges` is a list of:
+
+```text
+[label, target_node_id]
+```
+
+The root node is node `0`.
+
+### Canonical constraints
+
+- node `0` is the root,
+- node order is canonical graph-discovery order,
+- edges are sorted by label,
+- no duplicate labels occur within one node,
+- `any` has no outgoing edges.
+
+### Readable property-list notation
+
+For documentation and tests, the abstract graph may be rendered as:
+
+```json
+{
+  "pattern": [
+    ["closed-union", [["nil", 1], ["cons", 2]]],
+    ["closed-product", []],
+    ["closed-product", [["car", 3], ["cdr", 0]]],
+    ["closed-union", [["_", 1], ["0", 3], ["1", 3]]]
+  ]
+}
+```
+
+This syntax is only a readable notation for the abstract graph.
+
+## Pattern Construction From A Witness Tree
+
+When no external pattern or type is supplied, the codec tools derive a
+pattern from the value tree itself.
+
+- `{}` derives the closed product leaf `["closed-product", []]`.
+- A textual tree node with multiple children derives a closed product.
+- A textual tree node with exactly one child derives an open union.
+- If an explicit product pattern is supplied, that pattern can disambiguate the
+  same one-child textual node as a singleton product.
+
+Example:
+
+```text
+{a:{b:x,c:{}}}
+```
+
+derives:
+
+```json
+[
+  ["open-union", [["a", 1]]],
+  ["closed-product", [["b", 2], ["c", 3]]],
+  ["open-union", [["x", 3]]],
+  ["closed-product", []]
+]
+```
+
+The derived graph is canonicalized by a bottom-up closed-node collapse. Starting
+from the closed empty product, two closed nodes are collapsible only when their
+kind, labels, and already-collapsed children are identical. Open nodes are not
+collapsed, and this rule is not value-DAG compression.
+
+## Value Encoding
+
+### General shape
+
+The value encoding is defined recursively relative to the current pattern node.
+
+#### Product node
+
+For `open-product` or `closed-product`:
+
+```text
+encode(v_0) encode(v_1) ... encode(v_(n-1))
+```
+
+in canonical edge order.
+
+There are no separators between children. The pattern determines where each
+child begins and ends.
+
+#### Union node
+
+For `open-union` or `closed-union`:
+
+```text
+encode-choice(tag_ordinal, tag_count)
+encode(selected_payload)
+```
+
+#### Any node
+
+`any` is not directly value-carrying in the base format.
+
+If a value would need to descend into an `any` position, the pattern must first
+be refined.
+
+## Choice Encoding
+
+The initial concrete choice encoding is fixed-width by cardinality:
+
+```text
+width(cardinality) = ceil(log2(cardinality))
+```
+
+A choice with:
+
+- `cardinality = n`
+- `ordinal in [0, n)`
+
+is encoded as the binary representation of `ordinal` in `width(n)` bits.
+
+Examples:
+
+- `n = 1`: `0` bits,
+- `n = 2`: `1` bit,
+- `n = 3`: `2` bits,
+- `n = 4`: `2` bits,
+- `n = 5`: `3` bits.
+
+This is the initial canonical rule for the rewrite. More aggressive coding
+schemes may be considered later, but only after the abstract semantics are
+stable.
+
+## Operational Properties
+
+### `/tag`
+
+At a union node:
+
+1. read the choice code,
+2. compare it to the requested tag ordinal,
+3. if it matches, continue with the selected payload.
+
+This makes `/tag` naturally efficient.
+
+### `.field`
+
+At a product node, child values appear in canonical field order.
+
+So `.field` is resolved by:
+
+1. locating the field ordinal in the pattern,
+2. decoding or skipping earlier children,
+3. decoding the requested child subtree.
+
+The base codec does not include field-offset indexes. Such indexes may be added
+later as separate higher-level acceleration structures.
+
+In the materialized evaluator, `.field` applies the analogous pattern operation:
+the output `Value` carries the subpattern reached by the selected field.
+Likewise, `/tag` carries the subpattern reached by the selected variant tag.
+
+## Recursive Types
+
+Recursive pattern graphs are first-class.
+
+For example, a bit-list-like recursive type:
+
+```text
+["closed-union", [["_", 1], ["0", 3], ["1", 3]]]
+```
+
+is encoded as repeated union choices in a value tree, not as a byte-aligned node
+table and not as a built-in primitive integer format.
+
+This keeps the codec uniform: recursion is handled by the pattern graph itself.
+
+## Canonical Wire Format
+
+The transport format is:
+
+```text
+encode($pattern_value : $pattern) encode(value : decoded_pattern)
+```
+
+The first segment is a normal k value of type `$pattern`, encoded under the
+fixed singleton pattern for `$pattern` defined by `core.k`. The decoded pattern
+is then used as the structural authority for the second segment.
+
+Decoding the stream yields `Value(pattern, tree)`. Encoding a runtime `Value`
+uses its carried pattern by default, or an explicit caller-supplied pattern when
+one is provided.
+
+## Out Of Scope For The Base Codec
+
+The following are intentionally outside the primitive codec:
+
+- DAG compression,
+- subtree deduplication,
+- projection indexes,
+- alternate transport-specific framing,
+- alternate binary pattern serialization,
+- exact-type-only specializations.
+
+These may be added later, but they are not part of the base pattern-plus-tree
+prefix codec.

package/codecs/README.md

@@ -0,0 +1,215 @@
+# K Codecs
+
+This directory is being rewritten around a new codec design:
+
+```text
+abstract pattern graph + prefix-free tree encoding
+```
+
+The default concrete format is now the binary encoding of a `$pattern` value,
+using the `$pattern` type from [`../core.k`](../core.k), immediately followed by
+the binary encoding of the value under that decoded pattern.
+
+There is no separate JSON container format.
+
+## Objectives
+
+The rewrite is guided by two hard requirements:
+
+1. compactness close to the structural lower bound,
+2. efficient `k` projections, especially `.field` and `/tag`.
+
+The current target architecture is the pattern graph plus prefix-free value-tree
+stream described here and in
+[`POLYMORPHIC_BINARY_FORMAT.md`](./POLYMORPHIC_BINARY_FORMAT.md).
+
+## Core Idea
+
+The semantic object serialized by the new codec is:
+
+```text
+(P, v)
+```
+
+where:
+
+- `P` is an abstract rooted pattern graph,
+- `v` is a value tree compatible with `P`.
+
+The same pair is now the in-memory runtime model as well. Decoding a stream
+attaches `P` to the resulting `Value` as `value.pattern`; evaluation propagates
+that pattern through projections and constructors; encoding a `Value` uses its
+carried pattern unless an explicit pattern is supplied.
+
+The base value encoding is:
+
+- tree-based, not DAG-based,
+- prefix-free,
+- interpreted relative to the pattern graph,
+- free of repeated field names and tag names in the value payload.
+
+The base codec does **not** perform DAG compression. If sharing is desired, it
+should happen as a higher-level transform, not inside the primitive value
+format.
+
+## Wire Format
+
+The default transport format is:
+
+```text
+encode($pattern_value : $pattern) encode(value : decoded_pattern)
+```
+
+`$pattern_value` is a normal k value whose shape is defined in `core.k`. After
+that value is decoded, it becomes the pattern used to decode the remaining
+value payload.
+
+The runtime boundary is therefore:
+
+```text
+binary pattern+value stream <-> Value(pattern, tree)
+```
+
+`k.mjs` is only the command-line adapter for that boundary. The operational
+runtime sees and preserves the pattern on the `Value` itself.
+
+## Pattern Graph Representation
+
+For documentation and tests, a pattern graph may be shown as a property-list
+style vector of nodes:
+
+```text
+[kind, edges]
+```
+
+where:
+
+- `kind` is one of:
+  - `"any"`
+  - `"open-product"`
+  - `"open-union"`
+  - `"closed-product"`
+  - `"closed-union"`
+- `edges` is a list of:
+  - `[label, target_node_id]`
+
+Example:
+
+```json
+[
+  ["closed-union", [["nil", 1], ["cons", 2]]],
+  ["closed-product", []],
+  ["closed-product", [["car", 3], ["cdr", 0]]],
+  ["closed-union", [["_", 1], ["0", 3], ["1", 3]]]
+]
+```
+
+Canonical rules:
+
+- root node is node `0`,
+- node list order is canonical graph-discovery order,
+- edges are sorted by label,
+- edge labels are unique within a node,
+- `"any"` must have no outgoing edges.
+
+This JSON-like graph is only a readable notation. The wire representation is
+the ordinary k `$pattern` value from `core.k`.
+
+## Witness-Derived Patterns
+
+When `k-parse` is used without an explicit input pattern or type, the wire
+pattern is derived from the parsed value tree.
+
+- An empty textual node is a closed product: `{}`.
+- A textual node with multiple children is a closed product.
+- A textual node with one child is interpreted as an open union by default.
+- An explicit product input pattern may force such a node to be treated as a
+  singleton product instead.
+
+For example, parsing:
+
+```text
+{a:{b:x,c:{}}}
+```
+
+without an input pattern derives:
+
+```json
+[
+  ["open-union", [["a", 1]]],
+  ["closed-product", [["b", 2], ["c", 3]]],
+  ["open-union", [["x", 3]]],
+  ["closed-product", []]
+]
+```
+
+During this construction, finite closed pattern subtrees are hash-consed from
+the leaves upward. The starting closed leaf is `["closed-product", []]`. Two
+closed nodes collapse only when they have the same closed kind, the same labels,
+and the same already-collapsed child targets. Open nodes keep their identity,
+and recursive closed nodes are not collapsed by this witness-tree rule.
+
+This is canonicalization of the pattern graph carried in the wire stream. It is
+not DAG compression of the value payload.
+
+## Prefix-Free Value Encoding
+
+The value payload is a bitstream interpreted relative to the pattern graph.
+
+Base rules:
+
+- closed or open product:
+  - encode child values in canonical edge order,
+- closed or open union:
+  - encode the selected tag position,
+  - then encode the selected payload,
+- `any`:
+  - not directly value-carrying in the base format.
+
+The initial concrete union encoding rule is:
+
+```text
+width = ceil(log2(cardinality))
+emit tag ordinal in width bits
+```
+
+This is the simplest canonical prefix-free choice encoding. It is not the final
+word on compression, but it gives a regular and compact base encoding that can
+later be refined without changing the abstract pattern/value model.
+
+## Projections
+
+The base tree encoding is chosen so that:
+
+- `/tag` only needs to read the union choice at the current node,
+- `.field` follows canonical product order and may later be accelerated by
+  higher-level indexes if necessary.
+
+In the materialized runtime, projections also project the carried pattern:
+`.field` returns the subpattern at that product field, and `/tag` returns the
+subpattern at that union tag. If no carried subpattern is available, later
+encoding can still derive a witness pattern from the result tree.
+
+The primitive codec stays minimal. Projection indexes, framing, or sharing
+schemes are explicitly separate concerns.
+
+## Design Boundary
+
+The new codec work is split into layers:
+
+1. abstract pattern graph semantics,
+2. prefix-free value-tree semantics relative to that graph,
+3. self-hosted `k` representation of the pattern graph,
+4. optional higher-level optimizations.
+
+## Files
+
+- [`POLYMORPHIC_BINARY_FORMAT.md`](./POLYMORPHIC_BINARY_FORMAT.md):
+  main design document for the new pattern-plus-prefix-tree codec.
+- [`BINARY_FORMAT.md`](./BINARY_FORMAT.md):
+  notes on future binary packaging of the same abstract semantics.
+
+## Status
+
+The active command-line pipeline emits and consumes the self-hosted binary
+pattern+value stream.

package/codecs/example-pipeline.mjs

@@ -0,0 +1,149 @@
+#!/usr/bin/env node
+/**
+ * Example: Demonstrating the codec pipeline
+ * 
+ * Shows how to:
+ * 1. Define types with canonical names
+ * 2. Create values
+ * 3. Encode to binary
+ * 4. Pass through "compiled program" (identity)
+ * 5. Decode from binary
+ * 6. Print result
+ */
+
+import { deriveClosedPattern } from './runtime/codec.mjs';
+import { encodeToWire, decodeWire } from './runtime/prefix-codec.mjs';
+import { patternToPropertyList } from './runtime/pattern-json.mjs';
+import { Product, Variant } from '../Value.mjs';
+import hash from '../hash.mjs';
+
+console.log('=== K Codec Pipeline Example ===\n');
+
+// ============================================================================
+// Step 1: Define types (normally from compiler)
+// ============================================================================
+
+const types = {};
+
+// Unit: {}
+const unitDef = { code: 'product', product: {} };
+const unitName = hash('$C0={};');
+types[unitName] = unitDef;
+
+// Bool: <{} false, {} true>
+const boolDef = {
+  code: 'union',
+  union: { false: unitName, true: unitName }
+};
+const boolName = hash(`$C0=<${unitName}"false",${unitName}"true">;`);
+types[boolName] = boolDef;
+
+// Maybe: <{} nothing, Bool something>
+const maybeDef = {
+  code: 'union',
+  union: { nothing: unitName, something: boolName }
+};
+const maybeName = hash(
+  `$C0=<${unitName}"nothing",${boolName}"something">;`
+);
+types[maybeName] = maybeDef;
+
+console.log('Types defined:');
+console.log('  unit:', unitName);
+console.log('  bool:', boolName);
+console.log('  maybe:', maybeName);
+console.log();
+
+// ============================================================================
+// Step 2: Type resolver
+// ============================================================================
+
+function resolveType(typeName) {
+  const def = types[typeName];
+  if (!def) {
+    throw new Error(`Unknown type: ${typeName}`);
+  }
+  return def;
+}
+
+// ============================================================================
+// Step 3: Create input value: Maybe Bool = something(true)
+// ============================================================================
+
+const inputValue = new Variant(
+  'something',
+  new Variant('true', new Product({}))
+);
+
+console.log('Input value:', inputValue.toString());
+console.log();
+
+const maybePattern = patternToPropertyList(
+  deriveClosedPattern(maybeName, maybeDef, resolveType)
+);
+
+// ============================================================================
+// Step 4: Parse (TEXT → BINARY)
+// ============================================================================
+
+console.log('--- PARSE (text → binary) ---');
+const binaryInput = encodeToWire(inputValue, maybePattern);
+console.log('Binary size:', binaryInput.length, 'bytes');
+console.log('  Bytes:', binaryInput.toString('hex'));
+console.log();
+
+// ============================================================================
+// Step 5: Compiled k-program (BINARY → BINARY)
+// ============================================================================
+
+console.log('--- COMPILED PROGRAM (binary → binary) ---');
+
+// Simulate a compiled k-program: decode → transform → encode
+const { value: programInput } = decodeWire(binaryInput);
+
+console.log('Program sees value:', programInput.toString());
+
+// Example transformation: NOT operation (flip bool inside Maybe)
+let programOutput;
+if (programInput instanceof Variant && programInput.tag === 'something') {
+  const innerBool = programInput.value;
+  if (innerBool instanceof Variant) {
+    const flipped = new Variant(
+      innerBool.tag === 'true' ? 'false' : 'true',
+      innerBool.value
+    );
+    programOutput = new Variant('something', flipped);
+  }
+} else {
+  programOutput = programInput; // nothing → nothing
+}
+
+console.log('Program output value:', programOutput.toString());
+
+// Encode output
+const binaryOutput = encodeToWire(programOutput, maybePattern);
+
+console.log('Binary output size:', binaryOutput.length, 'bytes');
+console.log('  Bytes:', binaryOutput.toString('hex'));
+console.log();
+
+// ============================================================================
+// Step 6: Print (BINARY → TEXT)
+// ============================================================================
+
+console.log('--- PRINT (binary → text) ---');
+const { value: finalValue } = decodeWire(binaryOutput);
+
+console.log('Output value:', finalValue.toString());
+console.log();
+
+// ============================================================================
+// Summary
+// ============================================================================
+
+console.log('=== Pipeline Summary ===');
+console.log('Input:  ', inputValue.toString(), '(true inside Maybe)');
+console.log('Transform: NOT operation on inner bool');
+console.log('Output: ', finalValue.toString(), '(false inside Maybe)');
+console.log();
+console.log('Binary representations are canonical and type-safe!');