@fraczak/k 5.1.2 → 6.0.1

package/codecs/README.md

@@ -0,0 +1,215 @@
+# K Codecs
+
+This directory is being rewritten around a new codec design:
+
+```text
+abstract pattern graph + prefix-free tree encoding
+```
+
+The default concrete format is now the binary encoding of a `$pattern` value,
+using the `$pattern` type from [`../core.k`](../core.k), immediately followed by
+the binary encoding of the value under that decoded pattern.
+
+There is no separate JSON container format.
+
+## Objectives
+
+The rewrite is guided by two hard requirements:
+
+1. compactness close to the structural lower bound,
+2. efficient `k` projections, especially `.field` and `/tag`.
+
+The current target architecture is the pattern graph plus prefix-free value-tree
+stream described here and in
+[`POLYMORPHIC_BINARY_FORMAT.md`](./POLYMORPHIC_BINARY_FORMAT.md).
+
+## Core Idea
+
+The semantic object serialized by the new codec is:
+
+```text
+(P, v)
+```
+
+where:
+
+- `P` is an abstract rooted pattern graph,
+- `v` is a value tree compatible with `P`.
+
+The same pair is now the in-memory runtime model as well. Decoding a stream
+attaches `P` to the resulting `Value` as `value.pattern`; evaluation propagates
+that pattern through projections and constructors; encoding a `Value` uses its
+carried pattern unless an explicit pattern is supplied.
+
+The base value encoding is:
+
+- tree-based, not DAG-based,
+- prefix-free,
+- interpreted relative to the pattern graph,
+- free of repeated field names and tag names in the value payload.
+
+The base codec does **not** perform DAG compression. If sharing is desired, it
+should happen as a higher-level transform, not inside the primitive value
+format.
+
+## Wire Format
+
+The default transport format is:
+
+```text
+encode($pattern_value : $pattern) encode(value : decoded_pattern)
+```
+
+`$pattern_value` is a normal k value whose shape is defined in `core.k`. After
+that value is decoded, it becomes the pattern used to decode the remaining
+value payload.
+
+The runtime boundary is therefore:
+
+```text
+binary pattern+value stream <-> Value(pattern, tree)
+```
+
+`k.mjs` is only the command-line adapter for that boundary. The operational
+runtime sees and preserves the pattern on the `Value` itself.
+
+## Pattern Graph Representation
+
+For documentation and tests, a pattern graph may be shown as a property-list
+style vector of nodes:
+
+```text
+[kind, edges]
+```
+
+where:
+
+- `kind` is one of:
+  - `"any"`
+  - `"open-product"`
+  - `"open-union"`
+  - `"closed-product"`
+  - `"closed-union"`
+- `edges` is a list of:
+  - `[label, target_node_id]`
+
+Example:
+
+```json
+[
+  ["closed-union", [["nil", 1], ["cons", 2]]],
+  ["closed-product", []],
+  ["closed-product", [["car", 3], ["cdr", 0]]],
+  ["closed-union", [["_", 1], ["0", 3], ["1", 3]]]
+]
+```
+
+Canonical rules:
+
+- root node is node `0`,
+- node list order is canonical graph-discovery order,
+- edges are sorted by label,
+- edge labels are unique within a node,
+- `"any"` must have no outgoing edges.
+
+This JSON-like graph is only a readable notation. The wire representation is
+the ordinary k `$pattern` value from `core.k`.
+
+## Witness-Derived Patterns
+
+When `k-parse` is used without an explicit input pattern or type, the wire
+pattern is derived from the parsed value tree.
+
+- An empty textual node is a closed product: `{}`.
+- A textual node with multiple children is a closed product.
+- A textual node with one child is interpreted as an open union by default.
+- An explicit product input pattern may force such a node to be treated as a
+  singleton product instead.
+
+For example, parsing:
+
+```text
+{a:{b:x,c:{}}}
+```
+
+without an input pattern derives:
+
+```json
+[
+  ["open-union", [["a", 1]]],
+  ["closed-product", [["b", 2], ["c", 3]]],
+  ["open-union", [["x", 3]]],
+  ["closed-product", []]
+]
+```
+
+During this construction, finite closed pattern subtrees are hash-consed from
+the leaves upward. The starting closed leaf is `["closed-product", []]`. Two
+closed nodes collapse only when they have the same closed kind, the same labels,
+and the same already-collapsed child targets. Open nodes keep their identity,
+and recursive closed nodes are not collapsed by this witness-tree rule.
+
+This is canonicalization of the pattern graph carried in the wire stream. It is
+not DAG compression of the value payload.
+
+## Prefix-Free Value Encoding
+
+The value payload is a bitstream interpreted relative to the pattern graph.
+
+Base rules:
+
+- closed or open product:
+  - encode child values in canonical edge order,
+- closed or open union:
+  - encode the selected tag position,
+  - then encode the selected payload,
+- `any`:
+  - not directly value-carrying in the base format.
+
+The initial concrete union encoding rule is:
+
+```text
+width = ceil(log2(cardinality))
+emit tag ordinal in width bits
+```
+
+This is the simplest canonical prefix-free choice encoding. It is not the final
+word on compression, but it gives a regular and compact base encoding that can
+later be refined without changing the abstract pattern/value model.
+
+## Projections
+
+The base tree encoding is chosen so that:
+
+- `/tag` only needs to read the union choice at the current node,
+- `.field` follows canonical product order and may later be accelerated by
+  higher-level indexes if necessary.
+
+In the materialized runtime, projections also project the carried pattern:
+`.field` returns the subpattern at that product field, and `/tag` returns the
+subpattern at that union tag. If no carried subpattern is available, later
+encoding can still derive a witness pattern from the result tree.
+
+The primitive codec stays minimal. Projection indexes, framing, or sharing
+schemes are explicitly separate concerns.
+
+## Design Boundary
+
+The new codec work is split into layers:
+
+1. abstract pattern graph semantics,
+2. prefix-free value-tree semantics relative to that graph,
+3. self-hosted `k` representation of the pattern graph,
+4. optional higher-level optimizations.
+
+## Files
+
+- [`POLYMORPHIC_BINARY_FORMAT.md`](./POLYMORPHIC_BINARY_FORMAT.md):
+  main design document for the new pattern-plus-prefix-tree codec.
+- [`BINARY_FORMAT.md`](./BINARY_FORMAT.md):
+  notes on future binary packaging of the same abstract semantics.
+
+## Status
+
+The active command-line pipeline emits and consumes the self-hosted binary
+pattern+value stream.

package/codecs/example-pipeline.mjs

@@ -0,0 +1,149 @@
+#!/usr/bin/env node
+/**
+ * Example: Demonstrating the codec pipeline
+ * 
+ * Shows how to:
+ * 1. Define types with canonical names
+ * 2. Create values
+ * 3. Encode to binary
+ * 4. Pass through "compiled program" (identity)
+ * 5. Decode from binary
+ * 6. Print result
+ */
+
+import { deriveClosedPattern } from './runtime/codec.mjs';
+import { encodeToWire, decodeWire } from './runtime/prefix-codec.mjs';
+import { patternToPropertyList } from './runtime/pattern-json.mjs';
+import { Product, Variant } from '../Value.mjs';
+import hash from '../hash.mjs';
+
+console.log('=== K Codec Pipeline Example ===\n');
+
+// ============================================================================
+// Step 1: Define types (normally from compiler)
+// ============================================================================
+
+const types = {};
+
+// Unit: {}
+const unitDef = { code: 'product', product: {} };
+const unitName = hash('$C0={};');
+types[unitName] = unitDef;
+
+// Bool: <{} false, {} true>
+const boolDef = {
+  code: 'union',
+  union: { false: unitName, true: unitName }
+};
+const boolName = hash(`$C0=<${unitName}"false",${unitName}"true">;`);
+types[boolName] = boolDef;
+
+// Maybe: <{} nothing, Bool something>
+const maybeDef = {
+  code: 'union',
+  union: { nothing: unitName, something: boolName }
+};
+const maybeName = hash(
+  `$C0=<${unitName}"nothing",${boolName}"something">;`
+);
+types[maybeName] = maybeDef;
+
+console.log('Types defined:');
+console.log('  unit:', unitName);
+console.log('  bool:', boolName);
+console.log('  maybe:', maybeName);
+console.log();
+
+// ============================================================================
+// Step 2: Type resolver
+// ============================================================================
+
+function resolveType(typeName) {
+  const def = types[typeName];
+  if (!def) {
+    throw new Error(`Unknown type: ${typeName}`);
+  }
+  return def;
+}
+
+// ============================================================================
+// Step 3: Create input value: Maybe Bool = something(true)
+// ============================================================================
+
+const inputValue = new Variant(
+  'something',
+  new Variant('true', new Product({}))
+);
+
+console.log('Input value:', inputValue.toString());
+console.log();
+
+const maybePattern = patternToPropertyList(
+  deriveClosedPattern(maybeName, maybeDef, resolveType)
+);
+
+// ============================================================================
+// Step 4: Parse (TEXT → BINARY)
+// ============================================================================
+
+console.log('--- PARSE (text → binary) ---');
+const binaryInput = encodeToWire(inputValue, maybePattern);
+console.log('Binary size:', binaryInput.length, 'bytes');
+console.log('  Bytes:', binaryInput.toString('hex'));
+console.log();
+
+// ============================================================================
+// Step 5: Compiled k-program (BINARY → BINARY)
+// ============================================================================
+
+console.log('--- COMPILED PROGRAM (binary → binary) ---');
+
+// Simulate a compiled k-program: decode → transform → encode
+const { value: programInput } = decodeWire(binaryInput);
+
+console.log('Program sees value:', programInput.toString());
+
+// Example transformation: NOT operation (flip bool inside Maybe)
+let programOutput;
+if (programInput instanceof Variant && programInput.tag === 'something') {
+  const innerBool = programInput.value;
+  if (innerBool instanceof Variant) {
+    const flipped = new Variant(
+      innerBool.tag === 'true' ? 'false' : 'true',
+      innerBool.value
+    );
+    programOutput = new Variant('something', flipped);
+  }
+} else {
+  programOutput = programInput; // nothing → nothing
+}
+
+console.log('Program output value:', programOutput.toString());
+
+// Encode output
+const binaryOutput = encodeToWire(programOutput, maybePattern);
+
+console.log('Binary output size:', binaryOutput.length, 'bytes');
+console.log('  Bytes:', binaryOutput.toString('hex'));
+console.log();
+
+// ============================================================================
+// Step 6: Print (BINARY → TEXT)
+// ============================================================================
+
+console.log('--- PRINT (binary → text) ---');
+const { value: finalValue } = decodeWire(binaryOutput);
+
+console.log('Output value:', finalValue.toString());
+console.log();
+
+// ============================================================================
+// Summary
+// ============================================================================
+
+console.log('=== Pipeline Summary ===');
+console.log('Input:  ', inputValue.toString(), '(true inside Maybe)');
+console.log('Transform: NOT operation on inner bool');
+console.log('Output: ', finalValue.toString(), '(false inside Maybe)');
+console.log();
+console.log('Binary representations are canonical and type-safe!');

package/codecs/ieee.mjs

@@ -0,0 +1,140 @@
+#!/usr/bin/env node
+
+import { stdin, stdout, argv, exit } from "node:process";
+import { Product, Variant } from "../Value.mjs";
+import { decodeWire, encodeToWire } from "./runtime/prefix-codec.mjs";
+import { FLOAT64_PATTERN } from "./runtime/ieee-pattern.mjs";
+const UNIT = new Product({});
+
+function bitValue(bit) {
+  return new Variant(bit === 0 ? "0" : "1", UNIT);
+}
+
+function bitsProduct(width, value) {
+  const big = BigInt(value);
+  const product = {};
+  for (let i = width - 1; i >= 0; i--) {
+    product[String(i)] = bitValue(Number((big >> BigInt(i)) & 1n));
+  }
+  return new Product(product);
+}
+
+function encodeNumberToValue(number) {
+  const buf = Buffer.alloc(8);
+  buf.writeDoubleBE(number, 0);
+  const bits = buf.readBigUInt64BE(0);
+  const sign = Number((bits >> 63n) & 1n);
+  const exponent = Number((bits >> 52n) & 0x7ffn);
+  const fraction = bits & ((1n << 52n) - 1n);
+
+  return new Product({
+    sign: new Variant(sign === 0 ? "+" : "-", UNIT),
+    exponent: bitsProduct(11, exponent),
+    fraction: bitsProduct(52, fraction)
+  });
+}
+
+function requireProduct(value, where) {
+  if (!(value instanceof Product)) {
+    throw new Error(`${where}: expected Product`);
+  }
+  return value.product;
+}
+
+function requireVariant(value, where) {
+  if (!(value instanceof Variant)) {
+    throw new Error(`${where}: expected Variant`);
+  }
+  return value;
+}
+
+function parseBitsProduct(value, maxBit, where) {
+  const product = requireProduct(value, where);
+  let n = 0n;
+  for (let i = maxBit; i >= 0; i--) {
+    const entry = requireVariant(product[String(i)], `${where}.bit${i}`);
+    if (entry.tag !== "0" && entry.tag !== "1") {
+      throw new Error(`${where}.bit${i}: expected tag 0 or 1`);
+    }
+    n = (n << 1n) | BigInt(entry.tag === "1" ? 1 : 0);
+  }
+  return n;
+}
+
+function decodeValueToNumber(value) {
+  const product = requireProduct(value, "float64");
+  const sign = requireVariant(product.sign, "float64.sign");
+  if (sign.tag !== "+" && sign.tag !== "-") {
+    throw new Error(`float64.sign: expected + or -, got ${sign.tag}`);
+  }
+  const exponent = parseBitsProduct(product.exponent, 10, "float64.exponent");
+  const fraction = parseBitsProduct(product.fraction, 51, "float64.fraction");
+
+  const bits =
+    (BigInt(sign.tag === "-" ? 1 : 0) << 63n) |
+    (exponent << 52n) |
+    fraction;
+
+  const buf = Buffer.alloc(8);
+  buf.writeBigUInt64BE(bits, 0);
+  return buf.readDoubleBE(0);
+}
+
+function parseFloatText(text) {
+  const trimmed = text.trim();
+  if (trimmed === "") {
+    throw new Error("Expected a floating-point literal");
+  }
+
+  if (/^[+-]?nan$/i.test(trimmed)) return NaN;
+  if (/^[+]?inf(inity)?$/i.test(trimmed)) return Infinity;
+  if (/^-inf(inity)?$/i.test(trimmed)) return -Infinity;
+
+  const value = Number(trimmed);
+  if (Number.isNaN(value)) {
+    throw new Error(`Invalid floating-point literal: ${JSON.stringify(trimmed)}`);
+  }
+  return value;
+}
+
+function printFloatText(number) {
+  if (Number.isNaN(number)) return "NaN";
+  if (number === Infinity) return "Infinity";
+  if (number === -Infinity) return "-Infinity";
+  if (Object.is(number, -0)) return "-0";
+  return String(number);
+}
+
+function readAll(stream) {
+  return new Promise((resolve, reject) => {
+    const chunks = [];
+    stream.on("data", (chunk) => chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)));
+    stream.on("end", () => resolve(Buffer.concat(chunks)));
+    stream.on("error", reject);
+  });
+}
+
+async function main() {
+  const args = argv.slice(2);
+  if (args.length !== 1 || (args[0] !== "--parse" && args[0] !== "--print")) {
+    console.error("Usage: ieee.mjs --parse | --print");
+    console.error("  --parse  read a float literal from stdin, write binary pattern+value stream");
+    console.error("  --print  read binary pattern+value stream from stdin, write a float literal");
+    exit(1);
+  }
+
+  const buf = await readAll(stdin);
+
+  if (args[0] === "--parse") {
+    const value = encodeNumberToValue(parseFloatText(buf.toString("utf8")));
+    stdout.write(encodeToWire(value, FLOAT64_PATTERN));
+  } else {
+    const { value } = decodeWire(buf);
+    stdout.write(`${printFloatText(decodeValueToNumber(value))}\n`);
+  }
+}
+
+main().catch((error) => {
+  console.error(error.message || String(error));
+  exit(1);
+});

package/codecs/int.mjs

@@ -0,0 +1,104 @@
+#!/usr/bin/env node
+
+/**
+ * int: parse/print integers in the k 'int' binary pattern+value stream.
+ *
+ * $ bits = < {} _, bits 0, bits 1 >;
+ * $ int  = < bits '+', bits '-' >;
+ *
+ * Bits are stored MSB-outermost (remove_leading_zeros strips outermost 0s).
+ *
+ * Usage:
+ *   echo "-21"  | int.mjs --parse   # decimal → binary pattern+value stream
+ *   <wire>      | int.mjs --print   # binary pattern+value stream → decimal
+ */
+
+import { stdin, stdout, argv, exit } from "node:process";
+import { Product, Variant } from "../Value.mjs";
+import { decodeWire, encodeToWire } from "./runtime/prefix-codec.mjs";
+
+// Closed pattern for $ int = < bits '+', bits '-' >
+// with $ bits = < {} _, bits 0, bits 1 >
+const INT_PATTERN = [
+  ["closed-union", [["+", 1], ["-", 1]]],
+  ["closed-union", [["0", 1], ["1", 1], ["_", 2]]],
+  ["closed-product", []]
+];
+
+function readAll(stream) {
+  return new Promise((resolve, reject) => {
+    const chunks = [];
+    stream.on("data", c => chunks.push(Buffer.isBuffer(c) ? c : Buffer.from(c)));
+    stream.on("end", () => resolve(Buffer.concat(chunks)));
+    stream.on("error", reject);
+  });
+}
+
+// Build bits Value (MSB outermost) from a non-negative BigInt.
+function buildBits(n) {
+  const bitChars = n === 0n ? ["0"] : n.toString(2).split("");
+  let v = new Variant("_", new Product({}));
+  for (let i = bitChars.length - 1; i >= 0; i--) {
+    v = new Variant(bitChars[i], v);
+  }
+  return v;
+}
+
+// Parse a decimal integer string ([-+]?[ ]*[1-9][0-9]*|0) into a k int Value.
+function parseIntStr(str) {
+  str = str.trim();
+  let sign = "+";
+  if (str[0] === "-") { sign = "-"; str = str.slice(1).replace(/\s+/g, ""); }
+  else if (str[0] === "+") { str = str.slice(1).replace(/\s+/g, ""); }
+  if (!/^(0|[1-9][0-9]*)$/.test(str)) {
+    throw new Error(`Invalid integer syntax: ${JSON.stringify(str)}`);
+  }
+  const n = BigInt(str);
+  if (n === 0n) sign = "+"; // zero is always '+'
+  return new Variant(sign, buildBits(n));
+}
+
+// Walk a decoded int Value (Variant sign → bits) and return a decimal string.
+function printIntValue(value) {
+  if (!(value instanceof Variant) || (value.tag !== "+" && value.tag !== "-")) {
+    throw new Error("Not a valid k int value");
+  }
+  const sign = value.tag;
+  let bits = "";
+  let node = value.value;
+  while (node instanceof Variant && node.tag !== "_") {
+    bits += node.tag;
+    node = node.value;
+  }
+  // bits is MSB-first binary string (may be empty for zero represented as "0")
+  const n = bits === "" ? 0n : BigInt("0b" + bits);
+  const digits = n.toString(10);
+  if (n === 0n) return "0";
+  return (sign === "-" ? "-" : "") + digits;
+}
+
+async function main() {
+  const args = argv.slice(2);
+  if (args.length !== 1 || (args[0] !== "--parse" && args[0] !== "--print")) {
+    console.error("Usage: int.mjs --parse | --print");
+    console.error("  --parse  read a decimal integer from stdin, write binary pattern+value stream");
+    console.error("  --print  read binary pattern+value stream from stdin, write decimal integer");
+    exit(1);
+  }
+
+  const buf = await readAll(stdin);
+
+  if (args[0] === "--parse") {
+    const text = buf.toString("utf8").trim();
+    const value = parseIntStr(text);
+    stdout.write(encodeToWire(value, INT_PATTERN));
+  } else {
+    const { value } = decodeWire(buf);
+    stdout.write(printIntValue(value) + "\n");
+  }
+}
+
+main().catch(err => {
+  console.error(err.message || String(err));
+  exit(1);
+});