@fraczak/k 7.1.0 → 7.2.0

package/CHANGELOG.md

@@ -5,6 +5,39 @@ All notable changes to this project will be documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [7.2.0] — 2026-10-17 — Dynamic JSON Codec (djson) & Lazy Wire Decoding
+
+### Features & Performance Improvements
+
+- **Universal Dynamic JSON Codec (`codecs/djson.mjs`, `k-djson`)**:
+  - Added a standalone dynamic JSON codec (`djson.mjs` / `k-djson`)
+    based on a single 29-node recursive pattern `JSON = < number,
+    string, bool, null, list, object >`, where arrays are encoded as
+    arbitrary-length `cons`/`nil` lists and objects as `cons`/`nil`
+    lists of `{ key, val }` pairs.
+  - Kept the existing schema-specializing `json.mjs` (`k-json`) as an
+    alternative record-oriented JSON codec.
+  - Integrated `djson.mjs` into the CLI, REPL, and Web REPL (`repl.html`
+    VFS).
+
+- **Lazy Prefix-Free Wire Decoding & Fast Bitstream I/O**:
+  - Implemented lazy wire decoding (`LazyGeneralProduct` and
+    `LazyGeneralVariant`) with precomputed fixed-width bit sizes and
+    static field offsets on pattern graphs.
+  - Optimized bitstream reading/writing (`BitReader` / `BitWriter`) with
+    word-level operations, pre-encoded closed pattern bytes, and direct
+    bitstream-to-codepoint UTF-8 string decoding without intermediate
+    AST allocations.
+
+- **Stack-Safe & Bounded Value Rendering**:
+  - Converted `valueToK`, `valueWithEnvelopeToK`, and wire serialization
+    to iterative stack-safe algorithms with early character truncation
+    (`maxChars`).
+  - Added `:limit [num|none]` command and `--limit=<num>` CLI flag in
+    `repl.mjs` (default: `4096` chars) and bounded DOM tree previews in
+    `repl.html`.
+  - Handled broken pipe (`EPIPE`) cleanly across codec CLI entrypoints.
+
 ## [7.1.0] — 2026-10-16 — Polymorphic K IR (PKIR) & CLI Standardization
 
 ### Features & Architecture Improvements

package/DOCS/CODECS.md

@@ -141,6 +141,40 @@ printf 'yes\n' | ./codecs/yesno.mjs --parse | ./codecs/yesno.mjs --print
 # Output: yes
 ```
 
+## Built-in Codecs
+
+The `codecs/` directory provides the following standard codecs:
+
+- **`int.mjs` (`k-int`)**: Decimal integers and bracketed integer lists
+  (e.g. `42`, `-15`, `[1, 2, 3]`) mapped to signed binary bit trees and
+  `cons`/`nil` lists.
+- **`ieee.mjs` (`k-ieee`)**: IEEE 754 64-bit floating-point numbers
+  (`sign`, `exponent`, `fraction`).
+- **`utf8.mjs` (`k-utf8`) & `utf16.mjs` (`k-utf16`)**: Unicode strings
+  encoded as `cons`/`nil` code-point lists (`str` in `core.k`).
+- **`unit.mjs` (`k-unit`)**: The unit value `{}`.
+- **`json.mjs` (`k-json`) — Schema-Specializing JSON**:
+  Derives a document-specific pattern for each JSON value:
+  - JSON objects map directly to products `{ field: val, ... }`.
+  - JSON arrays map to indexed products `{ "0": a0, "1": a1, ... }`.
+  - Ideal for record-oriented projection (e.g. `.user .name`).
+- **`djson.mjs` (`k-djson`) — Universal Dynamic JSON**:
+  Maps every JSON document to a single universal recursive pattern
+  `JSON`:
+  ```k
+  ?<
+    float64 number,
+    str string,
+    < {} true, {} false > bool,
+    {} null,
+    < {} nil, { JSON car, LIST cdr } cons >=LIST list,
+    < {} nil, { { str key, JSON val } car, OBJ cdr } cons >=OBJ object
+  >=JSON
+  ```
+  Because every JSON value shares the exact same static 29-node pattern,
+  a `k` relation can be compiled once against `JSON` and process
+  arbitrary JSON documents of any shape or array/object length.
+
 ## Testing a Codec
 
 For CLI codecs, test both directions in a pipeline:
@@ -174,8 +208,9 @@ npm test
 ## Web REPL (VFS) Bundling
 
 When `repl.html` is generated (`npm run build:repl-html`):
-- Codecs (`int.mjs`, `json.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`) are
-  bundled into 100% self-contained ES modules with zero imports.
+- Codecs (`int.mjs`, `json.mjs`, `djson.mjs`, `utf8.mjs`, `unit.mjs`,
+  `ieee.mjs`) are bundled into 100% self-contained ES modules with zero
+  imports.
 - `runCodecCLI` is shimmed as a no-op, and `Value` helpers are directly
   inlined.
 - In the browser, the VFS serves these self-contained modules directly to

package/DOCS/CODEC_ARCHITECTURE.md

@@ -192,11 +192,16 @@ runCodecCLI(import.meta.url, { parse, print, doc });
    pathToFileURL(process.argv[1]).href`). If imported as a module or
    running in a browser, it is a no-op.
 2. **Flag Parsing**: Handles `--parse`, `--print`, and `-h` / `--help`.
-3. **Stream Piping**:
+3. **Stream Piping & Lazy Wire Decoding**:
    - For `--parse`: reads `stdin` to string, calls `parse(text)`, encodes
      enveloped value `(P, v)` to binary wire format on `stdout`.
-   - For `--print`: decodes wire format from `stdin` into `(P, v)`, calls
-     `print(value)`, writes formatted text to `stdout`.
+   - For `--print`: lazily decodes the wire format from `stdin` into
+     bitstream-backed `Value` proxies (`LazyGeneralProduct`,
+     `LazyGeneralVariant`) with precomputed fixed-width bit sizes and
+     static field offsets, calls `print(value)`, and writes formatted text
+     to `stdout`.
+   - Cleanly exits on `EPIPE` when piped into downstream consumers like
+     `head`.
 
 ---
 
@@ -207,7 +212,8 @@ To make `codecs/` intuitive and self-contained:
 ```text
 codecs/
 ├── int.mjs                    # Integer & list codec
-├── json.mjs                   # JSON codec
+├── json.mjs                   # JSON codec (schema-specializing)
+├── djson.mjs                  # Dynamic JSON codec (singleton pattern)
 ├── utf8.mjs                   # UTF-8 string codec
 ├── utf16.mjs                  # UTF-16 string codec
 ├── ieee.mjs                   # IEEE 754 float codec
@@ -236,7 +242,8 @@ In the browser REPL (`repl.html`):
   imports**.
 - **No Runtime Files in VFS**: `codecs/runtime/*` is not included in the
   browser VFS. The VFS only contains verified, self-contained user-facing
-  files (`int.mjs`, `json.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`).
+  files (`int.mjs`, `json.mjs`, `djson.mjs`, `utf8.mjs`, `unit.mjs`,
+  `ieee.mjs`).
 - **Native Browser Loading**: In `repl-codecs.mjs`, the browser reads the
   codec source from VFS, wraps it in a Blob, and invokes native `await
   import(blobUrl)`. No regex manipulation, import stripping, or global

package/DOCS/REPL.md

@@ -33,10 +33,11 @@ is complete.
 | Command | Meaning |
 | --- | --- |
 | `:help` | Show command summary |
-| `:engine [wasm|js]` | Display or switch evaluation engine (`wasm` or `js`) |
-| `:wasm` / `:js` | Shortcut to switch engine |
+| `:engine [wasm\|pkir\|js]` | Display or switch evaluation engine (`wasm`, `pkir`, or `js`) |
+| `:wasm` / `:pkir` / `:js` | Shortcut to switch engine |
 | `:rel name` | Show relation definition |
 | `:rels` | List relation aliases |
+| `:limit [num\|none]` | Set maximum characters for value output (default: 4096) |
 | `:codec load file` | Load a codec from an ES module file (e.g. `:codec load codecs/int.mjs`) |
 | `:codec unload name` | Unload a registered codec |
 | `:codec list` | List loaded REPL codecs (alias: `:codecs`) |
@@ -231,11 +232,13 @@ The REPL supports two evaluation engines:
 
 ### CLI Options
 
-When starting `k-repl`, choose the engine with:
+When starting `k-repl`, choose the engine or output limit with:
 
 ```bash
 k-repl --engine=js    # or k-repl --js
+k-repl --engine=pkir  # or k-repl --pkir
 k-repl --engine=wasm  # or k-repl --wasm (default)
+k-repl --limit=8192   # set max characters for rendered output
 ```
 
 ## Timing and Profiling
@@ -297,6 +300,11 @@ Evaluated values print in k syntax together with the inferred envelope:
 {}|0|+1 ?<<{} 0, ...> +1, ...>
 ```
 
+Large values are rendered iteratively and truncated at `maxChars`
+(default: `4096` characters) so deeply nested or large values print
+instantly without exhausting stack or terminal buffers. Use
+`:limit <num>` or `:limit none` to adjust or disable the limit.
+
 `undefined` prints as:
 
 ```text

package/DOCS/WEB_REPL_REQUIREMENTS.md

@@ -113,7 +113,7 @@ The bundled VFS and UI examples must include only verified useful files:
 #### 1. Curated Files Explorer
 In the Web REPL "Files" (VFS) explorer, the visible files are strictly curated to verified examples and codecs:
 - **k-codes**: `core.k`, `arithmetics.k`, `ieee.k`
-- **codecs**: `json.mjs`, `utf8.mjs`, `int.mjs`, `unit.mjs`, `ieee.mjs`
+- **codecs**: `json.mjs`, `djson.mjs`, `utf8.mjs`, `int.mjs`, `unit.mjs`, `ieee.mjs`
 - User-created or uploaded files remain visible.
 - Internal runtime artifacts (`backends/wasm/runtime.wat`) and redundant directory prefixes (`Examples/`, `codecs/`) are kept accessible internally but hidden from the default file listing.
 - Clicking "Load" on a `.mjs` file automatically executes `:codec load <filename>`.

package/README.md

@@ -223,7 +223,8 @@ For a detailed guide to all four formats, commands, and workflows, see
 
 | Command | Purpose |
 | --- | --- |
-| `k-json` | Convert JSON to/from the binary stream |
+| `k-json` | Convert JSON to/from the binary stream (schema-specializing) |
+| `k-djson` | Convert JSON to/from a universal dynamic JSON singleton pattern |
 | `k-int` | Convert decimal integers to/from the binary stream |
 | `k-ieee` | Convert float literals to/from the binary stream |
 | `k-unit` | Produce or validate the unit value |

package/Value.mjs

@@ -193,15 +193,40 @@ class Value {
 
   toJSON() {
     if (isVariant(this)) {
-      if (isProduct(this.value) && Object.keys(this.value.product).length === 0) {
+      if (
+        isProduct(this.value) &&
+        Object.keys(this.value.product).length === 0
+      ) {
         return this.tag;
       }
-      return {[this.tag]: this.value.toJSON()};
+      const tags = [];
+      let curr = this;
+      while (isVariant(curr)) {
+        if (
+          isProduct(curr.value) &&
+          Object.keys(curr.value.product).length === 0
+        ) {
+          let res = curr.tag;
+          for (let i = tags.length - 1; i >= 0; i--) {
+            res = { [tags[i]]: res };
+          }
+          return res;
+        }
+        tags.push(curr.tag);
+        curr = curr.value;
+      }
+      let res = curr
+        ? (typeof curr.toJSON === "function" ? curr.toJSON() : curr)
+        : null;
+      for (let i = tags.length - 1; i >= 0; i--) {
+        res = { [tags[i]]: res };
+      }
+      return res;
     }
     if (!isProduct(this)) return { type: this.type };
     let vector = toVector(this.product);
     if (vector.length > 0) return vector;
-    return this.product; 
+    return this.product;
   }
 }
 export { Value, fromObject, cloneSubpattern, edgeSubpattern, composePattern, mergePatterns, withPattern, isProduct, isVariant, isValue };

package/backend-api.mjs

@@ -1,6 +1,10 @@
 export { annotate } from "./index.mjs";
 export { KVM_FORMAT, KVM_VERSION, lowerKIRRelationToKVM, lowerKIRToKVM, lowerToKVM, executeKVM, objectToKVMArtifact, specializeKVM } from "./kvm.mjs";
-export { decodeWire, encodeToWire } from "./codecs/runtime/prefix-codec.mjs";
+export {
+  decodeWire,
+  decodeWireLazy,
+  encodeToWire
+} from "./codecs/runtime/prefix-codec.mjs";
 export { exportPatternGraph, NODE_KIND } from "./codecs/runtime/codec.mjs";
 export { patternToPropertyList, propertyListToPattern } from "./codecs/runtime/pattern-json.mjs";
 export { Value, fromObject, isProduct, isVariant, isValue } from "./Value.mjs";

package/browser/tree-view.mjs

@@ -101,7 +101,9 @@ export function renderValueNode(val, currentDepth = 1, depthLimit = 3) {
     const populateChildren = () => {
       if (populated) return;
       populated = true;
-      for (const k of keys) {
+      const maxFields = 100;
+      const displayKeys = keys.slice(0, maxFields);
+      for (const k of displayKeys) {
         const row = document.createElement("div");
         row.className = "k-tree-row";
 
@@ -113,12 +115,25 @@ export function renderValueNode(val, currentDepth = 1, depthLimit = 3) {
         const childContainer = document.createElement("div");
         childContainer.className = "k-tree-child-val";
         childContainer.appendChild(
-          renderValueNode(val.product[k], isOpen ? currentDepth + 1 : 1, depthLimit)
+          renderValueNode(
+            val.product[k],
+            isOpen ? currentDepth + 1 : 1,
+            depthLimit
+          )
         );
         row.appendChild(childContainer);
 
         childrenContainer.appendChild(row);
       }
+      if (keys.length > maxFields) {
+        const moreRow = document.createElement("div");
+        moreRow.className = "k-tree-row k-tree-more";
+        moreRow.style.color = "var(--color-muted)";
+        moreRow.style.paddingLeft = "1.2rem";
+        moreRow.textContent =
+          `... and ${keys.length - maxFields} more fields`;
+        childrenContainer.appendChild(moreRow);
+      }
     };
 
     if (isOpen) {
@@ -169,13 +184,15 @@ export function renderValueNode(val, currentDepth = 1, depthLimit = 3) {
     if (isVariant(val.value)) {
       let chainCount = 0;
       let curr = val.value;
-      while (isVariant(curr)) {
+      while (isVariant(curr) && chainCount < 20) {
         chainCount++;
         curr = curr.value;
       }
       const badge = document.createElement("span");
       badge.className = "k-tree-badge";
-      badge.textContent = `+${chainCount} tags`;
+      badge.textContent = isVariant(curr)
+        ? "+20+ tags"
+        : `+${chainCount} tags`;
       summary.appendChild(badge);
     }
 
@@ -486,7 +503,15 @@ export function createOutputEntryElement({
 
   const valSummary = document.createElement("summary");
   valSummary.className = "k-tree-section-summary";
-  valSummary.innerHTML = `<span class="k-tree-arrow">▶</span> <span class="k-section-title">🌲 Value</span> <span class="k-section-preview">${escapeHtml(valueToK(value))}</span>`;
+  const rawValPreview = valueToK(value, 120);
+  const valPreview = rawValPreview.length > 120
+    ? `${rawValPreview.slice(0, 120)}...`
+    : rawValPreview;
+  const valEscaped = escapeHtml(valPreview);
+  valSummary.innerHTML =
+    `<span class="k-tree-arrow">▶</span> ` +
+    `<span class="k-section-title">🌲 Value</span> ` +
+    `<span class="k-section-preview">${valEscaped}</span>`;
   valSection.appendChild(valSummary);
 
   const valBody = document.createElement("div");
@@ -504,8 +529,15 @@ export function createOutputEntryElement({
 
     const patSummary = document.createElement("summary");
     patSummary.className = "k-tree-section-summary";
-    const patPreview = propertyListToFilter(value.pattern);
-    patSummary.innerHTML = `<span class="k-tree-arrow">▶</span> <span class="k-section-title">🌿 Pattern</span> <span class="k-section-preview">?${escapeHtml(patPreview)}</span>`;
+    const rawPatPreview = propertyListToFilter(value.pattern, "X", 120);
+    const patPreview = rawPatPreview.length > 120
+      ? `${rawPatPreview.slice(0, 120)}...`
+      : rawPatPreview;
+    const patEscaped = escapeHtml(patPreview);
+    patSummary.innerHTML =
+      `<span class="k-tree-arrow">▶</span> ` +
+      `<span class="k-section-title">🌿 Pattern</span> ` +
+      `<span class="k-section-preview">?${patEscaped}</span>`;
     patSection.appendChild(patSummary);
 
     patBody = document.createElement("div");
@@ -545,7 +577,14 @@ export function createOutputEntryElement({
 
   const rawPre = document.createElement("pre");
   rawPre.className = "k-raw-pre";
-  rawPre.textContent = rawText;
+  const MAX_RAW_PRE_CHARS = 100000;
+  if (rawText.length > MAX_RAW_PRE_CHARS) {
+    rawPre.textContent =
+      rawText.slice(0, MAX_RAW_PRE_CHARS) +
+      `\n\n... [truncated, ${rawText.length - MAX_RAW_PRE_CHARS} more characters]`;
+  } else {
+    rawPre.textContent = rawText;
+  }
   rawWrapper.appendChild(rawPre);
 
   container.appendChild(rawWrapper);