@fraczak/k 7.1.0 → 8.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (64) hide show
  1. package/CHANGELOG.md +99 -0
  2. package/DOCS/BIG_PICTURE.md +54 -7
  3. package/DOCS/CODECS.md +47 -2
  4. package/DOCS/CODEC_ARCHITECTURE.md +14 -5
  5. package/DOCS/DICTIONARY.md +13 -4
  6. package/DOCS/FILE_FORMATS.md +3 -2
  7. package/DOCS/JSON_APPROACHES_COMPARISON.md +306 -0
  8. package/DOCS/NAMED_PATTERNS.md +317 -0
  9. package/DOCS/PATTERNS.md +59 -11
  10. package/DOCS/REPL.md +24 -5
  11. package/DOCS/WEB_REPL_REQUIREMENTS.md +4 -2
  12. package/Examples/arithmetics.k +22 -17
  13. package/Examples/bnat.k +21 -11
  14. package/Examples/buda.k +37 -20
  15. package/Examples/byte.k +3 -1
  16. package/Examples/code.k +38 -39
  17. package/Examples/djson-btrie.k +414 -0
  18. package/Examples/djson-ptrie.k +663 -0
  19. package/Examples/djson.k +323 -0
  20. package/Examples/esc.k +1 -1
  21. package/Examples/karatsuba-mult.k +16 -13
  22. package/Examples/list.k +8 -4
  23. package/Examples/nat.k +17 -11
  24. package/Examples/poly.k +11 -3
  25. package/Examples/vhdl_to_k.k +43 -31
  26. package/README.md +42 -15
  27. package/Value.mjs +28 -3
  28. package/backend-api.mjs +5 -1
  29. package/browser/repl-browser.mjs +6 -0
  30. package/browser/tree-view.mjs +82 -24
  31. package/browser/vfs-data.mjs +21 -12
  32. package/codecs/README.md +3 -0
  33. package/codecs/djson-btrie.mjs +560 -0
  34. package/codecs/djson-ptrie.mjs +587 -0
  35. package/codecs/djson.mjs +270 -0
  36. package/codecs/int.mjs +2 -1
  37. package/codecs/json.mjs +7 -2
  38. package/codecs/k-parse.mjs +7 -1
  39. package/codecs/k-print.mjs +4 -0
  40. package/codecs/runtime/codec-sdk.mjs +16 -2
  41. package/codecs/runtime/pattern-json.mjs +80 -1
  42. package/codecs/runtime/prefix-codec.mjs +629 -67
  43. package/codecs/runtime/show-value.mjs +93 -51
  44. package/codecs/runtime/unicode-string.mjs +29 -6
  45. package/codecs/show.mjs +4 -0
  46. package/codecs/utf16.mjs +2 -1
  47. package/core.k +79 -115
  48. package/index.mjs +3 -0
  49. package/named-patterns.mjs +267 -0
  50. package/object.mjs +31 -3
  51. package/package.json +5 -1
  52. package/parser.jison +68 -5
  53. package/parser.mjs +111 -60
  54. package/pretty.mjs +5 -0
  55. package/repl.mjs +245 -28
  56. package/scripts/build-repl-html.mjs +76 -5
  57. package/scripts/perf-djson.mjs +1142 -0
  58. package/scripts/run-tests.mjs +5 -0
  59. package/symbol-table.mjs +20 -2
  60. package/tests/bench-djson-tries.mjs +340 -0
  61. package/tests/test-codecs.mjs +211 -0
  62. package/tests/test-named-patterns.mjs +189 -0
  63. package/tests/test-repl-html.mjs +52 -4
  64. package/tests/test-repl.mjs +223 -0
package/CHANGELOG.md CHANGED
@@ -5,6 +5,90 @@ All notable changes to this project will be documented here.
5
5
  The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
6
6
  This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [8.0.0] — 2026-10-20 — Named Patterns, Type Schemes & Trie Codecs
9
+
10
+ ### Language & Type System
11
+
12
+ - **Named Patterns & Parameterized Schemes (`named-patterns.mjs`)**:
13
+ - Introduced first-class syntax for closed named patterns:
14
+ `byte[] = { bit 0, bit 1, ... };` and `bool[] = < bit false, ... >;`.
15
+ - Added parameterized type schemes with explicit type variables:
16
+ `list[X] = < X head, (list[X]) tail >;` and
17
+ `pair[A, B] = { A first, B second };`.
18
+ - Implemented monomorphic expansion with cycle detection and sharing.
19
+ - Full object file support: `.ko` and `.klib` payloads serialize and
20
+ deserialize named pattern tables alongside relations.
21
+
22
+ - **Standard Library & Examples Refactoring**:
23
+ - Adopted named patterns throughout `core.k` (`bit[]`, `byte[]`,
24
+ `pair[A, B]`, `either[A, B]`, `option[X]`).
25
+ - Refactored `Examples/*.k` (`djson.k`, `arithmetics.k`, `bnat.k`,
26
+ `nat.k`, `code.k`, `karatsuba-mult.k`, `poly.k`, `vhdl_to_k.k`)
27
+ for clarity and conciseness using named patterns.
28
+
29
+ ### Dynamic JSON Codecs
30
+
31
+ - **Bit-Trie Codec (`codecs/djson-btrie.mjs`, `k-djson-btrie`)**:
32
+ - Added binary bit-trie dynamic JSON codec with variable-length
33
+ bit prefixing for compact wire encodings.
34
+ - Accompanied by formal K specification in `Examples/djson-btrie.k`.
35
+
36
+ - **Packed Nibble-Trie Codec (`codecs/djson-ptrie.mjs`)**:
37
+ - Added packed 4-bit nibble trie codec (`k-djson-ptrie`) for
38
+ optimized traversal.
39
+ - Accompanied by formal K specification in `Examples/djson-ptrie.k`.
40
+
41
+ - **JSON Approaches Comparison & Benchmarks**:
42
+ - Added comparative architecture and benchmark documentation in
43
+ `DOCS/JSON_APPROACHES_COMPARISON.md`.
44
+ - Added benchmark suite in `scripts/perf-djson.mjs` and
45
+ `tests/bench-djson-tries.mjs`.
46
+
47
+ ### Interactive REPL & Tooling
48
+
49
+ - **Pattern Inspection & Persistence in REPL**:
50
+ - Added `:patterns` / `:pat` commands to inspect loaded patterns.
51
+ - Display canonical type code / relation hashes:
52
+ `name[] = ... -- @hash`.
53
+ - Added pattern lookup by `@hash` (`:pattern @hash`, `:pat @hash`).
54
+ - Tab-completion support for pattern names and canonical `@hashes`.
55
+ - Full pattern persistence across REPL sessions via `.klib` files.
56
+
57
+ ---
58
+
59
+ ## [7.2.0] — 2026-10-17 — Dynamic JSON Codec (djson) & Lazy Wire Decoding
60
+
61
+ ### Features & Performance Improvements
62
+
63
+ - **Universal Dynamic JSON Codec (`codecs/djson.mjs`, `k-djson`)**:
64
+ - Added a standalone dynamic JSON codec (`djson.mjs` / `k-djson`)
65
+ based on a single 29-node recursive pattern `JSON = < number,
66
+ string, bool, null, list, object >`, where arrays are encoded as
67
+ arbitrary-length `cons`/`nil` lists and objects as `cons`/`nil`
68
+ lists of `{ key, val }` pairs.
69
+ - Kept the existing schema-specializing `json.mjs` (`k-json`) as an
70
+ alternative record-oriented JSON codec.
71
+ - Integrated `djson.mjs` into the CLI, REPL, and Web REPL (`repl.html`
72
+ VFS).
73
+
74
+ - **Lazy Prefix-Free Wire Decoding & Fast Bitstream I/O**:
75
+ - Implemented lazy wire decoding (`LazyGeneralProduct` and
76
+ `LazyGeneralVariant`) with precomputed fixed-width bit sizes and
77
+ static field offsets on pattern graphs.
78
+ - Optimized bitstream reading/writing (`BitReader` / `BitWriter`) with
79
+ word-level operations, pre-encoded closed pattern bytes, and direct
80
+ bitstream-to-codepoint UTF-8 string decoding without intermediate
81
+ AST allocations.
82
+
83
+ - **Stack-Safe & Bounded Value Rendering**:
84
+ - Converted `valueToK`, `valueWithEnvelopeToK`, and wire serialization
85
+ to iterative stack-safe algorithms with early character truncation
86
+ (`maxChars`).
87
+ - Added `:limit [num|none]` command and `--limit=<num>` CLI flag in
88
+ `repl.mjs` (default: `4096` chars) and bounded DOM tree previews in
89
+ `repl.html`.
90
+ - Handled broken pipe (`EPIPE`) cleanly across codec CLI entrypoints.
91
+
8
92
  ## [7.1.0] — 2026-10-16 — Polymorphic K IR (PKIR) & CLI Standardization
9
93
 
10
94
  ### Features & Architecture Improvements
@@ -407,6 +491,21 @@ This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
407
491
 
408
492
  ---
409
493
 
494
+ [8.0.0]: https://github.com/fraczak/k/releases/tag/v8.0.0
495
+ [7.2.0]: https://github.com/fraczak/k/releases/tag/v7.2.0
496
+ [7.1.0]: https://github.com/fraczak/k/releases/tag/v7.1.0
497
+ [7.0.2]: https://github.com/fraczak/k/releases/tag/v7.0.2
498
+ [7.0.1]: https://github.com/fraczak/k/releases/tag/v7.0.1
499
+ [7.0.0]: https://github.com/fraczak/k/releases/tag/v7.0.0
500
+ [6.10.6]: https://github.com/fraczak/k/releases/tag/v6.10.6
501
+ [6.10.5]: https://github.com/fraczak/k/releases/tag/v6.10.5
502
+ [6.10.4]: https://github.com/fraczak/k/releases/tag/v6.10.4
503
+ [6.10.3]: https://github.com/fraczak/k/releases/tag/v6.10.3
504
+ [6.10.2]: https://github.com/fraczak/k/releases/tag/v6.10.2
505
+ [6.10.1]: https://github.com/fraczak/k/releases/tag/v6.10.1
506
+ [6.10.0]: https://github.com/fraczak/k/releases/tag/v6.10.0
507
+ [6.9.0]: https://github.com/fraczak/k/releases/tag/v6.9.0
508
+ [6.5.0]: https://github.com/fraczak/k/releases/tag/v6.5.0
410
509
  [6.2.3]: https://github.com/fraczak/k/releases/tag/v6.2.3
411
510
  [6.1.0]: https://github.com/fraczak/k/releases/tag/v6.1.0
412
511
  [6.0.0]: https://github.com/fraczak/k/releases/tag/v6.0.0
@@ -69,21 +69,24 @@ states, so any two structurally equivalent schemas yield identical canonical tex
69
69
 
70
70
  ### Filters
71
71
 
72
- A _filter_ is a syntactic expression `?schema` that acts as a partial identity
73
- relation on values conforming to `schema`. If the input value matches the
74
- schema, it passes through unchanged; otherwise, the relation is undefined.
72
+ A _filter_ is a syntactic expression `?schema` that acts as a partial
73
+ identity relation on values conforming to `schema`. If the input value
74
+ matches the schema, it passes through unchanged; otherwise, the
75
+ relation is undefined.
75
76
 
76
- In recursive relations, filters also serve as **inductive termination witnesses**,
77
- ensuring type and contract derivation converges.
77
+ In recursive relations, filters also serve as **inductive termination
78
+ witnesses**, ensuring type and contract derivation converges.
78
79
 
79
80
  In general, a filter schema is defined by:
80
81
 
81
82
  ```bnf
82
83
  filter ::= name | '{' filter_label_list '}'
83
84
  | '<' filter_label_list '>' | '(' filter_label_list ')'
85
+ | name '[' filter_args ']'
84
86
  | filter '=' name
85
87
  filter_label_list ::= /* empty */ | '...'
86
88
  | ( filter label ',' ) * filter label
89
+ filter_args ::= /* empty */ | ( filter ',' ) * filter
87
90
  ```
88
91
 
89
92
  Examples:
@@ -92,10 +95,54 @@ Examples:
92
95
  X -- unconstrained schema variable
93
96
  ( ... ) -- unconstrained schema
94
97
  { ... } -- any product schema
95
- ( X x, X y ) -- any product schema with two fields of the same schema X
96
- < X x, ... > = X -- a recursive variant schema X with variant x of schema X
98
+ { X x, X y } -- product with two fields of the same schema X
99
+ < X x, ... > = X -- recursive variant X with variant x of schema X
100
+ list[int[]] -- instantiated named pattern
97
101
  ```
98
102
 
103
+ ### Named Patterns and Parameterized Schemas
104
+
105
+ k supports reusable, parameterized regular tree grammar templates via
106
+ square-bracket syntax `name[...]`:
107
+
108
+ ```k
109
+ name[ P1, P2, ... Pn ] = filter;
110
+ ```
111
+
112
+ For 0-arity templates, empty brackets `[]` are used:
113
+
114
+ ```k
115
+ bit[] = < {} 0, {} 1 >;
116
+ bits[] = < {} _, B 0, B 1 > = B;
117
+ byte[] = {
118
+ bit[] 0, bit[] 1, bit[] 2, bit[] 3,
119
+ bit[] 4, bit[] 5, bit[] 6, bit[] 7
120
+ };
121
+ ```
122
+
123
+ Parameterized templates introduce external ports:
124
+
125
+ ```k
126
+ list[X] = < {} nil, { X car, Y cdr } cons > = Y;
127
+ pair[A, B] = { A first, B second };
128
+ ```
129
+
130
+ Callers instantiate templates using `? name[...]`:
131
+
132
+ ```k
133
+ list? = ? list[bits[]];
134
+ ```
135
+
136
+ Or directly inside relation parameter guards:
137
+
138
+ ```k
139
+ add = ?{ nat[] = N x, ... } < ... >;
140
+ ```
141
+
142
+ Template expansion is purely structural and static: internal knots
143
+ (`= Y`) are scoped locally and alpha-renamed, and template calls form an
144
+ acyclic dependency DAG, preventing non-regular divergence.
145
+
99
146
  ### Partial functions
100
147
 
101
148
  A _partial function_ is a function that is not defined for every value
package/DOCS/CODECS.md CHANGED
@@ -141,6 +141,50 @@ printf 'yes\n' | ./codecs/yesno.mjs --parse | ./codecs/yesno.mjs --print
141
141
  # Output: yes
142
142
  ```
143
143
 
144
+ ## Built-in Codecs
145
+
146
+ The `codecs/` directory provides the following standard codecs:
147
+
148
+ - **`int.mjs` (`k-int`)**: Decimal integers and bracketed integer lists
149
+ (e.g. `42`, `-15`, `[1, 2, 3]`) mapped to signed binary bit trees and
150
+ `cons`/`nil` lists.
151
+ - **`ieee.mjs` (`k-ieee`)**: IEEE 754 64-bit floating-point numbers
152
+ (`sign`, `exponent`, `fraction`).
153
+ - **`utf8.mjs` (`k-utf8`) & `utf16.mjs` (`k-utf16`)**: Unicode strings
154
+ encoded as `cons`/`nil` code-point lists (`str` in `core.k`).
155
+ - **`unit.mjs` (`k-unit`)**: The unit value `{}`.
156
+ - **`json.mjs` (`k-json`) — Schema-Specializing JSON**:
157
+ Derives a document-specific pattern for each JSON value:
158
+ - JSON objects map directly to products `{ field: val, ... }`.
159
+ - JSON arrays map to indexed products `{ "0": a0, "1": a1, ... }`.
160
+ - Ideal for record-oriented projection (e.g. `.user .name`).
161
+ - **`djson.mjs` (`k-djson`) — Universal Dynamic JSON (Cons-List)**:
162
+ Maps every JSON document to a single universal recursive named pattern
163
+ `djson[]`:
164
+ ```k
165
+ djson[] = <
166
+ float64 number,
167
+ str string,
168
+ < {} true, {} false > bool,
169
+ {} null,
170
+ < {} nil, { J car, L cdr } cons > = L list,
171
+ < {} nil, { { str key, J val } car, O cdr } cons > = O object
172
+ > = J;
173
+ ```
174
+ Because every JSON value shares the exact same static 29-node pattern,
175
+ a `k` relation can be compiled once against `djson[]` and process
176
+ arbitrary JSON documents of any shape or array/object length.
177
+ - **`djson-btrie.mjs` (`k-djson-btrie`) — Binary Bit-Trie Dynamic JSON**:
178
+ Replaces the linear `list` and `object` cons-lists with a unified
179
+ 27-node binary bit-trie (`TRIE`). Array indices (`bits?`) and string
180
+ keys are stored along MSB-to-LSB bit paths, enabling $O(|K|)$ lookup
181
+ (`dot` and `nth` in `Examples/djson-btrie.k`) via direct `.0`/`.1`
182
+ projections with zero character-equality checks.
183
+ - **`djson-ptrie.mjs` (`k-djson-ptrie`) — Patricia/Radix Dynamic JSON**:
184
+ Models `list` and `object` as path-compressed radix tries (`trie-array`
185
+ style) with `prefix` and $O(1)$ subtree `size` (`list_size` and
186
+ `obj_size` in `Examples/djson-ptrie.k`).
187
+
144
188
  ## Testing a Codec
145
189
 
146
190
  For CLI codecs, test both directions in a pipeline:
@@ -174,8 +218,9 @@ npm test
174
218
  ## Web REPL (VFS) Bundling
175
219
 
176
220
  When `repl.html` is generated (`npm run build:repl-html`):
177
- - Codecs (`int.mjs`, `json.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`) are
178
- bundled into 100% self-contained ES modules with zero imports.
221
+ - Codecs (`int.mjs`, `json.mjs`, `djson.mjs`, `djson-btrie.mjs`,
222
+ `djson-ptrie.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`) are bundled into
223
+ 100% self-contained ES modules with zero imports.
179
224
  - `runCodecCLI` is shimmed as a no-op, and `Value` helpers are directly
180
225
  inlined.
181
226
  - In the browser, the VFS serves these self-contained modules directly to
@@ -192,11 +192,16 @@ runCodecCLI(import.meta.url, { parse, print, doc });
192
192
  pathToFileURL(process.argv[1]).href`). If imported as a module or
193
193
  running in a browser, it is a no-op.
194
194
  2. **Flag Parsing**: Handles `--parse`, `--print`, and `-h` / `--help`.
195
- 3. **Stream Piping**:
195
+ 3. **Stream Piping & Lazy Wire Decoding**:
196
196
  - For `--parse`: reads `stdin` to string, calls `parse(text)`, encodes
197
197
  enveloped value `(P, v)` to binary wire format on `stdout`.
198
- - For `--print`: decodes wire format from `stdin` into `(P, v)`, calls
199
- `print(value)`, writes formatted text to `stdout`.
198
+ - For `--print`: lazily decodes the wire format from `stdin` into
199
+ bitstream-backed `Value` proxies (`LazyGeneralProduct`,
200
+ `LazyGeneralVariant`) with precomputed fixed-width bit sizes and
201
+ static field offsets, calls `print(value)`, and writes formatted text
202
+ to `stdout`.
203
+ - Cleanly exits on `EPIPE` when piped into downstream consumers like
204
+ `head`.
200
205
 
201
206
  ---
202
207
 
@@ -207,7 +212,10 @@ To make `codecs/` intuitive and self-contained:
207
212
  ```text
208
213
  codecs/
209
214
  ├── int.mjs # Integer & list codec
210
- ├── json.mjs # JSON codec
215
+ ├── json.mjs # JSON codec (schema-specializing)
216
+ ├── djson.mjs # Dynamic JSON codec (cons-list)
217
+ ├── djson-btrie.mjs # Dynamic JSON codec (binary bit-trie)
218
+ ├── djson-ptrie.mjs # Dynamic JSON codec (Patricia/radix trie)
211
219
  ├── utf8.mjs # UTF-8 string codec
212
220
  ├── utf16.mjs # UTF-16 string codec
213
221
  ├── ieee.mjs # IEEE 754 float codec
@@ -236,7 +244,8 @@ In the browser REPL (`repl.html`):
236
244
  imports**.
237
245
  - **No Runtime Files in VFS**: `codecs/runtime/*` is not included in the
238
246
  browser VFS. The VFS only contains verified, self-contained user-facing
239
- files (`int.mjs`, `json.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`).
247
+ files (`int.mjs`, `json.mjs`, `djson.mjs`, `djson-btrie.mjs`,
248
+ `djson-ptrie.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`).
240
249
  - **Native Browser Loading**: In `repl-codecs.mjs`, the browser reads the
241
250
  codec source from VFS, wraps it in a Blob, and invokes native `await
242
251
  import(blobUrl)`. No regex manipulation, import stripping, or global
@@ -67,11 +67,20 @@ schema registry; its `@hash` is a fixed-width indexing digest.
67
67
 
68
68
  ### Filter
69
69
 
70
- A `filter` is an expression `?schema` that constrains the input payload to a
71
- schema and otherwise behaves like identity.
70
+ A `filter` is an expression `?schema` that constrains the input payload
71
+ to a schema and otherwise behaves like identity.
72
72
 
73
- In recursive relations, filters also serve as **inductive termination witnesses**,
74
- ensuring contract derivation converges.
73
+ In recursive relations, filters also serve as **inductive termination
74
+ witnesses**, ensuring contract derivation converges.
75
+
76
+ ### Named Pattern
77
+
78
+ A `named pattern` is a parameterized regular tree grammar template
79
+ defined using square-bracket syntax (`name[P1, ..., Pn] = filter;`).
80
+ Named patterns declare external input ports in their parameter list
81
+ and may contain internal unification variables and cycle knots
82
+ (`= KnotName`). They are expanded statically without recursive
83
+ divergence, compiling into canonical schema automata nodes.
75
84
 
76
85
  ### Canonical Content-Addressed Hash (`@hash`)
77
86
 
@@ -97,8 +97,9 @@ flowchart TD
97
97
  ### What It Is
98
98
  `.k` files contain the high-level, human-readable source code of the
99
99
  `k` language. A `.k` file defines:
100
- - **Algebraic Data Types:** Products (`{ field1 label1, field2 label2 }`)
101
- and tagged unions (`< tag1 arm1, tag2 arm2 >`).
100
+ - **Algebraic Schemas & Named Patterns:** Products (`{ ... }`),
101
+ tagged unions (`< ... >`), and reusable parameterized pattern
102
+ templates (`name[...] = filter;`).
102
103
  - **Relations:** First-order partial functions composed via sequence,
103
104
  union, projection, or product construction.
104
105
  - **Top-level Expressions:** Computations evaluated when the file is run
@@ -0,0 +1,306 @@
1
+ # Comparison of JSON Approaches in k
2
+
3
+ This document compares the four JSON codec and schema implementations in
4
+ the **k** repository:
5
+
6
+ 1. **Schema-Specializing JSON** (`json`):
7
+ [`codecs/json.mjs`][json-mjs]
8
+ 2. **Dynamic Cons-List JSON** (`djson`):
9
+ [`codecs/djson.mjs`][djson-mjs] & [`Examples/djson.k`][djson-k]
10
+ 3. **Dynamic Binary Bit-Trie JSON** (`djson-btrie`):
11
+ [`codecs/djson-btrie.mjs`][djson-btrie-mjs] &
12
+ [`Examples/djson-btrie.k`][djson-btrie-k]
13
+ 4. **Dynamic Patricia/Radix Trie JSON** (`djson-ptrie`):
14
+ [`codecs/djson-ptrie.mjs`][djson-ptrie-mjs] &
15
+ [`Examples/djson-ptrie.k`][djson-ptrie-k]
16
+
17
+ [json-mjs]: file:///home/wojtek/gits/K/k/codecs/json.mjs
18
+ [djson-mjs]: file:///home/wojtek/gits/K/k/codecs/djson.mjs
19
+ [djson-k]: file:///home/wojtek/gits/K/k/Examples/djson.k
20
+ [djson-btrie-mjs]: file:///home/wojtek/gits/K/k/codecs/djson-btrie.mjs
21
+ [djson-btrie-k]: file:///home/wojtek/gits/K/k/Examples/djson-btrie.k
22
+ [djson-ptrie-mjs]: file:///home/wojtek/gits/K/k/codecs/djson-ptrie.mjs
23
+ [djson-ptrie-k]: file:///home/wojtek/gits/K/k/Examples/djson-ptrie.k
24
+
25
+ ---
26
+
27
+ ## 1. Architectural Motivation
28
+
29
+ In **k**, all values are algebraic data types classified by closed type
30
+ patterns. External formats like JSON can be embedded into **k** via two
31
+ fundamentally distinct strategies:
32
+
33
+ ### Static Schema Specialization (`json`)
34
+ - Infers a concrete closed pattern specialized to the exact shape and
35
+ field names of each input JSON document.
36
+ - Objects become concrete labeled products `{ "name": Str, "age": Num }`.
37
+ - Lists become homogeneous or heterogeneous tuples/cons-lists.
38
+ - **Advantage**: Zero interpretive overhead for known static schemas;
39
+ direct field projection in compiled code.
40
+ - **Limitation**: The pattern is dynamic and grows unboundedly with the
41
+ document. Cannot write universal JSON transformation relations (like a
42
+ generic JSON query engine or REPL printer) under a single fixed closed
43
+ pattern.
44
+
45
+ ### Dynamic Uniform JSON (`djson*` family)
46
+ - Defines a single, fixed closed pattern `json?` once for all valid JSON
47
+ values. Any arbitrary JSON document parses into an injection of this
48
+ universal pattern.
49
+ - Allows authoring universal helper relations in `k`:
50
+ - `dot: {key, object} -> val`
51
+ - `has_key?: {key, object} -> ()`
52
+ - `nth: {index, list} -> val`
53
+ - `obj_size: object -> bits?`
54
+ - `list_size: list -> bits?`
55
+ - `remove: {key, object} -> object`
56
+ - `put: {key, val, object} -> object`
57
+ - Compiles once to Wasm or LLVM `-O3` binaries that execute over any
58
+ input JSON document without recompilation.
59
+
60
+ ---
61
+
62
+ ## 2. Structural Schema Comparison
63
+
64
+ | Dimension | `djson` (Cons) | `djson-btrie` | `djson-ptrie` |
65
+ | :--- | :--- | :--- | :--- |
66
+ | **Nodes** | 29 | 27 | 33 |
67
+ | **Numbers** | float64 | float64 | float64 |
68
+ | **Strings** | Cons-list | Cons-list | Cons-list |
69
+ | **Lists** | Cons-list | Bit-trie | Patricia trie |
70
+ | **Objects** | Sorted cons | Bit-trie | Radix trie |
71
+ | **Node** | `{car, cdr}` | `{val, 0, 1}` | `{pfx, sz, val, d}` |
72
+ | **Indexing**| $O(N)$ linear | $O(\log N)$ | $O(\log N)$ |
73
+ | **Lookup** | $O(K \cdot L)$ | $O(L \cdot 8)$ | $O(L)$ radix |
74
+ | **Size** | $O(N)$ walk | $O(N)$ walk | **$O(1)$** field |
75
+
76
+ ### Schema Definitions
77
+
78
+ #### Dynamic Cons-List (`djson`)
79
+ ```k
80
+ LIST = < {} nil, { JSON car, LIST cdr } cons >;
81
+ ENTRY = { Str key, JSON val };
82
+ OBJECT = < {} nil, { ENTRY car, OBJECT cdr } cons >;
83
+ ```
84
+
85
+ #### Dynamic Binary Bit-Trie (`djson-btrie`)
86
+ ```k
87
+ TRIE = <
88
+ {} nil,
89
+ { < {} nil, JSON val > val, TRIE 0, TRIE 1 } trie
90
+ >;
91
+ LIST = TRIE;
92
+ OBJECT = TRIE;
93
+ ```
94
+
95
+ #### Dynamic Patricia/Radix Trie (`djson-ptrie`)
96
+ ```k
97
+ LIST = <
98
+ {} nil,
99
+ {
100
+ BITS prefix,
101
+ BITS size,
102
+ < {} nil, JSON val > val,
103
+ LIST 0,
104
+ LIST 1
105
+ } trie
106
+ >;
107
+ DICT = < {} nil, { { CHAR char, OBJECT trie } car, DICT cdr } cons >;
108
+ OBJECT = <
109
+ {} nil,
110
+ { Str prefix, BITS size, < {} nil, JSON val > val, DICT dict } trie
111
+ >;
112
+ ```
113
+
114
+ ---
115
+
116
+ ## 3. Algorithmic Mechanics
117
+
118
+ ### 3.1 Object Key Lookup (`dot`)
119
+ - **`djson`**: Iterates through a linked list of `{key, val}` pairs.
120
+ On each entry, performs character-by-character string equality
121
+ `eq_string?`. For $N$ entries with key length $L$, worst-case time is
122
+ $O(N \cdot L)$.
123
+ - **`djson-btrie`**: Uses `step_char` to walk 8 to 24 bits directly down
124
+ the binary trie per character. Because character bits are decoded
125
+ destructively into branch selectors without allocating `bits?` nodes,
126
+ it executes with **zero heap allocations** in compiled LLVM code.
127
+ - **`djson-ptrie`**: Matches the query key against the node's `prefix`
128
+ run using `match_str_prefix`. If the prefix matches, it looks up the
129
+ next branching character in a small local `dict` cons-list.
130
+ Non-branching paths of arbitrary length are traversed in one step.
131
+
132
+ ### 3.2 Key Membership (`has_key?`)
133
+ - **Hit**: All three navigate to the target key.
134
+ - **Miss**:
135
+ - `djson` scans all $N$ entries to the end of the cons-list.
136
+ - `djson-btrie` and `djson-ptrie` terminate immediately as soon as a bit
137
+ or character branch is missing, achieving sub-microsecond rejection
138
+ even on large documents.
139
+
140
+ ### 3.3 Array Indexing (`nth`)
141
+ - **`djson`**: Decodes a `bits?` natural number with `dec_nat` and walks
142
+ cons cells one by one ($O(I)$).
143
+ - **`djson-btrie`**: Dispatches directly on the bits of the index from
144
+ most-significant to least-significant bit ($O(\log I)$).
145
+ - **`djson-ptrie`**: Compares the remaining index bits against the
146
+ node's `prefix: BITS` in chunks, descending along compressed paths
147
+ ($O(\log I)$).
148
+
149
+ ### 3.4 Subtree Cardinality (`obj_size`, `list_size`)
150
+ - **`djson` & `djson-btrie`**: Must recursively traverse the entire cons
151
+ list or bit-trie, adding 1 for each present entry using binary addition
152
+ (`inc_nat`). Complexity is $O(N \log N)$.
153
+ - **`djson-ptrie`**: Stores a precomputed binary natural number
154
+ `size: BITS` at every node. Size queries are simple $O(1)$ field
155
+ projections (`ptrie_obj? < /nil 0, /trie .size >`). In LLVM `-O3`, this
156
+ executes in **5 nanoseconds**, independent of document size.
157
+
158
+ ### 3.5 Functional Updates (`remove`, `put`)
159
+ - **`djson`**: Copies the cons spine until the modified entry ($O(N)$).
160
+ - **`djson-btrie`**: Rebuilds the bit path down to the leaf. `put` uses
161
+ `str_to_bits` to construct the bit path, while `remove` clears `.val` to
162
+ `nil` along the path.
163
+ - **`djson-ptrie`**:
164
+ - `remove` checks `has_key?`, decrements `size` with `dec_nat` along
165
+ the path, and clears `.val` to `nil`.
166
+ - `put` uses `split_str_prefix` to find the common prefix between the
167
+ new key and existing node. If prefixes diverge, it splits the node
168
+ into a shared parent with a 2-entry branch dictionary in $O(L)$ time.
169
+
170
+ ---
171
+
172
+ ## 4. Empirical Benchmark Results
173
+
174
+ Measured with `npm run perf:djson` on Linux x86_64 across sizes
175
+ $N \in \{10, 100, 500\}$.
176
+
177
+ ### 4.1 Wire Representation Sizes
178
+
179
+ #### Shared-Prefix Objects (`{"field_prefix_0000": 0, ...}`)
180
+ - **$N = 10$** (Raw: 230 B):
181
+ - `djson`: 1,029 B (4.47x)
182
+ - `djson-btrie`: **913 B (3.97x)**
183
+ - `djson-ptrie`: 1,032 B (4.49x)
184
+ - `json` (Spec): 370 B (1.61x)
185
+ - **$N = 100$** (Raw: 2,750 B):
186
+ - `djson`: 3,365 B (1.22x)
187
+ - `djson-btrie`: **2,096 B (0.76x)**
188
+ - `djson-ptrie`: 2,367 B (0.86x)
189
+ - `json` (Spec): 1,529 B (0.56x)
190
+ - **$N = 500$** (Raw: 14,750 B):
191
+ - `djson`: 14,357 B (0.97x)
192
+ - `djson-btrie`: **7,370 B (0.50x)**
193
+ - `djson-ptrie`: 8,241 B (0.56x)
194
+ - `json` (Spec): 7,229 B (0.49x)
195
+
196
+ #### Varied-Key Objects (`{"account_0_account_attr": 0, ...}`)
197
+ - **$N = 10$** (Raw: 270 B):
198
+ - `djson`: **1,114 B (4.13x)**
199
+ - `djson-btrie`: 1,460 B (5.41x)
200
+ - `djson-ptrie`: 1,236 B (4.58x)
201
+ - `json` (Spec): 496 B (1.84x)
202
+ - **$N = 100$** (Raw: 3,120 B):
203
+ - `djson`: 4,987 B (1.60x)
204
+ - `djson-btrie`: 6,729 B (2.16x)
205
+ - `djson-ptrie`: **4,309 B (1.38x)**
206
+ - `json` (Spec): 4,460 B (1.43x)
207
+ - **$N = 500$** (Raw: 16,598 B):
208
+ - `djson`: 22,581 B (1.36x)
209
+ - `djson-btrie`: 28,478 B (1.72x)
210
+ - `djson-ptrie`: **17,036 B (1.03x)**
211
+ - `json` (Spec): 22,104 B (1.33x)
212
+
213
+ #### Homogeneous Number Arrays (`[0, 3, 6, ...]`)
214
+ - **$N = 10$** (Raw: 27 B):
215
+ - `djson`: 880 B (32.6x)
216
+ - `djson-btrie`: **878 B (32.5x)**
217
+ - `djson-ptrie`: 999 B (37.0x)
218
+ - **$N = 100$** (Raw: 363 B):
219
+ - `djson`: **1,645 B (4.53x)**
220
+ - `djson-btrie`: 1,666 B (4.59x)
221
+ - `djson-ptrie`: 1,877 B (5.17x)
222
+ - **$N = 500$** (Raw: 2,129 B):
223
+ - `djson`: **5,045 B (2.37x)**
224
+ - `djson-btrie`: 5,166 B (2.43x)
225
+ - `djson-ptrie`: 5,776 B (2.71x)
226
+
227
+ ---
228
+
229
+ ### 4.2 Execution Latency Across Backends ($N=500$)
230
+
231
+ #### Object Lookup (`dot` on last key of 500-key object)
232
+ - **Shared-Prefix Keys**:
233
+ - `djson`: Wasm $580.4\ \mu\text{s}$, LLVM $23.6\ \mu\text{s}$
234
+ - `djson-btrie`: Wasm $7.4\ \mu\text{s}$, LLVM **$2.4\ \mu\text{s}$**
235
+ - `djson-ptrie`: Wasm $11.1\ \mu\text{s}$, LLVM $5.1\ \mu\text{s}$
236
+ - **Varied Keys**:
237
+ - `djson`: Wasm $26.8\ \mu\text{s}$, LLVM $4.2\ \mu\text{s}$
238
+ - `djson-btrie`: Wasm $10.9\ \mu\text{s}$, LLVM **$3.1\ \mu\text{s}$**
239
+ - `djson-ptrie`: Wasm $10.8\ \mu\text{s}$, LLVM $4.4\ \mu\text{s}$
240
+
241
+ #### Key Membership Filter (`has_key?` miss on nonexistent key)
242
+ - `djson`: Wasm $63.9\ \mu\text{s}$, LLVM $1,168\ \text{ns}$
243
+ - `djson-btrie`: Wasm $1,022\ \text{ns}$, LLVM **$466\ \text{ns}$**
244
+ - `djson-ptrie`: Wasm $1,908\ \text{ns}$, LLVM $1,228\ \text{ns}$
245
+
246
+ #### Array Element Lookup (`nth` on index 499 of 500 elements)
247
+ - `djson`:
248
+ kVM $10.74\ \text{ms}$, Wasm $97.2\ \mu\text{s}$,
249
+ LLVM $100.7\ \mu\text{s}$
250
+ - `djson-btrie`:
251
+ kVM $26.0\ \mu\text{s}$, Wasm $733\ \text{ns}$,
252
+ LLVM **$114\ \text{ns}$** (**881x speedup**)
253
+ - `djson-ptrie`:
254
+ kVM $35.7\ \mu\text{s}$, Wasm $1,267\ \text{ns}$,
255
+ LLVM $371\ \text{ns}$ (**271x speedup**)
256
+
257
+ #### Cardinality Queries (`obj_size` for $N=500$)
258
+ - `djson`:
259
+ kVM $13.57\ \text{ms}$, Wasm $89.2\ \mu\text{s}$,
260
+ LLVM $127.4\ \mu\text{s}$
261
+ - `djson-btrie`:
262
+ kVM $210.88\ \text{ms}$, Wasm $3.02\ \text{ms}$,
263
+ LLVM $8.06\ \text{ms}$
264
+ - `djson-ptrie`:
265
+ kVM $2,054\ \text{ns}$, Wasm $172\ \text{ns}$,
266
+ LLVM **$5\ \text{ns}$** (**26,106x speedup**)
267
+
268
+ #### Object Key Updates ($N=500$)
269
+ - **`remove`**:
270
+ - `djson`: LLVM $79.6\ \mu\text{s}$
271
+ - `djson-btrie`: LLVM **$9.5\ \mu\text{s}$** (8.4x speedup)
272
+ - `djson-ptrie`: LLVM $13.3\ \mu\text{s}$ (6.0x speedup)
273
+ - **`put`**:
274
+ - `djson`: LLVM $50.8\ \mu\text{s}$
275
+ - `djson-btrie`: LLVM $12.6\ \mu\text{s}$ (4.0x speedup)
276
+ - `djson-ptrie`: LLVM **$3.9\ \mu\text{s}$** (13.1x speedup)
277
+
278
+ ---
279
+
280
+ ## 5. Trade-off Analysis & Recommendation Matrix
281
+
282
+ | Feature | `json` | `djson` | `btrie` | `ptrie` |
283
+ | :--- | :---: | :---: | :---: | :---: |
284
+ | **Universal Pattern** | ❌ No | ✅ 29 | ✅ 27 | ✅ 33 |
285
+ | **REPL Inspection** | Native | Cons | Bit-tree | Radix |
286
+ | **Shared Prefix Wire**| Small | Med | **Small** | Small |
287
+ | **Varied Keys Wire** | Large | Med | Large | **Small** |
288
+ | **Array ($N=500$)** | Direct | $100\ \mu\text{s}$ | **114ns**| 371ns |
289
+ | **Dot ($N=500$)** | Direct | $23.6\ \mu\text{s}$| **2.4µs**| 5.1µs |
290
+ | **Put ($N=500$)** | N/A | $50.8\ \mu\text{s}$| 12.6µs | **3.9µs**|
291
+ | **Size ($N=500$)** | N/A | $127\ \mu\text{s}$ | 8.1ms | **5ns** |
292
+
293
+ ### Recommendations
294
+ 1. **Choose `djson-ptrie` for general-purpose dynamic JSON**:
295
+ It achieves the best wire compression on real-world varied keys
296
+ (1.03x of raw JSON), provides instant $O(1)$ cardinality queries
297
+ (5 ns in LLVM), and delivers the fastest updates (`put` in 3.9 µs).
298
+ 2. **Choose `djson-btrie` for extreme lookup throughput on shared keys**:
299
+ Destructive bit-walking in `dot` avoids all node allocation,
300
+ achieving the lowest lookup latency (2.4 µs in LLVM).
301
+ 3. **Keep `djson` for simple, lightweight tooling**:
302
+ Simple cons-list structure is human-readable in REPL raw outputs and
303
+ has minimal code complexity for small payloads ($N < 50$).
304
+ 4. **Choose `json` when document schema is fixed at compile time**:
305
+ Avoids all dynamic dictionary lookups in exchange for a
306
+ document-specialized pattern.