@fraczak/k 7.2.0 → 8.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +90 -0
- package/DOCS/BIG_PICTURE.md +54 -7
- package/DOCS/CODECS.md +21 -13
- package/DOCS/CODEC_ARCHITECTURE.md +1 -1
- package/DOCS/DICTIONARY.md +13 -4
- package/DOCS/FILE_FORMATS.md +3 -2
- package/DOCS/JSON_APPROACHES_COMPARISON.md +302 -0
- package/DOCS/NAMED_PATTERNS.md +317 -0
- package/DOCS/PATTERNS.md +59 -11
- package/DOCS/REPL.md +13 -2
- package/DOCS/WEB_REPL_REQUIREMENTS.md +3 -2
- package/Examples/arithmetics.k +22 -17
- package/Examples/bnat.k +21 -11
- package/Examples/buda.k +37 -20
- package/Examples/byte.k +3 -1
- package/Examples/code.k +38 -39
- package/Examples/djson.k +789 -0
- package/Examples/esc.k +1 -1
- package/Examples/karatsuba-mult.k +16 -13
- package/Examples/list.k +8 -4
- package/Examples/nat.k +17 -11
- package/Examples/poly.k +11 -3
- package/Examples/vhdl_to_k.k +43 -31
- package/README.md +40 -16
- package/browser/repl-browser.mjs +2 -0
- package/browser/tree-view.mjs +35 -16
- package/browser/vfs-data.mjs +10 -8
- package/codecs/README.md +1 -1
- package/codecs/djson.mjs +375 -84
- package/core.k +79 -115
- package/index.mjs +3 -0
- package/named-patterns.mjs +267 -0
- package/object.mjs +31 -3
- package/package.json +1 -1
- package/parser.jison +68 -5
- package/parser.mjs +111 -60
- package/pretty.mjs +5 -0
- package/repl.mjs +188 -22
- package/scripts/build-repl-html.mjs +58 -4
- package/scripts/run-tests.mjs +10 -0
- package/symbol-table.mjs +20 -2
- package/tests/test-codecs.mjs +30 -17
- package/tests/test-djson-methods.mjs +285 -0
- package/tests/test-named-patterns.mjs +189 -0
- package/tests/test-repl-html.mjs +47 -4
- package/tests/test-repl.mjs +149 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,81 @@ All notable changes to this project will be documented here.
|
|
|
5
5
|
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
|
6
6
|
This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [8.1.0] — 2026-10-20 — Dynamic JSON Bit-Trie & json? Standard Library
|
|
9
|
+
|
|
10
|
+
### Dynamic JSON (djson) Consolidation
|
|
11
|
+
|
|
12
|
+
- **Unified Binary Bit-Trie Representation**:
|
|
13
|
+
- Consolidated all dynamic JSON approaches onto the binary bit-trie
|
|
14
|
+
implementation (`btrie[V]`), eliminating linear cons-lists and
|
|
15
|
+
heap allocations in dynamic document navigation.
|
|
16
|
+
- Standardized `Examples/djson.k` as the single dynamic JSON
|
|
17
|
+
specification and `codecs/djson.mjs` (`k-djson`) as its codec.
|
|
18
|
+
- Retired obsolete cons-list and Patricia trie dynamic JSON
|
|
19
|
+
variants (`djson-btrie.k`, `djson-ptrie.k`, `djson-btrie.mjs`,
|
|
20
|
+
`djson-ptrie.mjs`, `k-djson-btrie`, `k-djson-ptrie`).
|
|
21
|
+
- **Complete Standard Library on `json?`**:
|
|
22
|
+
- Implemented standard dynamic JSON methods in `Examples/djson.k`
|
|
23
|
+
operating directly on the universal `json?` type:
|
|
24
|
+
* Object operations: `get`, `has?`, `set`, `del`
|
|
25
|
+
* List operations: `nth`, `has_index?`, `set_nth`, `del_nth`,
|
|
26
|
+
`push`
|
|
27
|
+
* Enumerations: `keys`, `keys_raw`, `values`, `values_raw`
|
|
28
|
+
* Cardinality: `length` (JSON float64 number), `length_nat`
|
|
29
|
+
(`bits?`)
|
|
30
|
+
* Type inspection: `type_of` and `is_*?` predicates
|
|
31
|
+
|
|
32
|
+
## [8.0.0] — 2026-10-20 — Named Patterns, Type Schemes & Trie Codecs
|
|
33
|
+
|
|
34
|
+
### Language & Type System
|
|
35
|
+
|
|
36
|
+
- **Named Patterns & Parameterized Schemes (`named-patterns.mjs`)**:
|
|
37
|
+
- Introduced first-class syntax for closed named patterns:
|
|
38
|
+
`byte[] = { bit 0, bit 1, ... };` and `bool[] = < bit false, ... >;`.
|
|
39
|
+
- Added parameterized type schemes with explicit type variables:
|
|
40
|
+
`list[X] = < X head, (list[X]) tail >;` and
|
|
41
|
+
`pair[A, B] = { A first, B second };`.
|
|
42
|
+
- Implemented monomorphic expansion with cycle detection and sharing.
|
|
43
|
+
- Full object file support: `.ko` and `.klib` payloads serialize and
|
|
44
|
+
deserialize named pattern tables alongside relations.
|
|
45
|
+
|
|
46
|
+
- **Standard Library & Examples Refactoring**:
|
|
47
|
+
- Adopted named patterns throughout `core.k` (`bit[]`, `byte[]`,
|
|
48
|
+
`pair[A, B]`, `either[A, B]`, `option[X]`).
|
|
49
|
+
- Refactored `Examples/*.k` (`djson.k`, `arithmetics.k`, `bnat.k`,
|
|
50
|
+
`nat.k`, `code.k`, `karatsuba-mult.k`, `poly.k`, `vhdl_to_k.k`)
|
|
51
|
+
for clarity and conciseness using named patterns.
|
|
52
|
+
|
|
53
|
+
### Dynamic JSON Codecs
|
|
54
|
+
|
|
55
|
+
- **Bit-Trie Codec (`codecs/djson-btrie.mjs`, `k-djson-btrie`)**:
|
|
56
|
+
- Added binary bit-trie dynamic JSON codec with variable-length
|
|
57
|
+
bit prefixing for compact wire encodings.
|
|
58
|
+
- Accompanied by formal K specification in `Examples/djson-btrie.k`.
|
|
59
|
+
|
|
60
|
+
- **Packed Nibble-Trie Codec (`codecs/djson-ptrie.mjs`)**:
|
|
61
|
+
- Added packed 4-bit nibble trie codec (`k-djson-ptrie`) for
|
|
62
|
+
optimized traversal.
|
|
63
|
+
- Accompanied by formal K specification in `Examples/djson-ptrie.k`.
|
|
64
|
+
|
|
65
|
+
- **JSON Approaches Comparison & Benchmarks**:
|
|
66
|
+
- Added comparative architecture and benchmark documentation in
|
|
67
|
+
`DOCS/JSON_APPROACHES_COMPARISON.md`.
|
|
68
|
+
- Added benchmark suite in `scripts/perf-djson.mjs` and
|
|
69
|
+
`tests/bench-djson-tries.mjs`.
|
|
70
|
+
|
|
71
|
+
### Interactive REPL & Tooling
|
|
72
|
+
|
|
73
|
+
- **Pattern Inspection & Persistence in REPL**:
|
|
74
|
+
- Added `:patterns` / `:pat` commands to inspect loaded patterns.
|
|
75
|
+
- Display canonical type code / relation hashes:
|
|
76
|
+
`name[] = ... -- @hash`.
|
|
77
|
+
- Added pattern lookup by `@hash` (`:pattern @hash`, `:pat @hash`).
|
|
78
|
+
- Tab-completion support for pattern names and canonical `@hashes`.
|
|
79
|
+
- Full pattern persistence across REPL sessions via `.klib` files.
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
8
83
|
## [7.2.0] — 2026-10-17 — Dynamic JSON Codec (djson) & Lazy Wire Decoding
|
|
9
84
|
|
|
10
85
|
### Features & Performance Improvements
|
|
@@ -440,6 +515,21 @@ This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
440
515
|
|
|
441
516
|
---
|
|
442
517
|
|
|
518
|
+
[8.0.0]: https://github.com/fraczak/k/releases/tag/v8.0.0
|
|
519
|
+
[7.2.0]: https://github.com/fraczak/k/releases/tag/v7.2.0
|
|
520
|
+
[7.1.0]: https://github.com/fraczak/k/releases/tag/v7.1.0
|
|
521
|
+
[7.0.2]: https://github.com/fraczak/k/releases/tag/v7.0.2
|
|
522
|
+
[7.0.1]: https://github.com/fraczak/k/releases/tag/v7.0.1
|
|
523
|
+
[7.0.0]: https://github.com/fraczak/k/releases/tag/v7.0.0
|
|
524
|
+
[6.10.6]: https://github.com/fraczak/k/releases/tag/v6.10.6
|
|
525
|
+
[6.10.5]: https://github.com/fraczak/k/releases/tag/v6.10.5
|
|
526
|
+
[6.10.4]: https://github.com/fraczak/k/releases/tag/v6.10.4
|
|
527
|
+
[6.10.3]: https://github.com/fraczak/k/releases/tag/v6.10.3
|
|
528
|
+
[6.10.2]: https://github.com/fraczak/k/releases/tag/v6.10.2
|
|
529
|
+
[6.10.1]: https://github.com/fraczak/k/releases/tag/v6.10.1
|
|
530
|
+
[6.10.0]: https://github.com/fraczak/k/releases/tag/v6.10.0
|
|
531
|
+
[6.9.0]: https://github.com/fraczak/k/releases/tag/v6.9.0
|
|
532
|
+
[6.5.0]: https://github.com/fraczak/k/releases/tag/v6.5.0
|
|
443
533
|
[6.2.3]: https://github.com/fraczak/k/releases/tag/v6.2.3
|
|
444
534
|
[6.1.0]: https://github.com/fraczak/k/releases/tag/v6.1.0
|
|
445
535
|
[6.0.0]: https://github.com/fraczak/k/releases/tag/v6.0.0
|
package/DOCS/BIG_PICTURE.md
CHANGED
|
@@ -69,21 +69,24 @@ states, so any two structurally equivalent schemas yield identical canonical tex
|
|
|
69
69
|
|
|
70
70
|
### Filters
|
|
71
71
|
|
|
72
|
-
A _filter_ is a syntactic expression `?schema` that acts as a partial
|
|
73
|
-
relation on values conforming to `schema`. If the input value
|
|
74
|
-
schema, it passes through unchanged; otherwise, the
|
|
72
|
+
A _filter_ is a syntactic expression `?schema` that acts as a partial
|
|
73
|
+
identity relation on values conforming to `schema`. If the input value
|
|
74
|
+
matches the schema, it passes through unchanged; otherwise, the
|
|
75
|
+
relation is undefined.
|
|
75
76
|
|
|
76
|
-
In recursive relations, filters also serve as **inductive termination
|
|
77
|
-
ensuring type and contract derivation converges.
|
|
77
|
+
In recursive relations, filters also serve as **inductive termination
|
|
78
|
+
witnesses**, ensuring type and contract derivation converges.
|
|
78
79
|
|
|
79
80
|
In general, a filter schema is defined by:
|
|
80
81
|
|
|
81
82
|
```bnf
|
|
82
83
|
filter ::= name | '{' filter_label_list '}'
|
|
83
84
|
| '<' filter_label_list '>' | '(' filter_label_list ')'
|
|
85
|
+
| name '[' filter_args ']'
|
|
84
86
|
| filter '=' name
|
|
85
87
|
filter_label_list ::= /* empty */ | '...'
|
|
86
88
|
| ( filter label ',' ) * filter label
|
|
89
|
+
filter_args ::= /* empty */ | ( filter ',' ) * filter
|
|
87
90
|
```
|
|
88
91
|
|
|
89
92
|
Examples:
|
|
@@ -92,10 +95,54 @@ Examples:
|
|
|
92
95
|
X -- unconstrained schema variable
|
|
93
96
|
( ... ) -- unconstrained schema
|
|
94
97
|
{ ... } -- any product schema
|
|
95
|
-
|
|
96
|
-
< X x, ... > = X --
|
|
98
|
+
{ X x, X y } -- product with two fields of the same schema X
|
|
99
|
+
< X x, ... > = X -- recursive variant X with variant x of schema X
|
|
100
|
+
list[int[]] -- instantiated named pattern
|
|
97
101
|
```
|
|
98
102
|
|
|
103
|
+
### Named Patterns and Parameterized Schemas
|
|
104
|
+
|
|
105
|
+
k supports reusable, parameterized regular tree grammar templates via
|
|
106
|
+
square-bracket syntax `name[...]`:
|
|
107
|
+
|
|
108
|
+
```k
|
|
109
|
+
name[ P1, P2, ... Pn ] = filter;
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
For 0-arity templates, empty brackets `[]` are used:
|
|
113
|
+
|
|
114
|
+
```k
|
|
115
|
+
bit[] = < {} 0, {} 1 >;
|
|
116
|
+
bits[] = < {} _, B 0, B 1 > = B;
|
|
117
|
+
byte[] = {
|
|
118
|
+
bit[] 0, bit[] 1, bit[] 2, bit[] 3,
|
|
119
|
+
bit[] 4, bit[] 5, bit[] 6, bit[] 7
|
|
120
|
+
};
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Parameterized templates introduce external ports:
|
|
124
|
+
|
|
125
|
+
```k
|
|
126
|
+
list[X] = < {} nil, { X car, Y cdr } cons > = Y;
|
|
127
|
+
pair[A, B] = { A first, B second };
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Callers instantiate templates using `? name[...]`:
|
|
131
|
+
|
|
132
|
+
```k
|
|
133
|
+
list? = ? list[bits[]];
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Or directly inside relation parameter guards:
|
|
137
|
+
|
|
138
|
+
```k
|
|
139
|
+
add = ?{ nat[] = N x, ... } < ... >;
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
Template expansion is purely structural and static: internal knots
|
|
143
|
+
(`= Y`) are scoped locally and alpha-renamed, and template calls form an
|
|
144
|
+
acyclic dependency DAG, preventing non-regular divergence.
|
|
145
|
+
|
|
99
146
|
### Partial functions
|
|
100
147
|
|
|
101
148
|
A _partial function_ is a function that is not defined for every value
|
package/DOCS/CODECS.md
CHANGED
|
@@ -158,22 +158,30 @@ The `codecs/` directory provides the following standard codecs:
|
|
|
158
158
|
- JSON objects map directly to products `{ field: val, ... }`.
|
|
159
159
|
- JSON arrays map to indexed products `{ "0": a0, "1": a1, ... }`.
|
|
160
160
|
- Ideal for record-oriented projection (e.g. `.user .name`).
|
|
161
|
-
- **`djson.mjs` (`k-djson`) — Universal Dynamic JSON**:
|
|
162
|
-
Maps every JSON document to a single universal recursive
|
|
163
|
-
`
|
|
161
|
+
- **`djson.mjs` (`k-djson`) — Universal Dynamic JSON (Binary Bit-Trie)**:
|
|
162
|
+
Maps every JSON document to a single universal recursive named
|
|
163
|
+
pattern `djson[]`:
|
|
164
164
|
```k
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
165
|
+
btrie[V] = < {} nil, { opt[V] val, T 0, T 1 } trie > = T;
|
|
166
|
+
|
|
167
|
+
djson[] = <
|
|
168
|
+
number[] number,
|
|
169
|
+
string[] string,
|
|
170
|
+
bool[] bool,
|
|
169
171
|
{} null,
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
172
|
+
btrie[J] list,
|
|
173
|
+
btrie[J] object
|
|
174
|
+
> = J;
|
|
173
175
|
```
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
176
|
+
Uses a unified 27-node binary bit-trie for both arrays and objects.
|
|
177
|
+
Array indices (`bits?`) and string keys are stored along MSB-to-LSB
|
|
178
|
+
bit paths, enabling $O(|K|)$ lookup via direct `.0`/`.1` projections
|
|
179
|
+
with zero allocations and zero character-equality loops.
|
|
180
|
+
Accompanied by a complete standard library on `json?` in
|
|
181
|
+
[`Examples/djson.k`](../Examples/djson.k) (`get`, `nth`, `has?`,
|
|
182
|
+
`has_index?`, `set`, `del`, `set_nth`, `del_nth`, `push`, `keys`,
|
|
183
|
+
`keys_raw`, `values`, `values_raw`, `length`, `length_nat`,
|
|
184
|
+
`type_of`).
|
|
177
185
|
|
|
178
186
|
## Testing a Codec
|
|
179
187
|
|
|
@@ -213,7 +213,7 @@ To make `codecs/` intuitive and self-contained:
|
|
|
213
213
|
codecs/
|
|
214
214
|
├── int.mjs # Integer & list codec
|
|
215
215
|
├── json.mjs # JSON codec (schema-specializing)
|
|
216
|
-
├── djson.mjs # Dynamic JSON codec (
|
|
216
|
+
├── djson.mjs # Dynamic JSON codec (binary bit-trie)
|
|
217
217
|
├── utf8.mjs # UTF-8 string codec
|
|
218
218
|
├── utf16.mjs # UTF-16 string codec
|
|
219
219
|
├── ieee.mjs # IEEE 754 float codec
|
package/DOCS/DICTIONARY.md
CHANGED
|
@@ -67,11 +67,20 @@ schema registry; its `@hash` is a fixed-width indexing digest.
|
|
|
67
67
|
|
|
68
68
|
### Filter
|
|
69
69
|
|
|
70
|
-
A `filter` is an expression `?schema` that constrains the input payload
|
|
71
|
-
schema and otherwise behaves like identity.
|
|
70
|
+
A `filter` is an expression `?schema` that constrains the input payload
|
|
71
|
+
to a schema and otherwise behaves like identity.
|
|
72
72
|
|
|
73
|
-
In recursive relations, filters also serve as **inductive termination
|
|
74
|
-
ensuring contract derivation converges.
|
|
73
|
+
In recursive relations, filters also serve as **inductive termination
|
|
74
|
+
witnesses**, ensuring contract derivation converges.
|
|
75
|
+
|
|
76
|
+
### Named Pattern
|
|
77
|
+
|
|
78
|
+
A `named pattern` is a parameterized regular tree grammar template
|
|
79
|
+
defined using square-bracket syntax (`name[P1, ..., Pn] = filter;`).
|
|
80
|
+
Named patterns declare external input ports in their parameter list
|
|
81
|
+
and may contain internal unification variables and cycle knots
|
|
82
|
+
(`= KnotName`). They are expanded statically without recursive
|
|
83
|
+
divergence, compiling into canonical schema automata nodes.
|
|
75
84
|
|
|
76
85
|
### Canonical Content-Addressed Hash (`@hash`)
|
|
77
86
|
|
package/DOCS/FILE_FORMATS.md
CHANGED
|
@@ -97,8 +97,9 @@ flowchart TD
|
|
|
97
97
|
### What It Is
|
|
98
98
|
`.k` files contain the high-level, human-readable source code of the
|
|
99
99
|
`k` language. A `.k` file defines:
|
|
100
|
-
- **Algebraic
|
|
101
|
-
|
|
100
|
+
- **Algebraic Schemas & Named Patterns:** Products (`{ ... }`),
|
|
101
|
+
tagged unions (`< ... >`), and reusable parameterized pattern
|
|
102
|
+
templates (`name[...] = filter;`).
|
|
102
103
|
- **Relations:** First-order partial functions composed via sequence,
|
|
103
104
|
union, projection, or product construction.
|
|
104
105
|
- **Top-level Expressions:** Computations evaluated when the file is run
|
|
@@ -0,0 +1,302 @@
|
|
|
1
|
+
# Comparison of JSON Approaches in k
|
|
2
|
+
|
|
3
|
+
> [!NOTE]
|
|
4
|
+
> **Resolution**: The **binary bit-trie** was selected and standardized
|
|
5
|
+
> as the sole dynamic JSON implementation in `codecs/djson.mjs` and
|
|
6
|
+
> `Examples/djson.k`. The cons-list and Patricia trie implementations
|
|
7
|
+
> were removed. This document preserves the architectural comparison
|
|
8
|
+
> and benchmark findings that guided this design decision.
|
|
9
|
+
|
|
10
|
+
This document compares the JSON codec and schema implementations:
|
|
11
|
+
|
|
12
|
+
1. **Schema-Specializing JSON** (`json`):
|
|
13
|
+
[`codecs/json.mjs`][json-mjs]
|
|
14
|
+
2. **Universal Dynamic JSON (Binary Bit-Trie)** (`djson`):
|
|
15
|
+
[`codecs/djson.mjs`][djson-mjs] & [`Examples/djson.k`][djson-k]
|
|
16
|
+
|
|
17
|
+
[json-mjs]: file:///home/wojtek/gits/K/k/codecs/json.mjs
|
|
18
|
+
[djson-mjs]: file:///home/wojtek/gits/K/k/codecs/djson.mjs
|
|
19
|
+
[djson-k]: file:///home/wojtek/gits/K/k/Examples/djson.k
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## 1. Architectural Motivation
|
|
24
|
+
|
|
25
|
+
In **k**, all values are algebraic data types classified by closed type
|
|
26
|
+
patterns. External formats like JSON can be embedded into **k** via two
|
|
27
|
+
fundamentally distinct strategies:
|
|
28
|
+
|
|
29
|
+
### Static Schema Specialization (`json`)
|
|
30
|
+
- Infers a concrete closed pattern specialized to the exact shape and
|
|
31
|
+
field names of each input JSON document.
|
|
32
|
+
- Objects become concrete labeled products `{ "name": Str, "age": Num }`.
|
|
33
|
+
- Lists become homogeneous or heterogeneous tuples/cons-lists.
|
|
34
|
+
- **Advantage**: Zero interpretive overhead for known static schemas;
|
|
35
|
+
direct field projection in compiled code.
|
|
36
|
+
- **Limitation**: The pattern is dynamic and grows unboundedly with the
|
|
37
|
+
document. Cannot write universal JSON transformation relations (like a
|
|
38
|
+
generic JSON query engine or REPL printer) under a single fixed closed
|
|
39
|
+
pattern.
|
|
40
|
+
|
|
41
|
+
### Dynamic Uniform JSON (`djson*` family)
|
|
42
|
+
- Defines a single, fixed closed pattern `json?` once for all valid JSON
|
|
43
|
+
values. Any arbitrary JSON document parses into an injection of this
|
|
44
|
+
universal pattern.
|
|
45
|
+
- Allows authoring universal helper relations in `k`:
|
|
46
|
+
- `dot: {key, object} -> val`
|
|
47
|
+
- `has_key?: {key, object} -> ()`
|
|
48
|
+
- `nth: {index, list} -> val`
|
|
49
|
+
- `obj_size: object -> bits?`
|
|
50
|
+
- `list_size: list -> bits?`
|
|
51
|
+
- `remove: {key, object} -> object`
|
|
52
|
+
- `put: {key, val, object} -> object`
|
|
53
|
+
- Compiles once to Wasm or LLVM `-O3` binaries that execute over any
|
|
54
|
+
input JSON document without recompilation.
|
|
55
|
+
|
|
56
|
+
---
|
|
57
|
+
|
|
58
|
+
## 2. Structural Schema Comparison
|
|
59
|
+
|
|
60
|
+
| Dimension | `djson` (Cons) | `djson-btrie` | `djson-ptrie` |
|
|
61
|
+
| :--- | :--- | :--- | :--- |
|
|
62
|
+
| **Nodes** | 29 | 27 | 33 |
|
|
63
|
+
| **Numbers** | float64 | float64 | float64 |
|
|
64
|
+
| **Strings** | Cons-list | Cons-list | Cons-list |
|
|
65
|
+
| **Lists** | Cons-list | Bit-trie | Patricia trie |
|
|
66
|
+
| **Objects** | Sorted cons | Bit-trie | Radix trie |
|
|
67
|
+
| **Node** | `{car, cdr}` | `{val, 0, 1}` | `{pfx, sz, val, d}` |
|
|
68
|
+
| **Indexing**| $O(N)$ linear | $O(\log N)$ | $O(\log N)$ |
|
|
69
|
+
| **Lookup** | $O(K \cdot L)$ | $O(L \cdot 8)$ | $O(L)$ radix |
|
|
70
|
+
| **Size** | $O(N)$ walk | $O(N)$ walk | **$O(1)$** field |
|
|
71
|
+
|
|
72
|
+
### Schema Definitions
|
|
73
|
+
|
|
74
|
+
#### Dynamic Cons-List (`djson`)
|
|
75
|
+
```k
|
|
76
|
+
LIST = < {} nil, { JSON car, LIST cdr } cons >;
|
|
77
|
+
ENTRY = { Str key, JSON val };
|
|
78
|
+
OBJECT = < {} nil, { ENTRY car, OBJECT cdr } cons >;
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
#### Dynamic Binary Bit-Trie (`djson-btrie`)
|
|
82
|
+
```k
|
|
83
|
+
TRIE = <
|
|
84
|
+
{} nil,
|
|
85
|
+
{ < {} nil, JSON val > val, TRIE 0, TRIE 1 } trie
|
|
86
|
+
>;
|
|
87
|
+
LIST = TRIE;
|
|
88
|
+
OBJECT = TRIE;
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
#### Dynamic Patricia/Radix Trie (`djson-ptrie`)
|
|
92
|
+
```k
|
|
93
|
+
LIST = <
|
|
94
|
+
{} nil,
|
|
95
|
+
{
|
|
96
|
+
BITS prefix,
|
|
97
|
+
BITS size,
|
|
98
|
+
< {} nil, JSON val > val,
|
|
99
|
+
LIST 0,
|
|
100
|
+
LIST 1
|
|
101
|
+
} trie
|
|
102
|
+
>;
|
|
103
|
+
DICT = < {} nil, { { CHAR char, OBJECT trie } car, DICT cdr } cons >;
|
|
104
|
+
OBJECT = <
|
|
105
|
+
{} nil,
|
|
106
|
+
{ Str prefix, BITS size, < {} nil, JSON val > val, DICT dict } trie
|
|
107
|
+
>;
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
## 3. Algorithmic Mechanics
|
|
113
|
+
|
|
114
|
+
### 3.1 Object Key Lookup (`dot`)
|
|
115
|
+
- **`djson`**: Iterates through a linked list of `{key, val}` pairs.
|
|
116
|
+
On each entry, performs character-by-character string equality
|
|
117
|
+
`eq_string?`. For $N$ entries with key length $L$, worst-case time is
|
|
118
|
+
$O(N \cdot L)$.
|
|
119
|
+
- **`djson-btrie`**: Uses `step_char` to walk 8 to 24 bits directly down
|
|
120
|
+
the binary trie per character. Because character bits are decoded
|
|
121
|
+
destructively into branch selectors without allocating `bits?` nodes,
|
|
122
|
+
it executes with **zero heap allocations** in compiled LLVM code.
|
|
123
|
+
- **`djson-ptrie`**: Matches the query key against the node's `prefix`
|
|
124
|
+
run using `match_str_prefix`. If the prefix matches, it looks up the
|
|
125
|
+
next branching character in a small local `dict` cons-list.
|
|
126
|
+
Non-branching paths of arbitrary length are traversed in one step.
|
|
127
|
+
|
|
128
|
+
### 3.2 Key Membership (`has_key?`)
|
|
129
|
+
- **Hit**: All three navigate to the target key.
|
|
130
|
+
- **Miss**:
|
|
131
|
+
- `djson` scans all $N$ entries to the end of the cons-list.
|
|
132
|
+
- `djson-btrie` and `djson-ptrie` terminate immediately as soon as a bit
|
|
133
|
+
or character branch is missing, achieving sub-microsecond rejection
|
|
134
|
+
even on large documents.
|
|
135
|
+
|
|
136
|
+
### 3.3 Array Indexing (`nth`)
|
|
137
|
+
- **`djson`**: Decodes a `bits?` natural number with `dec_nat` and walks
|
|
138
|
+
cons cells one by one ($O(I)$).
|
|
139
|
+
- **`djson-btrie`**: Dispatches directly on the bits of the index from
|
|
140
|
+
most-significant to least-significant bit ($O(\log I)$).
|
|
141
|
+
- **`djson-ptrie`**: Compares the remaining index bits against the
|
|
142
|
+
node's `prefix: BITS` in chunks, descending along compressed paths
|
|
143
|
+
($O(\log I)$).
|
|
144
|
+
|
|
145
|
+
### 3.4 Subtree Cardinality (`obj_size`, `list_size`)
|
|
146
|
+
- **`djson` & `djson-btrie`**: Must recursively traverse the entire cons
|
|
147
|
+
list or bit-trie, adding 1 for each present entry using binary addition
|
|
148
|
+
(`inc_nat`). Complexity is $O(N \log N)$.
|
|
149
|
+
- **`djson-ptrie`**: Stores a precomputed binary natural number
|
|
150
|
+
`size: BITS` at every node. Size queries are simple $O(1)$ field
|
|
151
|
+
projections (`ptrie_obj? < /nil 0, /trie .size >`). In LLVM `-O3`, this
|
|
152
|
+
executes in **5 nanoseconds**, independent of document size.
|
|
153
|
+
|
|
154
|
+
### 3.5 Functional Updates (`remove`, `put`)
|
|
155
|
+
- **`djson`**: Copies the cons spine until the modified entry ($O(N)$).
|
|
156
|
+
- **`djson-btrie`**: Rebuilds the bit path down to the leaf. `put` uses
|
|
157
|
+
`str_to_bits` to construct the bit path, while `remove` clears `.val` to
|
|
158
|
+
`nil` along the path.
|
|
159
|
+
- **`djson-ptrie`**:
|
|
160
|
+
- `remove` checks `has_key?`, decrements `size` with `dec_nat` along
|
|
161
|
+
the path, and clears `.val` to `nil`.
|
|
162
|
+
- `put` uses `split_str_prefix` to find the common prefix between the
|
|
163
|
+
new key and existing node. If prefixes diverge, it splits the node
|
|
164
|
+
into a shared parent with a 2-entry branch dictionary in $O(L)$ time.
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
## 4. Empirical Benchmark Results
|
|
169
|
+
|
|
170
|
+
Measured with `npm run perf:djson` on Linux x86_64 across sizes
|
|
171
|
+
$N \in \{10, 100, 500\}$.
|
|
172
|
+
|
|
173
|
+
### 4.1 Wire Representation Sizes
|
|
174
|
+
|
|
175
|
+
#### Shared-Prefix Objects (`{"field_prefix_0000": 0, ...}`)
|
|
176
|
+
- **$N = 10$** (Raw: 230 B):
|
|
177
|
+
- `djson`: 1,029 B (4.47x)
|
|
178
|
+
- `djson-btrie`: **913 B (3.97x)**
|
|
179
|
+
- `djson-ptrie`: 1,032 B (4.49x)
|
|
180
|
+
- `json` (Spec): 370 B (1.61x)
|
|
181
|
+
- **$N = 100$** (Raw: 2,750 B):
|
|
182
|
+
- `djson`: 3,365 B (1.22x)
|
|
183
|
+
- `djson-btrie`: **2,096 B (0.76x)**
|
|
184
|
+
- `djson-ptrie`: 2,367 B (0.86x)
|
|
185
|
+
- `json` (Spec): 1,529 B (0.56x)
|
|
186
|
+
- **$N = 500$** (Raw: 14,750 B):
|
|
187
|
+
- `djson`: 14,357 B (0.97x)
|
|
188
|
+
- `djson-btrie`: **7,370 B (0.50x)**
|
|
189
|
+
- `djson-ptrie`: 8,241 B (0.56x)
|
|
190
|
+
- `json` (Spec): 7,229 B (0.49x)
|
|
191
|
+
|
|
192
|
+
#### Varied-Key Objects (`{"account_0_account_attr": 0, ...}`)
|
|
193
|
+
- **$N = 10$** (Raw: 270 B):
|
|
194
|
+
- `djson`: **1,114 B (4.13x)**
|
|
195
|
+
- `djson-btrie`: 1,460 B (5.41x)
|
|
196
|
+
- `djson-ptrie`: 1,236 B (4.58x)
|
|
197
|
+
- `json` (Spec): 496 B (1.84x)
|
|
198
|
+
- **$N = 100$** (Raw: 3,120 B):
|
|
199
|
+
- `djson`: 4,987 B (1.60x)
|
|
200
|
+
- `djson-btrie`: 6,729 B (2.16x)
|
|
201
|
+
- `djson-ptrie`: **4,309 B (1.38x)**
|
|
202
|
+
- `json` (Spec): 4,460 B (1.43x)
|
|
203
|
+
- **$N = 500$** (Raw: 16,598 B):
|
|
204
|
+
- `djson`: 22,581 B (1.36x)
|
|
205
|
+
- `djson-btrie`: 28,478 B (1.72x)
|
|
206
|
+
- `djson-ptrie`: **17,036 B (1.03x)**
|
|
207
|
+
- `json` (Spec): 22,104 B (1.33x)
|
|
208
|
+
|
|
209
|
+
#### Homogeneous Number Arrays (`[0, 3, 6, ...]`)
|
|
210
|
+
- **$N = 10$** (Raw: 27 B):
|
|
211
|
+
- `djson`: 880 B (32.6x)
|
|
212
|
+
- `djson-btrie`: **878 B (32.5x)**
|
|
213
|
+
- `djson-ptrie`: 999 B (37.0x)
|
|
214
|
+
- **$N = 100$** (Raw: 363 B):
|
|
215
|
+
- `djson`: **1,645 B (4.53x)**
|
|
216
|
+
- `djson-btrie`: 1,666 B (4.59x)
|
|
217
|
+
- `djson-ptrie`: 1,877 B (5.17x)
|
|
218
|
+
- **$N = 500$** (Raw: 2,129 B):
|
|
219
|
+
- `djson`: **5,045 B (2.37x)**
|
|
220
|
+
- `djson-btrie`: 5,166 B (2.43x)
|
|
221
|
+
- `djson-ptrie`: 5,776 B (2.71x)
|
|
222
|
+
|
|
223
|
+
---
|
|
224
|
+
|
|
225
|
+
### 4.2 Execution Latency Across Backends ($N=500$)
|
|
226
|
+
|
|
227
|
+
#### Object Lookup (`dot` on last key of 500-key object)
|
|
228
|
+
- **Shared-Prefix Keys**:
|
|
229
|
+
- `djson`: Wasm $580.4\ \mu\text{s}$, LLVM $23.6\ \mu\text{s}$
|
|
230
|
+
- `djson-btrie`: Wasm $7.4\ \mu\text{s}$, LLVM **$2.4\ \mu\text{s}$**
|
|
231
|
+
- `djson-ptrie`: Wasm $11.1\ \mu\text{s}$, LLVM $5.1\ \mu\text{s}$
|
|
232
|
+
- **Varied Keys**:
|
|
233
|
+
- `djson`: Wasm $26.8\ \mu\text{s}$, LLVM $4.2\ \mu\text{s}$
|
|
234
|
+
- `djson-btrie`: Wasm $10.9\ \mu\text{s}$, LLVM **$3.1\ \mu\text{s}$**
|
|
235
|
+
- `djson-ptrie`: Wasm $10.8\ \mu\text{s}$, LLVM $4.4\ \mu\text{s}$
|
|
236
|
+
|
|
237
|
+
#### Key Membership Filter (`has_key?` miss on nonexistent key)
|
|
238
|
+
- `djson`: Wasm $63.9\ \mu\text{s}$, LLVM $1,168\ \text{ns}$
|
|
239
|
+
- `djson-btrie`: Wasm $1,022\ \text{ns}$, LLVM **$466\ \text{ns}$**
|
|
240
|
+
- `djson-ptrie`: Wasm $1,908\ \text{ns}$, LLVM $1,228\ \text{ns}$
|
|
241
|
+
|
|
242
|
+
#### Array Element Lookup (`nth` on index 499 of 500 elements)
|
|
243
|
+
- `djson`:
|
|
244
|
+
kVM $10.74\ \text{ms}$, Wasm $97.2\ \mu\text{s}$,
|
|
245
|
+
LLVM $100.7\ \mu\text{s}$
|
|
246
|
+
- `djson-btrie`:
|
|
247
|
+
kVM $26.0\ \mu\text{s}$, Wasm $733\ \text{ns}$,
|
|
248
|
+
LLVM **$114\ \text{ns}$** (**881x speedup**)
|
|
249
|
+
- `djson-ptrie`:
|
|
250
|
+
kVM $35.7\ \mu\text{s}$, Wasm $1,267\ \text{ns}$,
|
|
251
|
+
LLVM $371\ \text{ns}$ (**271x speedup**)
|
|
252
|
+
|
|
253
|
+
#### Cardinality Queries (`obj_size` for $N=500$)
|
|
254
|
+
- `djson`:
|
|
255
|
+
kVM $13.57\ \text{ms}$, Wasm $89.2\ \mu\text{s}$,
|
|
256
|
+
LLVM $127.4\ \mu\text{s}$
|
|
257
|
+
- `djson-btrie`:
|
|
258
|
+
kVM $210.88\ \text{ms}$, Wasm $3.02\ \text{ms}$,
|
|
259
|
+
LLVM $8.06\ \text{ms}$
|
|
260
|
+
- `djson-ptrie`:
|
|
261
|
+
kVM $2,054\ \text{ns}$, Wasm $172\ \text{ns}$,
|
|
262
|
+
LLVM **$5\ \text{ns}$** (**26,106x speedup**)
|
|
263
|
+
|
|
264
|
+
#### Object Key Updates ($N=500$)
|
|
265
|
+
- **`remove`**:
|
|
266
|
+
- `djson`: LLVM $79.6\ \mu\text{s}$
|
|
267
|
+
- `djson-btrie`: LLVM **$9.5\ \mu\text{s}$** (8.4x speedup)
|
|
268
|
+
- `djson-ptrie`: LLVM $13.3\ \mu\text{s}$ (6.0x speedup)
|
|
269
|
+
- **`put`**:
|
|
270
|
+
- `djson`: LLVM $50.8\ \mu\text{s}$
|
|
271
|
+
- `djson-btrie`: LLVM $12.6\ \mu\text{s}$ (4.0x speedup)
|
|
272
|
+
- `djson-ptrie`: LLVM **$3.9\ \mu\text{s}$** (13.1x speedup)
|
|
273
|
+
|
|
274
|
+
---
|
|
275
|
+
|
|
276
|
+
## 5. Trade-off Analysis & Recommendation Matrix
|
|
277
|
+
|
|
278
|
+
| Feature | `json` | `djson` | `btrie` | `ptrie` |
|
|
279
|
+
| :--- | :---: | :---: | :---: | :---: |
|
|
280
|
+
| **Universal Pattern** | ❌ No | ✅ 29 | ✅ 27 | ✅ 33 |
|
|
281
|
+
| **REPL Inspection** | Native | Cons | Bit-tree | Radix |
|
|
282
|
+
| **Shared Prefix Wire**| Small | Med | **Small** | Small |
|
|
283
|
+
| **Varied Keys Wire** | Large | Med | Large | **Small** |
|
|
284
|
+
| **Array ($N=500$)** | Direct | $100\ \mu\text{s}$ | **114ns**| 371ns |
|
|
285
|
+
| **Dot ($N=500$)** | Direct | $23.6\ \mu\text{s}$| **2.4µs**| 5.1µs |
|
|
286
|
+
| **Put ($N=500$)** | N/A | $50.8\ \mu\text{s}$| 12.6µs | **3.9µs**|
|
|
287
|
+
| **Size ($N=500$)** | N/A | $127\ \mu\text{s}$ | 8.1ms | **5ns** |
|
|
288
|
+
|
|
289
|
+
### Recommendations
|
|
290
|
+
1. **Choose `djson-ptrie` for general-purpose dynamic JSON**:
|
|
291
|
+
It achieves the best wire compression on real-world varied keys
|
|
292
|
+
(1.03x of raw JSON), provides instant $O(1)$ cardinality queries
|
|
293
|
+
(5 ns in LLVM), and delivers the fastest updates (`put` in 3.9 µs).
|
|
294
|
+
2. **Choose `djson-btrie` for extreme lookup throughput on shared keys**:
|
|
295
|
+
Destructive bit-walking in `dot` avoids all node allocation,
|
|
296
|
+
achieving the lowest lookup latency (2.4 µs in LLVM).
|
|
297
|
+
3. **Keep `djson` for simple, lightweight tooling**:
|
|
298
|
+
Simple cons-list structure is human-readable in REPL raw outputs and
|
|
299
|
+
has minimal code complexity for small payloads ($N < 50$).
|
|
300
|
+
4. **Choose `json` when document schema is fixed at compile time**:
|
|
301
|
+
Avoids all dynamic dictionary lookups in exchange for a
|
|
302
|
+
document-specialized pattern.
|