@fraczak/k 7.2.0 → 8.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +66 -0
- package/DOCS/BIG_PICTURE.md +54 -7
- package/DOCS/CODECS.md +21 -11
- package/DOCS/CODEC_ARCHITECTURE.md +5 -3
- package/DOCS/DICTIONARY.md +13 -4
- package/DOCS/FILE_FORMATS.md +3 -2
- package/DOCS/JSON_APPROACHES_COMPARISON.md +306 -0
- package/DOCS/NAMED_PATTERNS.md +317 -0
- package/DOCS/PATTERNS.md +59 -11
- package/DOCS/REPL.md +13 -2
- package/DOCS/WEB_REPL_REQUIREMENTS.md +4 -2
- package/Examples/arithmetics.k +22 -17
- package/Examples/bnat.k +21 -11
- package/Examples/buda.k +37 -20
- package/Examples/byte.k +3 -1
- package/Examples/code.k +38 -39
- package/Examples/djson-btrie.k +414 -0
- package/Examples/djson-ptrie.k +663 -0
- package/Examples/djson.k +323 -0
- package/Examples/esc.k +1 -1
- package/Examples/karatsuba-mult.k +16 -13
- package/Examples/list.k +8 -4
- package/Examples/nat.k +17 -11
- package/Examples/poly.k +11 -3
- package/Examples/vhdl_to_k.k +43 -31
- package/README.md +42 -16
- package/browser/repl-browser.mjs +6 -0
- package/browser/tree-view.mjs +35 -16
- package/browser/vfs-data.mjs +15 -7
- package/codecs/README.md +2 -0
- package/codecs/djson-btrie.mjs +560 -0
- package/codecs/djson-ptrie.mjs +587 -0
- package/core.k +79 -115
- package/index.mjs +3 -0
- package/named-patterns.mjs +267 -0
- package/object.mjs +31 -3
- package/package.json +4 -1
- package/parser.jison +68 -5
- package/parser.mjs +111 -60
- package/pretty.mjs +5 -0
- package/repl.mjs +188 -22
- package/scripts/build-repl-html.mjs +68 -4
- package/scripts/perf-djson.mjs +1142 -0
- package/scripts/run-tests.mjs +5 -0
- package/symbol-table.mjs +20 -2
- package/tests/bench-djson-tries.mjs +340 -0
- package/tests/test-codecs.mjs +59 -16
- package/tests/test-named-patterns.mjs +189 -0
- package/tests/test-repl-html.mjs +51 -4
- package/tests/test-repl.mjs +190 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,57 @@ All notable changes to this project will be documented here.
|
|
|
5
5
|
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
|
6
6
|
This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [8.0.0] — 2026-10-20 — Named Patterns, Type Schemes & Trie Codecs
|
|
9
|
+
|
|
10
|
+
### Language & Type System
|
|
11
|
+
|
|
12
|
+
- **Named Patterns & Parameterized Schemes (`named-patterns.mjs`)**:
|
|
13
|
+
- Introduced first-class syntax for closed named patterns:
|
|
14
|
+
`byte[] = { bit 0, bit 1, ... };` and `bool[] = < bit false, ... >;`.
|
|
15
|
+
- Added parameterized type schemes with explicit type variables:
|
|
16
|
+
`list[X] = < X head, (list[X]) tail >;` and
|
|
17
|
+
`pair[A, B] = { A first, B second };`.
|
|
18
|
+
- Implemented monomorphic expansion with cycle detection and sharing.
|
|
19
|
+
- Full object file support: `.ko` and `.klib` payloads serialize and
|
|
20
|
+
deserialize named pattern tables alongside relations.
|
|
21
|
+
|
|
22
|
+
- **Standard Library & Examples Refactoring**:
|
|
23
|
+
- Adopted named patterns throughout `core.k` (`bit[]`, `byte[]`,
|
|
24
|
+
`pair[A, B]`, `either[A, B]`, `option[X]`).
|
|
25
|
+
- Refactored `Examples/*.k` (`djson.k`, `arithmetics.k`, `bnat.k`,
|
|
26
|
+
`nat.k`, `code.k`, `karatsuba-mult.k`, `poly.k`, `vhdl_to_k.k`)
|
|
27
|
+
for clarity and conciseness using named patterns.
|
|
28
|
+
|
|
29
|
+
### Dynamic JSON Codecs
|
|
30
|
+
|
|
31
|
+
- **Bit-Trie Codec (`codecs/djson-btrie.mjs`, `k-djson-btrie`)**:
|
|
32
|
+
- Added binary bit-trie dynamic JSON codec with variable-length
|
|
33
|
+
bit prefixing for compact wire encodings.
|
|
34
|
+
- Accompanied by formal K specification in `Examples/djson-btrie.k`.
|
|
35
|
+
|
|
36
|
+
- **Packed Nibble-Trie Codec (`codecs/djson-ptrie.mjs`)**:
|
|
37
|
+
- Added packed 4-bit nibble trie codec (`k-djson-ptrie`) for
|
|
38
|
+
optimized traversal.
|
|
39
|
+
- Accompanied by formal K specification in `Examples/djson-ptrie.k`.
|
|
40
|
+
|
|
41
|
+
- **JSON Approaches Comparison & Benchmarks**:
|
|
42
|
+
- Added comparative architecture and benchmark documentation in
|
|
43
|
+
`DOCS/JSON_APPROACHES_COMPARISON.md`.
|
|
44
|
+
- Added benchmark suite in `scripts/perf-djson.mjs` and
|
|
45
|
+
`tests/bench-djson-tries.mjs`.
|
|
46
|
+
|
|
47
|
+
### Interactive REPL & Tooling
|
|
48
|
+
|
|
49
|
+
- **Pattern Inspection & Persistence in REPL**:
|
|
50
|
+
- Added `:patterns` / `:pat` commands to inspect loaded patterns.
|
|
51
|
+
- Display canonical type code / relation hashes:
|
|
52
|
+
`name[] = ... -- @hash`.
|
|
53
|
+
- Added pattern lookup by `@hash` (`:pattern @hash`, `:pat @hash`).
|
|
54
|
+
- Tab-completion support for pattern names and canonical `@hashes`.
|
|
55
|
+
- Full pattern persistence across REPL sessions via `.klib` files.
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
8
59
|
## [7.2.0] — 2026-10-17 — Dynamic JSON Codec (djson) & Lazy Wire Decoding
|
|
9
60
|
|
|
10
61
|
### Features & Performance Improvements
|
|
@@ -440,6 +491,21 @@ This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
440
491
|
|
|
441
492
|
---
|
|
442
493
|
|
|
494
|
+
[8.0.0]: https://github.com/fraczak/k/releases/tag/v8.0.0
|
|
495
|
+
[7.2.0]: https://github.com/fraczak/k/releases/tag/v7.2.0
|
|
496
|
+
[7.1.0]: https://github.com/fraczak/k/releases/tag/v7.1.0
|
|
497
|
+
[7.0.2]: https://github.com/fraczak/k/releases/tag/v7.0.2
|
|
498
|
+
[7.0.1]: https://github.com/fraczak/k/releases/tag/v7.0.1
|
|
499
|
+
[7.0.0]: https://github.com/fraczak/k/releases/tag/v7.0.0
|
|
500
|
+
[6.10.6]: https://github.com/fraczak/k/releases/tag/v6.10.6
|
|
501
|
+
[6.10.5]: https://github.com/fraczak/k/releases/tag/v6.10.5
|
|
502
|
+
[6.10.4]: https://github.com/fraczak/k/releases/tag/v6.10.4
|
|
503
|
+
[6.10.3]: https://github.com/fraczak/k/releases/tag/v6.10.3
|
|
504
|
+
[6.10.2]: https://github.com/fraczak/k/releases/tag/v6.10.2
|
|
505
|
+
[6.10.1]: https://github.com/fraczak/k/releases/tag/v6.10.1
|
|
506
|
+
[6.10.0]: https://github.com/fraczak/k/releases/tag/v6.10.0
|
|
507
|
+
[6.9.0]: https://github.com/fraczak/k/releases/tag/v6.9.0
|
|
508
|
+
[6.5.0]: https://github.com/fraczak/k/releases/tag/v6.5.0
|
|
443
509
|
[6.2.3]: https://github.com/fraczak/k/releases/tag/v6.2.3
|
|
444
510
|
[6.1.0]: https://github.com/fraczak/k/releases/tag/v6.1.0
|
|
445
511
|
[6.0.0]: https://github.com/fraczak/k/releases/tag/v6.0.0
|
package/DOCS/BIG_PICTURE.md
CHANGED
|
@@ -69,21 +69,24 @@ states, so any two structurally equivalent schemas yield identical canonical tex
|
|
|
69
69
|
|
|
70
70
|
### Filters
|
|
71
71
|
|
|
72
|
-
A _filter_ is a syntactic expression `?schema` that acts as a partial
|
|
73
|
-
relation on values conforming to `schema`. If the input value
|
|
74
|
-
schema, it passes through unchanged; otherwise, the
|
|
72
|
+
A _filter_ is a syntactic expression `?schema` that acts as a partial
|
|
73
|
+
identity relation on values conforming to `schema`. If the input value
|
|
74
|
+
matches the schema, it passes through unchanged; otherwise, the
|
|
75
|
+
relation is undefined.
|
|
75
76
|
|
|
76
|
-
In recursive relations, filters also serve as **inductive termination
|
|
77
|
-
ensuring type and contract derivation converges.
|
|
77
|
+
In recursive relations, filters also serve as **inductive termination
|
|
78
|
+
witnesses**, ensuring type and contract derivation converges.
|
|
78
79
|
|
|
79
80
|
In general, a filter schema is defined by:
|
|
80
81
|
|
|
81
82
|
```bnf
|
|
82
83
|
filter ::= name | '{' filter_label_list '}'
|
|
83
84
|
| '<' filter_label_list '>' | '(' filter_label_list ')'
|
|
85
|
+
| name '[' filter_args ']'
|
|
84
86
|
| filter '=' name
|
|
85
87
|
filter_label_list ::= /* empty */ | '...'
|
|
86
88
|
| ( filter label ',' ) * filter label
|
|
89
|
+
filter_args ::= /* empty */ | ( filter ',' ) * filter
|
|
87
90
|
```
|
|
88
91
|
|
|
89
92
|
Examples:
|
|
@@ -92,10 +95,54 @@ Examples:
|
|
|
92
95
|
X -- unconstrained schema variable
|
|
93
96
|
( ... ) -- unconstrained schema
|
|
94
97
|
{ ... } -- any product schema
|
|
95
|
-
|
|
96
|
-
< X x, ... > = X --
|
|
98
|
+
{ X x, X y } -- product with two fields of the same schema X
|
|
99
|
+
< X x, ... > = X -- recursive variant X with variant x of schema X
|
|
100
|
+
list[int[]] -- instantiated named pattern
|
|
97
101
|
```
|
|
98
102
|
|
|
103
|
+
### Named Patterns and Parameterized Schemas
|
|
104
|
+
|
|
105
|
+
k supports reusable, parameterized regular tree grammar templates via
|
|
106
|
+
square-bracket syntax `name[...]`:
|
|
107
|
+
|
|
108
|
+
```k
|
|
109
|
+
name[ P1, P2, ... Pn ] = filter;
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
For 0-arity templates, empty brackets `[]` are used:
|
|
113
|
+
|
|
114
|
+
```k
|
|
115
|
+
bit[] = < {} 0, {} 1 >;
|
|
116
|
+
bits[] = < {} _, B 0, B 1 > = B;
|
|
117
|
+
byte[] = {
|
|
118
|
+
bit[] 0, bit[] 1, bit[] 2, bit[] 3,
|
|
119
|
+
bit[] 4, bit[] 5, bit[] 6, bit[] 7
|
|
120
|
+
};
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Parameterized templates introduce external ports:
|
|
124
|
+
|
|
125
|
+
```k
|
|
126
|
+
list[X] = < {} nil, { X car, Y cdr } cons > = Y;
|
|
127
|
+
pair[A, B] = { A first, B second };
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Callers instantiate templates using `? name[...]`:
|
|
131
|
+
|
|
132
|
+
```k
|
|
133
|
+
list? = ? list[bits[]];
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Or directly inside relation parameter guards:
|
|
137
|
+
|
|
138
|
+
```k
|
|
139
|
+
add = ?{ nat[] = N x, ... } < ... >;
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
Template expansion is purely structural and static: internal knots
|
|
143
|
+
(`= Y`) are scoped locally and alpha-renamed, and template calls form an
|
|
144
|
+
acyclic dependency DAG, preventing non-regular divergence.
|
|
145
|
+
|
|
99
146
|
### Partial functions
|
|
100
147
|
|
|
101
148
|
A _partial function_ is a function that is not defined for every value
|
package/DOCS/CODECS.md
CHANGED
|
@@ -158,22 +158,32 @@ The `codecs/` directory provides the following standard codecs:
|
|
|
158
158
|
- JSON objects map directly to products `{ field: val, ... }`.
|
|
159
159
|
- JSON arrays map to indexed products `{ "0": a0, "1": a1, ... }`.
|
|
160
160
|
- Ideal for record-oriented projection (e.g. `.user .name`).
|
|
161
|
-
- **`djson.mjs` (`k-djson`) — Universal Dynamic JSON**:
|
|
162
|
-
Maps every JSON document to a single universal recursive pattern
|
|
163
|
-
`
|
|
161
|
+
- **`djson.mjs` (`k-djson`) — Universal Dynamic JSON (Cons-List)**:
|
|
162
|
+
Maps every JSON document to a single universal recursive named pattern
|
|
163
|
+
`djson[]`:
|
|
164
164
|
```k
|
|
165
|
-
|
|
165
|
+
djson[] = <
|
|
166
166
|
float64 number,
|
|
167
167
|
str string,
|
|
168
168
|
< {} true, {} false > bool,
|
|
169
169
|
{} null,
|
|
170
|
-
< {} nil, {
|
|
171
|
-
< {} nil, { { str key,
|
|
172
|
-
|
|
170
|
+
< {} nil, { J car, L cdr } cons > = L list,
|
|
171
|
+
< {} nil, { { str key, J val } car, O cdr } cons > = O object
|
|
172
|
+
> = J;
|
|
173
173
|
```
|
|
174
174
|
Because every JSON value shares the exact same static 29-node pattern,
|
|
175
|
-
a `k` relation can be compiled once against `
|
|
175
|
+
a `k` relation can be compiled once against `djson[]` and process
|
|
176
176
|
arbitrary JSON documents of any shape or array/object length.
|
|
177
|
+
- **`djson-btrie.mjs` (`k-djson-btrie`) — Binary Bit-Trie Dynamic JSON**:
|
|
178
|
+
Replaces the linear `list` and `object` cons-lists with a unified
|
|
179
|
+
27-node binary bit-trie (`TRIE`). Array indices (`bits?`) and string
|
|
180
|
+
keys are stored along MSB-to-LSB bit paths, enabling $O(|K|)$ lookup
|
|
181
|
+
(`dot` and `nth` in `Examples/djson-btrie.k`) via direct `.0`/`.1`
|
|
182
|
+
projections with zero character-equality checks.
|
|
183
|
+
- **`djson-ptrie.mjs` (`k-djson-ptrie`) — Patricia/Radix Dynamic JSON**:
|
|
184
|
+
Models `list` and `object` as path-compressed radix tries (`trie-array`
|
|
185
|
+
style) with `prefix` and $O(1)$ subtree `size` (`list_size` and
|
|
186
|
+
`obj_size` in `Examples/djson-ptrie.k`).
|
|
177
187
|
|
|
178
188
|
## Testing a Codec
|
|
179
189
|
|
|
@@ -208,9 +218,9 @@ npm test
|
|
|
208
218
|
## Web REPL (VFS) Bundling
|
|
209
219
|
|
|
210
220
|
When `repl.html` is generated (`npm run build:repl-html`):
|
|
211
|
-
- Codecs (`int.mjs`, `json.mjs`, `djson.mjs`, `
|
|
212
|
-
`ieee.mjs`) are bundled into
|
|
213
|
-
imports.
|
|
221
|
+
- Codecs (`int.mjs`, `json.mjs`, `djson.mjs`, `djson-btrie.mjs`,
|
|
222
|
+
`djson-ptrie.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`) are bundled into
|
|
223
|
+
100% self-contained ES modules with zero imports.
|
|
214
224
|
- `runCodecCLI` is shimmed as a no-op, and `Value` helpers are directly
|
|
215
225
|
inlined.
|
|
216
226
|
- In the browser, the VFS serves these self-contained modules directly to
|
|
@@ -213,7 +213,9 @@ To make `codecs/` intuitive and self-contained:
|
|
|
213
213
|
codecs/
|
|
214
214
|
├── int.mjs # Integer & list codec
|
|
215
215
|
├── json.mjs # JSON codec (schema-specializing)
|
|
216
|
-
├── djson.mjs # Dynamic JSON codec (
|
|
216
|
+
├── djson.mjs # Dynamic JSON codec (cons-list)
|
|
217
|
+
├── djson-btrie.mjs # Dynamic JSON codec (binary bit-trie)
|
|
218
|
+
├── djson-ptrie.mjs # Dynamic JSON codec (Patricia/radix trie)
|
|
217
219
|
├── utf8.mjs # UTF-8 string codec
|
|
218
220
|
├── utf16.mjs # UTF-16 string codec
|
|
219
221
|
├── ieee.mjs # IEEE 754 float codec
|
|
@@ -242,8 +244,8 @@ In the browser REPL (`repl.html`):
|
|
|
242
244
|
imports**.
|
|
243
245
|
- **No Runtime Files in VFS**: `codecs/runtime/*` is not included in the
|
|
244
246
|
browser VFS. The VFS only contains verified, self-contained user-facing
|
|
245
|
-
files (`int.mjs`, `json.mjs`, `djson.mjs`, `
|
|
246
|
-
`ieee.mjs`).
|
|
247
|
+
files (`int.mjs`, `json.mjs`, `djson.mjs`, `djson-btrie.mjs`,
|
|
248
|
+
`djson-ptrie.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`).
|
|
247
249
|
- **Native Browser Loading**: In `repl-codecs.mjs`, the browser reads the
|
|
248
250
|
codec source from VFS, wraps it in a Blob, and invokes native `await
|
|
249
251
|
import(blobUrl)`. No regex manipulation, import stripping, or global
|
package/DOCS/DICTIONARY.md
CHANGED
|
@@ -67,11 +67,20 @@ schema registry; its `@hash` is a fixed-width indexing digest.
|
|
|
67
67
|
|
|
68
68
|
### Filter
|
|
69
69
|
|
|
70
|
-
A `filter` is an expression `?schema` that constrains the input payload
|
|
71
|
-
schema and otherwise behaves like identity.
|
|
70
|
+
A `filter` is an expression `?schema` that constrains the input payload
|
|
71
|
+
to a schema and otherwise behaves like identity.
|
|
72
72
|
|
|
73
|
-
In recursive relations, filters also serve as **inductive termination
|
|
74
|
-
ensuring contract derivation converges.
|
|
73
|
+
In recursive relations, filters also serve as **inductive termination
|
|
74
|
+
witnesses**, ensuring contract derivation converges.
|
|
75
|
+
|
|
76
|
+
### Named Pattern
|
|
77
|
+
|
|
78
|
+
A `named pattern` is a parameterized regular tree grammar template
|
|
79
|
+
defined using square-bracket syntax (`name[P1, ..., Pn] = filter;`).
|
|
80
|
+
Named patterns declare external input ports in their parameter list
|
|
81
|
+
and may contain internal unification variables and cycle knots
|
|
82
|
+
(`= KnotName`). They are expanded statically without recursive
|
|
83
|
+
divergence, compiling into canonical schema automata nodes.
|
|
75
84
|
|
|
76
85
|
### Canonical Content-Addressed Hash (`@hash`)
|
|
77
86
|
|
package/DOCS/FILE_FORMATS.md
CHANGED
|
@@ -97,8 +97,9 @@ flowchart TD
|
|
|
97
97
|
### What It Is
|
|
98
98
|
`.k` files contain the high-level, human-readable source code of the
|
|
99
99
|
`k` language. A `.k` file defines:
|
|
100
|
-
- **Algebraic
|
|
101
|
-
|
|
100
|
+
- **Algebraic Schemas & Named Patterns:** Products (`{ ... }`),
|
|
101
|
+
tagged unions (`< ... >`), and reusable parameterized pattern
|
|
102
|
+
templates (`name[...] = filter;`).
|
|
102
103
|
- **Relations:** First-order partial functions composed via sequence,
|
|
103
104
|
union, projection, or product construction.
|
|
104
105
|
- **Top-level Expressions:** Computations evaluated when the file is run
|
|
@@ -0,0 +1,306 @@
|
|
|
1
|
+
# Comparison of JSON Approaches in k
|
|
2
|
+
|
|
3
|
+
This document compares the four JSON codec and schema implementations in
|
|
4
|
+
the **k** repository:
|
|
5
|
+
|
|
6
|
+
1. **Schema-Specializing JSON** (`json`):
|
|
7
|
+
[`codecs/json.mjs`][json-mjs]
|
|
8
|
+
2. **Dynamic Cons-List JSON** (`djson`):
|
|
9
|
+
[`codecs/djson.mjs`][djson-mjs] & [`Examples/djson.k`][djson-k]
|
|
10
|
+
3. **Dynamic Binary Bit-Trie JSON** (`djson-btrie`):
|
|
11
|
+
[`codecs/djson-btrie.mjs`][djson-btrie-mjs] &
|
|
12
|
+
[`Examples/djson-btrie.k`][djson-btrie-k]
|
|
13
|
+
4. **Dynamic Patricia/Radix Trie JSON** (`djson-ptrie`):
|
|
14
|
+
[`codecs/djson-ptrie.mjs`][djson-ptrie-mjs] &
|
|
15
|
+
[`Examples/djson-ptrie.k`][djson-ptrie-k]
|
|
16
|
+
|
|
17
|
+
[json-mjs]: file:///home/wojtek/gits/K/k/codecs/json.mjs
|
|
18
|
+
[djson-mjs]: file:///home/wojtek/gits/K/k/codecs/djson.mjs
|
|
19
|
+
[djson-k]: file:///home/wojtek/gits/K/k/Examples/djson.k
|
|
20
|
+
[djson-btrie-mjs]: file:///home/wojtek/gits/K/k/codecs/djson-btrie.mjs
|
|
21
|
+
[djson-btrie-k]: file:///home/wojtek/gits/K/k/Examples/djson-btrie.k
|
|
22
|
+
[djson-ptrie-mjs]: file:///home/wojtek/gits/K/k/codecs/djson-ptrie.mjs
|
|
23
|
+
[djson-ptrie-k]: file:///home/wojtek/gits/K/k/Examples/djson-ptrie.k
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## 1. Architectural Motivation
|
|
28
|
+
|
|
29
|
+
In **k**, all values are algebraic data types classified by closed type
|
|
30
|
+
patterns. External formats like JSON can be embedded into **k** via two
|
|
31
|
+
fundamentally distinct strategies:
|
|
32
|
+
|
|
33
|
+
### Static Schema Specialization (`json`)
|
|
34
|
+
- Infers a concrete closed pattern specialized to the exact shape and
|
|
35
|
+
field names of each input JSON document.
|
|
36
|
+
- Objects become concrete labeled products `{ "name": Str, "age": Num }`.
|
|
37
|
+
- Lists become homogeneous or heterogeneous tuples/cons-lists.
|
|
38
|
+
- **Advantage**: Zero interpretive overhead for known static schemas;
|
|
39
|
+
direct field projection in compiled code.
|
|
40
|
+
- **Limitation**: The pattern is dynamic and grows unboundedly with the
|
|
41
|
+
document. Cannot write universal JSON transformation relations (like a
|
|
42
|
+
generic JSON query engine or REPL printer) under a single fixed closed
|
|
43
|
+
pattern.
|
|
44
|
+
|
|
45
|
+
### Dynamic Uniform JSON (`djson*` family)
|
|
46
|
+
- Defines a single, fixed closed pattern `json?` once for all valid JSON
|
|
47
|
+
values. Any arbitrary JSON document parses into an injection of this
|
|
48
|
+
universal pattern.
|
|
49
|
+
- Allows authoring universal helper relations in `k`:
|
|
50
|
+
- `dot: {key, object} -> val`
|
|
51
|
+
- `has_key?: {key, object} -> ()`
|
|
52
|
+
- `nth: {index, list} -> val`
|
|
53
|
+
- `obj_size: object -> bits?`
|
|
54
|
+
- `list_size: list -> bits?`
|
|
55
|
+
- `remove: {key, object} -> object`
|
|
56
|
+
- `put: {key, val, object} -> object`
|
|
57
|
+
- Compiles once to Wasm or LLVM `-O3` binaries that execute over any
|
|
58
|
+
input JSON document without recompilation.
|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## 2. Structural Schema Comparison
|
|
63
|
+
|
|
64
|
+
| Dimension | `djson` (Cons) | `djson-btrie` | `djson-ptrie` |
|
|
65
|
+
| :--- | :--- | :--- | :--- |
|
|
66
|
+
| **Nodes** | 29 | 27 | 33 |
|
|
67
|
+
| **Numbers** | float64 | float64 | float64 |
|
|
68
|
+
| **Strings** | Cons-list | Cons-list | Cons-list |
|
|
69
|
+
| **Lists** | Cons-list | Bit-trie | Patricia trie |
|
|
70
|
+
| **Objects** | Sorted cons | Bit-trie | Radix trie |
|
|
71
|
+
| **Node** | `{car, cdr}` | `{val, 0, 1}` | `{pfx, sz, val, d}` |
|
|
72
|
+
| **Indexing**| $O(N)$ linear | $O(\log N)$ | $O(\log N)$ |
|
|
73
|
+
| **Lookup** | $O(K \cdot L)$ | $O(L \cdot 8)$ | $O(L)$ radix |
|
|
74
|
+
| **Size** | $O(N)$ walk | $O(N)$ walk | **$O(1)$** field |
|
|
75
|
+
|
|
76
|
+
### Schema Definitions
|
|
77
|
+
|
|
78
|
+
#### Dynamic Cons-List (`djson`)
|
|
79
|
+
```k
|
|
80
|
+
LIST = < {} nil, { JSON car, LIST cdr } cons >;
|
|
81
|
+
ENTRY = { Str key, JSON val };
|
|
82
|
+
OBJECT = < {} nil, { ENTRY car, OBJECT cdr } cons >;
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
#### Dynamic Binary Bit-Trie (`djson-btrie`)
|
|
86
|
+
```k
|
|
87
|
+
TRIE = <
|
|
88
|
+
{} nil,
|
|
89
|
+
{ < {} nil, JSON val > val, TRIE 0, TRIE 1 } trie
|
|
90
|
+
>;
|
|
91
|
+
LIST = TRIE;
|
|
92
|
+
OBJECT = TRIE;
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
#### Dynamic Patricia/Radix Trie (`djson-ptrie`)
|
|
96
|
+
```k
|
|
97
|
+
LIST = <
|
|
98
|
+
{} nil,
|
|
99
|
+
{
|
|
100
|
+
BITS prefix,
|
|
101
|
+
BITS size,
|
|
102
|
+
< {} nil, JSON val > val,
|
|
103
|
+
LIST 0,
|
|
104
|
+
LIST 1
|
|
105
|
+
} trie
|
|
106
|
+
>;
|
|
107
|
+
DICT = < {} nil, { { CHAR char, OBJECT trie } car, DICT cdr } cons >;
|
|
108
|
+
OBJECT = <
|
|
109
|
+
{} nil,
|
|
110
|
+
{ Str prefix, BITS size, < {} nil, JSON val > val, DICT dict } trie
|
|
111
|
+
>;
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## 3. Algorithmic Mechanics
|
|
117
|
+
|
|
118
|
+
### 3.1 Object Key Lookup (`dot`)
|
|
119
|
+
- **`djson`**: Iterates through a linked list of `{key, val}` pairs.
|
|
120
|
+
On each entry, performs character-by-character string equality
|
|
121
|
+
`eq_string?`. For $N$ entries with key length $L$, worst-case time is
|
|
122
|
+
$O(N \cdot L)$.
|
|
123
|
+
- **`djson-btrie`**: Uses `step_char` to walk 8 to 24 bits directly down
|
|
124
|
+
the binary trie per character. Because character bits are decoded
|
|
125
|
+
destructively into branch selectors without allocating `bits?` nodes,
|
|
126
|
+
it executes with **zero heap allocations** in compiled LLVM code.
|
|
127
|
+
- **`djson-ptrie`**: Matches the query key against the node's `prefix`
|
|
128
|
+
run using `match_str_prefix`. If the prefix matches, it looks up the
|
|
129
|
+
next branching character in a small local `dict` cons-list.
|
|
130
|
+
Non-branching paths of arbitrary length are traversed in one step.
|
|
131
|
+
|
|
132
|
+
### 3.2 Key Membership (`has_key?`)
|
|
133
|
+
- **Hit**: All three navigate to the target key.
|
|
134
|
+
- **Miss**:
|
|
135
|
+
- `djson` scans all $N$ entries to the end of the cons-list.
|
|
136
|
+
- `djson-btrie` and `djson-ptrie` terminate immediately as soon as a bit
|
|
137
|
+
or character branch is missing, achieving sub-microsecond rejection
|
|
138
|
+
even on large documents.
|
|
139
|
+
|
|
140
|
+
### 3.3 Array Indexing (`nth`)
|
|
141
|
+
- **`djson`**: Decodes a `bits?` natural number with `dec_nat` and walks
|
|
142
|
+
cons cells one by one ($O(I)$).
|
|
143
|
+
- **`djson-btrie`**: Dispatches directly on the bits of the index from
|
|
144
|
+
most-significant to least-significant bit ($O(\log I)$).
|
|
145
|
+
- **`djson-ptrie`**: Compares the remaining index bits against the
|
|
146
|
+
node's `prefix: BITS` in chunks, descending along compressed paths
|
|
147
|
+
($O(\log I)$).
|
|
148
|
+
|
|
149
|
+
### 3.4 Subtree Cardinality (`obj_size`, `list_size`)
|
|
150
|
+
- **`djson` & `djson-btrie`**: Must recursively traverse the entire cons
|
|
151
|
+
list or bit-trie, adding 1 for each present entry using binary addition
|
|
152
|
+
(`inc_nat`). Complexity is $O(N \log N)$.
|
|
153
|
+
- **`djson-ptrie`**: Stores a precomputed binary natural number
|
|
154
|
+
`size: BITS` at every node. Size queries are simple $O(1)$ field
|
|
155
|
+
projections (`ptrie_obj? < /nil 0, /trie .size >`). In LLVM `-O3`, this
|
|
156
|
+
executes in **5 nanoseconds**, independent of document size.
|
|
157
|
+
|
|
158
|
+
### 3.5 Functional Updates (`remove`, `put`)
|
|
159
|
+
- **`djson`**: Copies the cons spine until the modified entry ($O(N)$).
|
|
160
|
+
- **`djson-btrie`**: Rebuilds the bit path down to the leaf. `put` uses
|
|
161
|
+
`str_to_bits` to construct the bit path, while `remove` clears `.val` to
|
|
162
|
+
`nil` along the path.
|
|
163
|
+
- **`djson-ptrie`**:
|
|
164
|
+
- `remove` checks `has_key?`, decrements `size` with `dec_nat` along
|
|
165
|
+
the path, and clears `.val` to `nil`.
|
|
166
|
+
- `put` uses `split_str_prefix` to find the common prefix between the
|
|
167
|
+
new key and existing node. If prefixes diverge, it splits the node
|
|
168
|
+
into a shared parent with a 2-entry branch dictionary in $O(L)$ time.
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## 4. Empirical Benchmark Results
|
|
173
|
+
|
|
174
|
+
Measured with `npm run perf:djson` on Linux x86_64 across sizes
|
|
175
|
+
$N \in \{10, 100, 500\}$.
|
|
176
|
+
|
|
177
|
+
### 4.1 Wire Representation Sizes
|
|
178
|
+
|
|
179
|
+
#### Shared-Prefix Objects (`{"field_prefix_0000": 0, ...}`)
|
|
180
|
+
- **$N = 10$** (Raw: 230 B):
|
|
181
|
+
- `djson`: 1,029 B (4.47x)
|
|
182
|
+
- `djson-btrie`: **913 B (3.97x)**
|
|
183
|
+
- `djson-ptrie`: 1,032 B (4.49x)
|
|
184
|
+
- `json` (Spec): 370 B (1.61x)
|
|
185
|
+
- **$N = 100$** (Raw: 2,750 B):
|
|
186
|
+
- `djson`: 3,365 B (1.22x)
|
|
187
|
+
- `djson-btrie`: **2,096 B (0.76x)**
|
|
188
|
+
- `djson-ptrie`: 2,367 B (0.86x)
|
|
189
|
+
- `json` (Spec): 1,529 B (0.56x)
|
|
190
|
+
- **$N = 500$** (Raw: 14,750 B):
|
|
191
|
+
- `djson`: 14,357 B (0.97x)
|
|
192
|
+
- `djson-btrie`: **7,370 B (0.50x)**
|
|
193
|
+
- `djson-ptrie`: 8,241 B (0.56x)
|
|
194
|
+
- `json` (Spec): 7,229 B (0.49x)
|
|
195
|
+
|
|
196
|
+
#### Varied-Key Objects (`{"account_0_account_attr": 0, ...}`)
|
|
197
|
+
- **$N = 10$** (Raw: 270 B):
|
|
198
|
+
- `djson`: **1,114 B (4.13x)**
|
|
199
|
+
- `djson-btrie`: 1,460 B (5.41x)
|
|
200
|
+
- `djson-ptrie`: 1,236 B (4.58x)
|
|
201
|
+
- `json` (Spec): 496 B (1.84x)
|
|
202
|
+
- **$N = 100$** (Raw: 3,120 B):
|
|
203
|
+
- `djson`: 4,987 B (1.60x)
|
|
204
|
+
- `djson-btrie`: 6,729 B (2.16x)
|
|
205
|
+
- `djson-ptrie`: **4,309 B (1.38x)**
|
|
206
|
+
- `json` (Spec): 4,460 B (1.43x)
|
|
207
|
+
- **$N = 500$** (Raw: 16,598 B):
|
|
208
|
+
- `djson`: 22,581 B (1.36x)
|
|
209
|
+
- `djson-btrie`: 28,478 B (1.72x)
|
|
210
|
+
- `djson-ptrie`: **17,036 B (1.03x)**
|
|
211
|
+
- `json` (Spec): 22,104 B (1.33x)
|
|
212
|
+
|
|
213
|
+
#### Homogeneous Number Arrays (`[0, 3, 6, ...]`)
|
|
214
|
+
- **$N = 10$** (Raw: 27 B):
|
|
215
|
+
- `djson`: 880 B (32.6x)
|
|
216
|
+
- `djson-btrie`: **878 B (32.5x)**
|
|
217
|
+
- `djson-ptrie`: 999 B (37.0x)
|
|
218
|
+
- **$N = 100$** (Raw: 363 B):
|
|
219
|
+
- `djson`: **1,645 B (4.53x)**
|
|
220
|
+
- `djson-btrie`: 1,666 B (4.59x)
|
|
221
|
+
- `djson-ptrie`: 1,877 B (5.17x)
|
|
222
|
+
- **$N = 500$** (Raw: 2,129 B):
|
|
223
|
+
- `djson`: **5,045 B (2.37x)**
|
|
224
|
+
- `djson-btrie`: 5,166 B (2.43x)
|
|
225
|
+
- `djson-ptrie`: 5,776 B (2.71x)
|
|
226
|
+
|
|
227
|
+
---
|
|
228
|
+
|
|
229
|
+
### 4.2 Execution Latency Across Backends ($N=500$)
|
|
230
|
+
|
|
231
|
+
#### Object Lookup (`dot` on last key of 500-key object)
|
|
232
|
+
- **Shared-Prefix Keys**:
|
|
233
|
+
- `djson`: Wasm $580.4\ \mu\text{s}$, LLVM $23.6\ \mu\text{s}$
|
|
234
|
+
- `djson-btrie`: Wasm $7.4\ \mu\text{s}$, LLVM **$2.4\ \mu\text{s}$**
|
|
235
|
+
- `djson-ptrie`: Wasm $11.1\ \mu\text{s}$, LLVM $5.1\ \mu\text{s}$
|
|
236
|
+
- **Varied Keys**:
|
|
237
|
+
- `djson`: Wasm $26.8\ \mu\text{s}$, LLVM $4.2\ \mu\text{s}$
|
|
238
|
+
- `djson-btrie`: Wasm $10.9\ \mu\text{s}$, LLVM **$3.1\ \mu\text{s}$**
|
|
239
|
+
- `djson-ptrie`: Wasm $10.8\ \mu\text{s}$, LLVM $4.4\ \mu\text{s}$
|
|
240
|
+
|
|
241
|
+
#### Key Membership Filter (`has_key?` miss on nonexistent key)
|
|
242
|
+
- `djson`: Wasm $63.9\ \mu\text{s}$, LLVM $1,168\ \text{ns}$
|
|
243
|
+
- `djson-btrie`: Wasm $1,022\ \text{ns}$, LLVM **$466\ \text{ns}$**
|
|
244
|
+
- `djson-ptrie`: Wasm $1,908\ \text{ns}$, LLVM $1,228\ \text{ns}$
|
|
245
|
+
|
|
246
|
+
#### Array Element Lookup (`nth` on index 499 of 500 elements)
|
|
247
|
+
- `djson`:
|
|
248
|
+
kVM $10.74\ \text{ms}$, Wasm $97.2\ \mu\text{s}$,
|
|
249
|
+
LLVM $100.7\ \mu\text{s}$
|
|
250
|
+
- `djson-btrie`:
|
|
251
|
+
kVM $26.0\ \mu\text{s}$, Wasm $733\ \text{ns}$,
|
|
252
|
+
LLVM **$114\ \text{ns}$** (**881x speedup**)
|
|
253
|
+
- `djson-ptrie`:
|
|
254
|
+
kVM $35.7\ \mu\text{s}$, Wasm $1,267\ \text{ns}$,
|
|
255
|
+
LLVM $371\ \text{ns}$ (**271x speedup**)
|
|
256
|
+
|
|
257
|
+
#### Cardinality Queries (`obj_size` for $N=500$)
|
|
258
|
+
- `djson`:
|
|
259
|
+
kVM $13.57\ \text{ms}$, Wasm $89.2\ \mu\text{s}$,
|
|
260
|
+
LLVM $127.4\ \mu\text{s}$
|
|
261
|
+
- `djson-btrie`:
|
|
262
|
+
kVM $210.88\ \text{ms}$, Wasm $3.02\ \text{ms}$,
|
|
263
|
+
LLVM $8.06\ \text{ms}$
|
|
264
|
+
- `djson-ptrie`:
|
|
265
|
+
kVM $2,054\ \text{ns}$, Wasm $172\ \text{ns}$,
|
|
266
|
+
LLVM **$5\ \text{ns}$** (**26,106x speedup**)
|
|
267
|
+
|
|
268
|
+
#### Object Key Updates ($N=500$)
|
|
269
|
+
- **`remove`**:
|
|
270
|
+
- `djson`: LLVM $79.6\ \mu\text{s}$
|
|
271
|
+
- `djson-btrie`: LLVM **$9.5\ \mu\text{s}$** (8.4x speedup)
|
|
272
|
+
- `djson-ptrie`: LLVM $13.3\ \mu\text{s}$ (6.0x speedup)
|
|
273
|
+
- **`put`**:
|
|
274
|
+
- `djson`: LLVM $50.8\ \mu\text{s}$
|
|
275
|
+
- `djson-btrie`: LLVM $12.6\ \mu\text{s}$ (4.0x speedup)
|
|
276
|
+
- `djson-ptrie`: LLVM **$3.9\ \mu\text{s}$** (13.1x speedup)
|
|
277
|
+
|
|
278
|
+
---
|
|
279
|
+
|
|
280
|
+
## 5. Trade-off Analysis & Recommendation Matrix
|
|
281
|
+
|
|
282
|
+
| Feature | `json` | `djson` | `btrie` | `ptrie` |
|
|
283
|
+
| :--- | :---: | :---: | :---: | :---: |
|
|
284
|
+
| **Universal Pattern** | ❌ No | ✅ 29 | ✅ 27 | ✅ 33 |
|
|
285
|
+
| **REPL Inspection** | Native | Cons | Bit-tree | Radix |
|
|
286
|
+
| **Shared Prefix Wire**| Small | Med | **Small** | Small |
|
|
287
|
+
| **Varied Keys Wire** | Large | Med | Large | **Small** |
|
|
288
|
+
| **Array ($N=500$)** | Direct | $100\ \mu\text{s}$ | **114ns**| 371ns |
|
|
289
|
+
| **Dot ($N=500$)** | Direct | $23.6\ \mu\text{s}$| **2.4µs**| 5.1µs |
|
|
290
|
+
| **Put ($N=500$)** | N/A | $50.8\ \mu\text{s}$| 12.6µs | **3.9µs**|
|
|
291
|
+
| **Size ($N=500$)** | N/A | $127\ \mu\text{s}$ | 8.1ms | **5ns** |
|
|
292
|
+
|
|
293
|
+
### Recommendations
|
|
294
|
+
1. **Choose `djson-ptrie` for general-purpose dynamic JSON**:
|
|
295
|
+
It achieves the best wire compression on real-world varied keys
|
|
296
|
+
(1.03x of raw JSON), provides instant $O(1)$ cardinality queries
|
|
297
|
+
(5 ns in LLVM), and delivers the fastest updates (`put` in 3.9 µs).
|
|
298
|
+
2. **Choose `djson-btrie` for extreme lookup throughput on shared keys**:
|
|
299
|
+
Destructive bit-walking in `dot` avoids all node allocation,
|
|
300
|
+
achieving the lowest lookup latency (2.4 µs in LLVM).
|
|
301
|
+
3. **Keep `djson` for simple, lightweight tooling**:
|
|
302
|
+
Simple cons-list structure is human-readable in REPL raw outputs and
|
|
303
|
+
has minimal code complexity for small payloads ($N < 50$).
|
|
304
|
+
4. **Choose `json` when document schema is fixed at compile time**:
|
|
305
|
+
Avoids all dynamic dictionary lookups in exchange for a
|
|
306
|
+
document-specialized pattern.
|