@fraczak/k 6.8.0 → 6.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -0
- package/CLAUDE.md +1 -1
- package/DOCS/BIG_PICTURE.md +3 -5
- package/DOCS/CODECS.md +80 -181
- package/DOCS/CODEC_ARCHITECTURE.md +194 -0
- package/DOCS/REPL.md +69 -51
- package/Examples/arithmetics.k +1 -0
- package/Examples/poly.k +46 -0
- package/backends/llvm/runtime/krt.c +4 -0
- package/backends/llvm/src/kvm2llvm.mjs +2 -2
- package/backends/llvm/tests/test-llvm.mjs +19 -0
- package/backends/wasm/src/kvm2wasm.mjs +4 -0
- package/backends/wasm/tests/test-wasm.mjs +4 -0
- package/browser/repl-browser.mjs +131 -779
- package/browser/shims/process.mjs +7 -0
- package/browser/tree-view.mjs +5 -2
- package/browser/vfs-data.mjs +14 -14
- package/codecs/README.md +7 -0
- package/codecs/ieee.mjs +50 -75
- package/codecs/int.mjs +83 -62
- package/codecs/json-codec.mjs +5 -77
- package/codecs/json.mjs +12 -56
- package/codecs/k-parse.mjs +3 -19
- package/codecs/runtime/codec-sdk.mjs +94 -0
- package/codecs/runtime/pattern-k.mjs +45 -0
- package/codecs/string-codec.mjs +48 -74
- package/codecs/unit.mjs +13 -61
- package/codecs/utf16.mjs +13 -56
- package/codecs/utf8.mjs +11 -55
- package/package.json +1 -1
- package/patterns/from-k.mjs +2 -22
- package/repl-codecs.mjs +69 -196
- package/repl.mjs +177 -281
- package/scripts/build-repl-html.mjs +114 -385
- package/scripts/run-tests.mjs +1 -0
- package/tests/test-codecs.mjs +186 -0
- package/tests/test-poly.mjs +44 -0
- package/tests/test-repl-html.mjs +163 -68
- package/tests/test-repl.mjs +210 -156
- package/tests/test-structural-values.mjs +17 -0
- package/tests/test-tree-view.mjs +3 -0
- package/codecs/example-pipeline.mjs +0 -163
- package/codecs/runtime/ieee-pattern.mjs +0 -30
- package/codecs/runtime/test-codec.mjs +0 -89
- package/codecs/runtime/typeFromValue.mjs +0 -34
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,46 @@ All notable changes to this project will be documented here.
|
|
|
5
5
|
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
|
6
6
|
This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [6.10.1] — 2026-10-02 — Fix Compaction Mark Reset and False Tail Calls in LLVM & WASM Backends
|
|
9
|
+
|
|
10
|
+
### LLVM & WebAssembly Backends
|
|
11
|
+
|
|
12
|
+
- **Compaction Mark Reset**: Fixed an issue in both LLVM (`krt.c`) and WebAssembly (`kvm2wasm.mjs`) where tail-loop compaction never reset the baseline arena mark after compacting live objects. Once the threshold was reached, compaction was erroneously triggered on every subsequent iteration, turning $O(N)$ execution into $O(N^2)$ and causing call-stack overflow on large inputs.
|
|
13
|
+
- **Product Branch Tail-Call Fix**: Fixed LLVM lowering in `kvm2llvm.mjs` where recursive calls inside product constructor fields were incorrectly classified as self-tail calls.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## [6.10.0] — 2026-10-02 — Dual Evaluation Engines (WASM & JS) and REPL Command Modernization
|
|
18
|
+
|
|
19
|
+
### REPL & Evaluation Engines
|
|
20
|
+
|
|
21
|
+
- **Dual Evaluation Engines**: The REPL now supports two evaluation modes:
|
|
22
|
+
- `wasm` (default): Lowers expressions via polymorphic kVM to WebAssembly AOT bytecode for near-native performance, tail-call optimization, and stack safety.
|
|
23
|
+
- `js`: Direct tree-walking JavaScript interpreter evaluating parsed relations via `run_rel` with pattern constraints for instant interpretation without compilation latency.
|
|
24
|
+
- **Engine Switching & CLI**:
|
|
25
|
+
- REPL commands `:engine [wasm|js]`, `:wasm`, and `:js` dynamically inspect or switch the active evaluation engine.
|
|
26
|
+
- CLI options `--engine=wasm`, `--engine=js`, `--wasm`, and `--js` added to `k-repl`.
|
|
27
|
+
- Active engine displayed in execution timing annotations (`/* comp: Xms, exec: Yms (wasm) */` or `(js)`).
|
|
28
|
+
- **Web REPL (`repl.html`) Integration**:
|
|
29
|
+
- Added clickable navbar engine badge (`wasm-in-process` <-> `js-in-process`) for one-click engine toggling in the browser.
|
|
30
|
+
- Exported `getEngine()` and `setEngine()` on `window.kRepl`.
|
|
31
|
+
- **Command Modernization**:
|
|
32
|
+
- Removed deprecated REPL commands: `:run expr`, `:eval expr`, `:time expr`, and `:t name`.
|
|
33
|
+
- Renamed `:codes` to `:types` (listing type aliases).
|
|
34
|
+
- Renamed `:d name` to `:rel name` (displaying relation definitions).
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## [6.9.0] — 2026-10-01 — Reusable Codecs, Dynamic Pattern Derivation & Unified SDK
|
|
39
|
+
|
|
40
|
+
### Codecs & Type System
|
|
41
|
+
|
|
42
|
+
- **Dynamic Pattern Derivation**: All codecs derive their closed pattern graphs through `k`'s type system from filter and type expressions via `patternFromFilter`. Handwritten property-list JSON arrays have been completely eliminated from codecs.
|
|
43
|
+
- **Unified Codec SDK**: Added `codecs/runtime/codec-sdk.mjs` exporting a standard CLI harness (`runCodecCLI`), `Value` API, wire codecs, and `patternFromFilter`.
|
|
44
|
+
- **Recipe-Based REPL Codecs**: Refactored the REPL to use explicit recipe-driven input syntax (`:input <codec.mjs> [text]`) and multi-codec output formatting. Removed universal catch-all fallback and ungrounded types.
|
|
45
|
+
- **Self-Contained Web REPL VFS**: The browser REPL compiler (`scripts/build-repl-html.mjs`) bundles verified codecs into zero-import ES modules for the virtual file system, delegating `patternFromFilter` dynamically to the browser's bundled `k` engine.
|
|
46
|
+
- **Dead Code Clean-Up**: Removed dead files (`ieee-pattern.mjs`, `test-codec.mjs`, `typeFromValue.mjs`, `example-pipeline.mjs`).
|
|
47
|
+
|
|
8
48
|
---
|
|
9
49
|
|
|
10
50
|
## [6.5.0] — 2026-09-22 — Tail-loop arena compaction & OOM diagnostics
|
package/CLAUDE.md
CHANGED
|
@@ -61,4 +61,4 @@ npm test # Run comprehensive test suite (unit tests, derivation tests, shell te
|
|
|
61
61
|
- No external runtime dependencies (only jiwson for parser generation)
|
|
62
62
|
- Uses ES6 modules (.mjs files) throughout
|
|
63
63
|
- Extensive test suite in `tests/code-derivation/` for type system validation
|
|
64
|
-
- REPL supports commands like `:
|
|
64
|
+
- REPL supports commands like `:rel name` (show relation definition), `:type name` (show type definition), `:types`, `:rels`, etc.
|
package/DOCS/BIG_PICTURE.md
CHANGED
|
@@ -252,13 +252,11 @@ car = list? /cons .car;
|
|
|
252
252
|
cdr = list? /cons .cdr;
|
|
253
253
|
```
|
|
254
254
|
|
|
255
|
-
For example, the
|
|
255
|
+
For example, the definition for `car` is:
|
|
256
256
|
|
|
257
257
|
```text
|
|
258
|
-
> :
|
|
259
|
-
car
|
|
260
|
-
> :C KL
|
|
261
|
-
$ KL = {}; -- @...
|
|
258
|
+
> :rel car
|
|
259
|
+
car = list? /cons .car; -- @...
|
|
262
260
|
```
|
|
263
261
|
|
|
264
262
|
## Universal Schema Registry
|
package/DOCS/CODECS.md
CHANGED
|
@@ -1,113 +1,86 @@
|
|
|
1
1
|
# Writing Codecs
|
|
2
2
|
|
|
3
|
-
A codec is
|
|
4
|
-
and k `Value` objects. The same module may serve two entry points:
|
|
3
|
+
A codec is an ES module that translates between an external domain format and enveloped **k** `Value` objects `(P, v)`. The same module serves two entry points:
|
|
5
4
|
|
|
6
|
-
- a command-line
|
|
7
|
-
- a REPL
|
|
5
|
+
- a standalone command-line filter supporting `--parse`, `--print`, and `--help`,
|
|
6
|
+
- a REPL module loaded with `:codec load <file.mjs>`.
|
|
8
7
|
|
|
9
8
|
The command-line entry point reads or writes the binary pattern+value stream.
|
|
10
|
-
The REPL entry point
|
|
11
|
-
functions.
|
|
9
|
+
The REPL entry point provides interactive parsing (`:input <codec.mjs>`) and multi-codec output formatting.
|
|
12
10
|
|
|
13
|
-
##
|
|
11
|
+
## Architecture & Mental Model
|
|
14
12
|
|
|
15
|
-
|
|
16
|
-
module exports either `codes` or `patterns`.
|
|
13
|
+
A codec maps between external text and enveloped *k* values:
|
|
17
14
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
["closed-product", []]
|
|
22
|
-
]];
|
|
23
|
-
```
|
|
15
|
+
- `parse(text)` translates external text to an enveloped *k* value (a `Value` with an associated closed pattern envelope `(P, v)`).
|
|
16
|
+
- `print(value)` translates an enveloped *k* value into formatted external text, throwing an `Error` if the value is not representable.
|
|
17
|
+
- `doc` (optional string) documents the accepted syntax and semantic representation.
|
|
24
18
|
|
|
25
|
-
|
|
26
|
-
universal codec is available for any type named by `:input`; the REPL validates
|
|
27
|
-
the parsed value against that requested type before accepting it.
|
|
19
|
+
Codecs are identified directly by their file name. They do not export artificial identifiers like `name` or pattern registries like `patterns = [...]`.
|
|
28
20
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
21
|
+
In the REPL:
|
|
22
|
+
- **Input**: The user explicitly chooses which codec to parse with:
|
|
23
|
+
- Interactive mode: `:input <codec.mjs>` sets the prompt to `<codec.mjs>> `, and the next entered line is parsed with that codec.
|
|
24
|
+
- One-line mode: `:input <codec.mjs> <text>` immediately parses `<text>` using that codec.
|
|
25
|
+
- **Output**: Any evaluated *k* value is displayed in its standard *k* envelope representation, followed by formatting from **all loaded codecs** that can represent it:
|
|
26
|
+
```text
|
|
27
|
+
> 42 int
|
|
28
|
+
{}|_|0|1|0|1|0|1|+ ?<{} _, ...>
|
|
29
|
+
int.mjs: 42
|
|
30
|
+
```
|
|
31
|
+
Every loaded codec attempts `codec.print(value)`. If it succeeds, the REPL prints `<codec.mjs>: <formatted>`. If it throws, the codec is silently ignored for that value.
|
|
33
32
|
|
|
34
|
-
|
|
35
|
-
recomputes the canonical code hash. Use `codes` only when the codec is tied to a
|
|
36
|
-
known canonical hash:
|
|
37
|
-
|
|
38
|
-
```js
|
|
39
|
-
export const codes = ["@..."];
|
|
40
|
-
```
|
|
33
|
+
## Module Interface
|
|
41
34
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
A REPL codec exports:
|
|
35
|
+
A codec module exports:
|
|
45
36
|
|
|
46
37
|
```js
|
|
47
|
-
export const
|
|
48
|
-
export const patterns = [MY_PATTERN]; // or codes = ["@..."], or universal = true
|
|
38
|
+
export const doc = "Description of syntax and usage.";
|
|
49
39
|
|
|
50
|
-
export function parse(text
|
|
51
|
-
// text
|
|
52
|
-
// return a k Value
|
|
40
|
+
export function parse(text) {
|
|
41
|
+
// parse text into an enveloped k Value (P, v) using withPattern(value, pattern)
|
|
53
42
|
}
|
|
54
43
|
|
|
55
|
-
export function print(value
|
|
56
|
-
// return text
|
|
44
|
+
export function print(value) {
|
|
45
|
+
// return formatted text
|
|
46
|
+
// throw Error if value is not representable
|
|
57
47
|
}
|
|
58
48
|
```
|
|
59
49
|
|
|
60
|
-
`
|
|
61
|
-
|
|
62
|
-
- `codeHash`: the canonical code hash selected by `:input`,
|
|
63
|
-
- `pattern`: the closed property-list pattern for that type,
|
|
64
|
-
- `state`: the current REPL state.
|
|
65
|
-
|
|
66
|
-
Most codecs do not need `context`. Universal codecs can use it when the same
|
|
67
|
-
text syntax needs to adapt to the requested type.
|
|
68
|
-
|
|
69
|
-
`parse` should throw an `Error` for invalid external text. `print` should throw
|
|
70
|
-
when the value is not representable by the external format. For universal
|
|
71
|
-
codecs, REPL output suppresses `print` errors so a generic codec does not add
|
|
72
|
-
noise to every value.
|
|
50
|
+
`parse` should throw an `Error` for invalid external text. `print` should throw when the value is not representable by the external format. REPL output suppresses `print` errors so loaded codecs only output when they recognize the value.
|
|
73
51
|
|
|
74
|
-
##
|
|
52
|
+
## The Codec SDK & Runner
|
|
75
53
|
|
|
76
|
-
|
|
54
|
+
To eliminate CLI stream boilerplate, codecs import from `./runtime/codec-sdk.mjs` and invoke `runCodecCLI`:
|
|
77
55
|
|
|
78
56
|
```js
|
|
79
|
-
import { Value,
|
|
80
|
-
|
|
81
|
-
const unit = Value.product({});
|
|
82
|
-
const yes = Value.variant("true", unit);
|
|
83
|
-
const point = Value.product({
|
|
84
|
-
x: Value.variant("+", unit),
|
|
85
|
-
y: Value.variant("-", unit)
|
|
86
|
-
});
|
|
57
|
+
import { Value, isVariant, withPattern, runCodecCLI } from "./runtime/codec-sdk.mjs";
|
|
87
58
|
```
|
|
88
59
|
|
|
89
|
-
|
|
90
|
-
|
|
60
|
+
`runCodecCLI(import.meta.url, { parse, print, doc })` automatically handles:
|
|
61
|
+
1. Entrypoint detection (no-op when loaded as a module or in browsers).
|
|
62
|
+
2. `-h` and `--help` CLI flag handling.
|
|
63
|
+
3. `--parse` and `--print` stream piping and wire encoding/decoding.
|
|
91
64
|
|
|
92
|
-
## Minimal Type-Specific Codec
|
|
65
|
+
## Minimal Type-Specific Codec Example
|
|
93
66
|
|
|
94
|
-
This codec accepts `yes` and `no` for the type `<{}
|
|
67
|
+
This codec accepts `yes` and `no` for the type `<{} false, {} true>`. Patterns are derived by `k` from filter expressions via `patternFromFilter`:
|
|
95
68
|
|
|
96
69
|
```js
|
|
97
|
-
|
|
70
|
+
#!/usr/bin/env node
|
|
71
|
+
|
|
72
|
+
import { Value, isVariant, withPattern, patternFromFilter, runCodecCLI } from "./runtime/codec-sdk.mjs";
|
|
98
73
|
|
|
99
|
-
const BOOL_PATTERN =
|
|
100
|
-
["closed-union", [["false", 1], ["true", 1]]],
|
|
101
|
-
["closed-product", []]
|
|
102
|
-
];
|
|
74
|
+
const BOOL_PATTERN = patternFromFilter("?< {} false, {} true >");
|
|
103
75
|
|
|
104
|
-
export const
|
|
105
|
-
|
|
76
|
+
export const doc = `
|
|
77
|
+
Boolean codec: accepts 'yes' and 'no'.
|
|
78
|
+
`;
|
|
106
79
|
|
|
107
80
|
export function parse(text) {
|
|
108
81
|
const word = text.trim();
|
|
109
|
-
if (word === "yes") return Value.variant("true", Value.product({}));
|
|
110
|
-
if (word === "no") return Value.variant("false", Value.product({}));
|
|
82
|
+
if (word === "yes") return withPattern(Value.variant("true", Value.product({})), BOOL_PATTERN);
|
|
83
|
+
if (word === "no") return withPattern(Value.variant("false", Value.product({})), BOOL_PATTERN);
|
|
111
84
|
throw new Error("expected yes or no");
|
|
112
85
|
}
|
|
113
86
|
|
|
@@ -117,106 +90,30 @@ export function print(value) {
|
|
|
117
90
|
if (value.tag === "false") return "no";
|
|
118
91
|
throw new Error("expected true or false");
|
|
119
92
|
}
|
|
93
|
+
|
|
94
|
+
runCodecCLI(import.meta.url, { parse, print, doc });
|
|
120
95
|
```
|
|
121
96
|
|
|
122
|
-
|
|
97
|
+
### Using in the REPL:
|
|
123
98
|
|
|
124
99
|
```text
|
|
125
100
|
> :codec load ./codecs/yesno.mjs
|
|
126
|
-
loaded codec yesno
|
|
127
|
-
> :input
|
|
128
|
-
input @... using yesno: enter value text
|
|
129
|
-
yesno> yes
|
|
101
|
+
loaded codec yesno.mjs
|
|
102
|
+
> :input yesno.mjs yes
|
|
130
103
|
{}|true ?<{} false, {} true>
|
|
131
|
-
yesno: yes
|
|
132
|
-
```
|
|
133
|
-
|
|
134
|
-
## Add The CLI Boundary
|
|
135
|
-
|
|
136
|
-
To make the same module usable as an installed command, add a `main` function
|
|
137
|
-
that supports `--parse`, `--print`, and `--help`.
|
|
138
|
-
|
|
139
|
-
```js
|
|
140
|
-
#!/usr/bin/env node
|
|
141
|
-
|
|
142
|
-
import { stdin, stdout, argv, exit } from "node:process";
|
|
143
|
-
import { decodeWire, encodeToWire } from "./runtime/prefix-codec.mjs";
|
|
144
|
-
import { isMainEntrypoint } from "./runtime/cli-entry.mjs";
|
|
145
|
-
|
|
146
|
-
function usage(stream = console.error) {
|
|
147
|
-
stream(`Usage: ${argv[1]} --parse | --print`);
|
|
148
|
-
stream(" --parse Read yes/no text, write binary pattern+value stream.");
|
|
149
|
-
stream(" --print Read binary pattern+value stream, write yes/no text.");
|
|
150
|
-
stream(" -h, --help Show this help.");
|
|
151
|
-
}
|
|
152
|
-
|
|
153
|
-
function readAll(stream) {
|
|
154
|
-
return new Promise((resolve, reject) => {
|
|
155
|
-
const chunks = [];
|
|
156
|
-
stream.on("data", c => chunks.push(Buffer.isBuffer(c) ? c : Buffer.from(c)));
|
|
157
|
-
stream.on("end", () => resolve(Buffer.concat(chunks)));
|
|
158
|
-
stream.on("error", reject);
|
|
159
|
-
});
|
|
160
|
-
}
|
|
161
|
-
|
|
162
|
-
async function main() {
|
|
163
|
-
const args = argv.slice(2);
|
|
164
|
-
if (args.includes("-h") || args.includes("--help")) {
|
|
165
|
-
usage(console.log);
|
|
166
|
-
exit(0);
|
|
167
|
-
}
|
|
168
|
-
if (args.length !== 1 || (args[0] !== "--parse" && args[0] !== "--print")) {
|
|
169
|
-
usage();
|
|
170
|
-
exit(1);
|
|
171
|
-
}
|
|
172
|
-
|
|
173
|
-
const input = await readAll(stdin);
|
|
174
|
-
if (args[0] === "--parse") {
|
|
175
|
-
stdout.write(encodeToWire(parse(input.toString("utf8")), BOOL_PATTERN));
|
|
176
|
-
} else {
|
|
177
|
-
stdout.write(`${print(decodeWire(input).value)}\n`);
|
|
178
|
-
}
|
|
179
|
-
}
|
|
180
|
-
|
|
181
|
-
if (isMainEntrypoint(import.meta.url, argv[1])) {
|
|
182
|
-
main().catch(error => {
|
|
183
|
-
console.error(error.message || String(error));
|
|
184
|
-
exit(1);
|
|
185
|
-
});
|
|
186
|
-
}
|
|
104
|
+
yesno.mjs: yes
|
|
187
105
|
```
|
|
188
106
|
|
|
189
|
-
|
|
190
|
-
`--parse` mode. Diagnostics belong on stderr.
|
|
191
|
-
|
|
192
|
-
Installed codec binaries use the `k-` prefix plus the source basename without
|
|
193
|
-
`.mjs`. For example, `codecs/yesno.mjs` installs as `k-yesno` when added to
|
|
194
|
-
`package.json`.
|
|
107
|
+
### Using as a CLI Pipeline Filter:
|
|
195
108
|
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
```js
|
|
201
|
-
export const name = "json";
|
|
202
|
-
export const universal = true;
|
|
203
|
-
|
|
204
|
-
export function parse(text) {
|
|
205
|
-
// Convert external text to a Value.
|
|
206
|
-
}
|
|
207
|
-
|
|
208
|
-
export function print(value) {
|
|
209
|
-
// Convert a Value to external text.
|
|
210
|
-
}
|
|
109
|
+
```sh
|
|
110
|
+
printf 'yes\n' | ./codecs/yesno.mjs --parse | ./codecs/yesno.mjs --print
|
|
111
|
+
# Output: yes
|
|
211
112
|
```
|
|
212
113
|
|
|
213
|
-
|
|
214
|
-
example. It derives a pattern for command-line `--parse`, but in the REPL the
|
|
215
|
-
requested `:input` type supplies the target type.
|
|
216
|
-
|
|
217
|
-
## Test A Codec
|
|
114
|
+
## Testing a Codec
|
|
218
115
|
|
|
219
|
-
For CLI codecs, test both directions:
|
|
116
|
+
For CLI codecs, test both directions in a pipeline:
|
|
220
117
|
|
|
221
118
|
```sh
|
|
222
119
|
printf 'yes\n' | node codecs/yesno.mjs --parse | node codecs/yesno.mjs --print
|
|
@@ -227,17 +124,13 @@ For REPL codecs, add a focused case to [`../tests/test-repl.mjs`](../tests/test-
|
|
|
227
124
|
```js
|
|
228
125
|
const state = createState();
|
|
229
126
|
let output = await evaluateInput(":codec load ./codecs/yesno.mjs", state);
|
|
230
|
-
assert.match(output[0], /^loaded codec yesno
|
|
231
|
-
output = await evaluateInput(":input
|
|
232
|
-
assert.
|
|
127
|
+
assert.match(output[0], /^loaded codec yesno\.mjs/);
|
|
128
|
+
output = await evaluateInput(":input yesno.mjs", state);
|
|
129
|
+
assert.equal(promptForState(state), "yesno.mjs> ");
|
|
233
130
|
output = await evaluateInput("yes", state);
|
|
234
|
-
assert.match(output[0], /yesno: yes/);
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
Run the targeted test first:
|
|
238
|
-
|
|
239
|
-
```sh
|
|
240
|
-
node tests/test-repl.mjs
|
|
131
|
+
assert.match(output[0], /yesno\.mjs: yes/);
|
|
132
|
+
output = await evaluateInput(":input yesno.mjs no", state);
|
|
133
|
+
assert.match(output[0], /yesno\.mjs: no/);
|
|
241
134
|
```
|
|
242
135
|
|
|
243
136
|
Then run the full suite before committing:
|
|
@@ -246,13 +139,19 @@ Then run the full suite before committing:
|
|
|
246
139
|
npm test
|
|
247
140
|
```
|
|
248
141
|
|
|
142
|
+
## Web REPL (VFS) Bundling
|
|
143
|
+
|
|
144
|
+
When `repl.html` is generated (`npm run build:repl-html`):
|
|
145
|
+
- Codecs (`int.mjs`, `json.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`) are bundled into 100% self-contained ES modules with zero imports.
|
|
146
|
+
- `runCodecCLI` is shimmed as a no-op, and `Value` helpers are directly inlined.
|
|
147
|
+
- In the browser, the VFS serves these self-contained modules directly to native dynamic `import(blobUrl)`. No runtime helper files (`codecs/runtime/*`) need to be included in the VFS or exposed as globals.
|
|
148
|
+
|
|
249
149
|
## Checklist
|
|
250
150
|
|
|
251
|
-
-
|
|
252
|
-
-
|
|
253
|
-
-
|
|
254
|
-
-
|
|
255
|
-
-
|
|
256
|
-
-
|
|
257
|
-
|
|
258
|
-
- Add REPL coverage for `:codec load` and `:input`.
|
|
151
|
+
- Return an enveloped `Value` from `parse` using `withPattern(val, pattern)`.
|
|
152
|
+
- Keep `parse` and `print` deterministic and free of side effects.
|
|
153
|
+
- Validate shapes in `print` and throw clear errors when not representable.
|
|
154
|
+
- Use `runCodecCLI(import.meta.url, { parse, print, doc })` for CLI execution.
|
|
155
|
+
- No artificial `name` or `patterns` exports required.
|
|
156
|
+
- Add `-h` and `--help` support via `runCodecCLI`.
|
|
157
|
+
|
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
# Reusable Codec Architecture
|
|
2
|
+
|
|
3
|
+
## 1. Overview & Objectives
|
|
4
|
+
|
|
5
|
+
In **k**, a **codec** is a two-way translator between external domain data (e.g. decimal integers, JSON, UTF-8 text, IEEE 754 floats) and **k** enveloped values (a structural `Value` tree paired with a closed pattern graph).
|
|
6
|
+
|
|
7
|
+
Every codec in the **k** ecosystem is designed to serve three unified objectives from a single module definition:
|
|
8
|
+
|
|
9
|
+
1. **Document the Codec**: Formally specify the external syntax and its semantics as an enveloped **k** value `(P, v)`.
|
|
10
|
+
2. **REPL Module**: Provide interactive input parsing (`:input <codec.mjs>`) and multi-codec output inspection (`<codec.mjs>: <formatted>`) in both Node.js and browser REPLs.
|
|
11
|
+
3. **Standalone Shell Executable**: Function as a first-class Unix filter (`--parse`, `--print`, `--help`) for composition in command-line pipelines.
|
|
12
|
+
|
|
13
|
+
```text
|
|
14
|
+
+---------------------------------------+
|
|
15
|
+
| External Domain Data |
|
|
16
|
+
| (JSON, Decimals, UTF-8, IEEE floats) |
|
|
17
|
+
+---------------------------------------+
|
|
18
|
+
| ^
|
|
19
|
+
parse(text) | | print(value)
|
|
20
|
+
v |
|
|
21
|
+
+---------------------------------------+
|
|
22
|
+
| k Codec |
|
|
23
|
+
| (Syntax & Value Mapper) |
|
|
24
|
+
+---------------------------------------+
|
|
25
|
+
| ^
|
|
26
|
+
CLI: --parse | | CLI: --print
|
|
27
|
+
REPL: :input v | REPL: output
|
|
28
|
+
+---------------------------------------+
|
|
29
|
+
| Enveloped k Value |
|
|
30
|
+
| (pattern graph P, value tree v) |
|
|
31
|
+
| Binary Wire Stream (P, v) |
|
|
32
|
+
+---------------------------------------+
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## 2. Objective 1: Documentation & Specification
|
|
38
|
+
|
|
39
|
+
Every codec defines what external text it accepts and how that text maps to an enveloped **k** value.
|
|
40
|
+
|
|
41
|
+
### 2.1 External Syntax
|
|
42
|
+
The syntax specification defines the valid textual forms accepted by the codec:
|
|
43
|
+
- **Grammar & Rules**: Defines the accepted language. While regular expressions offer simple examples, the syntax can be an arbitrary formal language (context-free, recursive, JSON RFC 8259, etc.).
|
|
44
|
+
- **Normalization**: Rules for whitespace, casing, or escaping.
|
|
45
|
+
- **Rejection Criteria**: Explicit errors when invalid input is encountered.
|
|
46
|
+
|
|
47
|
+
### 2.2 Semantics as an Enveloped k Value
|
|
48
|
+
Parsing external text produces an **enveloped value** `(P, v)`:
|
|
49
|
+
- **Pattern Graph ($P$)**: A closed pattern graph specifying the algebraic structure.
|
|
50
|
+
- **Value Tree ($v$)**: A concrete tree matching $P$, constructed from products (`{...}`) and variants (`<...>`).
|
|
51
|
+
|
|
52
|
+
A codec does not produce an ungrounded type; it performs a concrete transformation from text to an enveloped value `(P, v)`.
|
|
53
|
+
|
|
54
|
+
### 2.3 Pattern Derivation via k Type System (`patternFromFilter`)
|
|
55
|
+
Pattern graphs ($P$) must never be handwritten as raw property-list JSON arrays (e.g. `[["closed-union", ...]]`). Instead, codecs derive pattern graphs directly through the **k** type system from filter or type expressions using `patternFromFilter`:
|
|
56
|
+
|
|
57
|
+
```js
|
|
58
|
+
import { patternFromFilter } from "./runtime/codec-sdk.mjs";
|
|
59
|
+
|
|
60
|
+
// Derived from k filter expression
|
|
61
|
+
const INT_PATTERN = patternFromFilter('?< <bits 0, bits 1, {} _>=bits "+", bits "-">');
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
This guarantees:
|
|
65
|
+
1. All patterns are verified, normalized, and canonicalized by **k**'s type derivation and graph unification engine.
|
|
66
|
+
2. Codecs remain concise, declarative, and maintainable.
|
|
67
|
+
3. Codecs run identically in Node.js and in the Web REPL (via the browser runtime shim).
|
|
68
|
+
|
|
69
|
+
### 2.4 Documentation Contract
|
|
70
|
+
A codec is identified solely by the file loaded—there is no artificial `name` export or `patterns` list export. A codec may optionally export a `doc` string:
|
|
71
|
+
- Describes accepted syntax, options, and representation.
|
|
72
|
+
- Displayed by `--help` in shell invocations.
|
|
73
|
+
- Displayed by `:help <codec.mjs>` or documentation commands in REPLs.
|
|
74
|
+
|
|
75
|
+
---
|
|
76
|
+
|
|
77
|
+
## 3. Objective 2: REPL Module Interface
|
|
78
|
+
|
|
79
|
+
In the REPL, codecs are loaded dynamically on demand:
|
|
80
|
+
```text
|
|
81
|
+
> :codec load int.mjs
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### 3.1 Module Exports
|
|
85
|
+
A codec module exports:
|
|
86
|
+
|
|
87
|
+
| Export | Type | Required | Description |
|
|
88
|
+
| :--- | :--- | :--- | :--- |
|
|
89
|
+
| `parse` | `(text: string) => EnvelopedValue \| Value` | **Yes** | Parses external text into an enveloped `Value`. Throws on syntax error. |
|
|
90
|
+
| `print` | `(value: Value) => string` | **Yes** | Formats a `Value` into external text. Throws if value is not representable. |
|
|
91
|
+
| `doc` | `string` | Optional | Documentation of syntax and usage for CLI help and REPL inspections. |
|
|
92
|
+
|
|
93
|
+
### 3.2 REPL Input Protocol
|
|
94
|
+
The user specifies the codec to parse an input payload:
|
|
95
|
+
- **Interactive Mode**:
|
|
96
|
+
```text
|
|
97
|
+
> :input int.mjs
|
|
98
|
+
int.mjs> 42
|
|
99
|
+
{}|_|0|1|0|1|0|1|+ ?<{} _, ...>
|
|
100
|
+
int.mjs: 42
|
|
101
|
+
```
|
|
102
|
+
- **One-Line Mode**:
|
|
103
|
+
```text
|
|
104
|
+
> :input int.mjs 42
|
|
105
|
+
{}|_|0|1|0|1|0|1|+ ?<{} _, ...>
|
|
106
|
+
int.mjs: 42
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
### 3.3 REPL Output Protocol (Multi-Codec Inspection)
|
|
110
|
+
Whenever an evaluation produces a resulting **k** `Value`:
|
|
111
|
+
1. The primary representation is the standard **k** value envelope.
|
|
112
|
+
2. The REPL iterates over loaded codecs and calls `codec.print(value)`.
|
|
113
|
+
3. If `codec.print(value)` succeeds, the REPL displays `<codec>: <formattedString>`.
|
|
114
|
+
4. If `codec.print(value)` throws, the codec is silently ignored for that value.
|
|
115
|
+
|
|
116
|
+
---
|
|
117
|
+
|
|
118
|
+
## 4. Objective 3: Standalone Shell Executable Contract
|
|
119
|
+
|
|
120
|
+
Every codec file is directly executable from the command line:
|
|
121
|
+
```bash
|
|
122
|
+
chmod +x codecs/int.mjs
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
### 4.1 CLI Invocations
|
|
126
|
+
|
|
127
|
+
1. **Parse Mode (`--parse`)**:
|
|
128
|
+
- Reads external text from `stdin`.
|
|
129
|
+
- Parses it into an enveloped value `(P, v)`.
|
|
130
|
+
- Writes the binary wire encoding of `(P, v)` to `stdout`.
|
|
131
|
+
- Diagnostics and errors are written to `stderr`.
|
|
132
|
+
- Exit code: `0` on success, non-zero on error.
|
|
133
|
+
|
|
134
|
+
2. **Print Mode (`--print`)**:
|
|
135
|
+
- Reads binary wire bytes `(P, v)` from `stdin`.
|
|
136
|
+
- Decodes the wire stream and calls `print(value)`.
|
|
137
|
+
- Writes formatted text to `stdout`.
|
|
138
|
+
- Exit code: `0` on success, non-zero if not representable under this codec.
|
|
139
|
+
|
|
140
|
+
3. **Help Mode (`-h`, `--help`)**:
|
|
141
|
+
- Outputs CLI usage and the codec's `doc` string to `stdout`.
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
## 5. Implementation: Codec CLI Runner
|
|
146
|
+
|
|
147
|
+
To avoid duplicating stdin/stdout stream handling and CLI flag parsing across codecs, a helper is provided:
|
|
148
|
+
|
|
149
|
+
```javascript
|
|
150
|
+
runCodecCLI(import.meta.url, { parse, print, doc });
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
### Responsibilities of `runCodecCLI`:
|
|
154
|
+
1. **Entrypoint Detection**: Checks if the module is invoked directly from the CLI (e.g. `import.meta.url === pathToFileURL(process.argv[1]).href`). If imported as a module or running in a browser, it is a no-op.
|
|
155
|
+
2. **Flag Parsing**: Handles `--parse`, `--print`, and `-h` / `--help`.
|
|
156
|
+
3. **Stream Piping**:
|
|
157
|
+
- For `--parse`: reads `stdin` to string, calls `parse(text)`, encodes enveloped value `(P, v)` to binary wire format on `stdout`.
|
|
158
|
+
- For `--print`: decodes wire format from `stdin` into `(P, v)`, calls `print(value)`, writes formatted text to `stdout`.
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
## 6. Directory Layout
|
|
163
|
+
|
|
164
|
+
To make `codecs/` intuitive and self-contained:
|
|
165
|
+
|
|
166
|
+
```text
|
|
167
|
+
codecs/
|
|
168
|
+
├── int.mjs # Integer & list codec
|
|
169
|
+
├── json.mjs # JSON codec
|
|
170
|
+
├── utf8.mjs # UTF-8 string codec
|
|
171
|
+
├── utf16.mjs # UTF-16 string codec
|
|
172
|
+
├── ieee.mjs # IEEE 754 float codec
|
|
173
|
+
├── unit.mjs # Unit {} codec
|
|
174
|
+
├── k-parse.mjs # Textual k value {...} -> binary wire stream
|
|
175
|
+
├── k-print.mjs # Binary wire stream -> textual k value
|
|
176
|
+
├── show.mjs # Wire stream pass-through inspector
|
|
177
|
+
└── runtime/ # Shared codec infrastructure
|
|
178
|
+
├── codec-sdk.mjs # runCodecCLI, Value, encodeToWire, decodeWire
|
|
179
|
+
├── prefix-codec.mjs # Low-level bitstream wire serialization
|
|
180
|
+
├── codec.mjs # Abstract pattern graph algorithms
|
|
181
|
+
└── cli-entry.mjs # Entrypoint detection
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Each codec file directly under `codecs/` is both an executable CLI script and an importable module.
|
|
185
|
+
|
|
186
|
+
---
|
|
187
|
+
|
|
188
|
+
## 7. Web REPL & VFS Bundling Architecture
|
|
189
|
+
|
|
190
|
+
In the browser REPL (`repl.html`):
|
|
191
|
+
- **Self-Contained Modules**: At build time (`scripts/build-repl-html.mjs`), verified codecs in `codecs/` are compiled via `esbuild` into 100% self-contained ES modules with **zero imports**.
|
|
192
|
+
- **No Runtime Files in VFS**: `codecs/runtime/*` is not included in the browser VFS. The VFS only contains verified, self-contained user-facing files (`int.mjs`, `json.mjs`, `utf8.mjs`, `unit.mjs`, `ieee.mjs`).
|
|
193
|
+
- **Native Browser Loading**: In `repl-codecs.mjs`, the browser reads the codec source from VFS, wraps it in a Blob, and invokes native `await import(blobUrl)`. No regex manipulation, import stripping, or global environment injection is required.
|
|
194
|
+
|