@fraczak/k 6.0.1 → 6.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +35 -0
- package/CHANGELOG.md +48 -0
- package/CLAUDE.md +64 -0
- package/DOCS/BIG_PICTURE.md +4 -8
- package/DOCS/BINARY_CODEC_STATUS_AND_TODO.md +74 -0
- package/DOCS/CONVERGENCE.md +92 -0
- package/DOCS/DIVERGENCE_DETECTION.md +24 -0
- package/DOCS/Makefile +99 -0
- package/DOCS/OBJECT_FILE_AND_PATTERN.md +217 -0
- package/DOCS/PATTERNS.md +114 -0
- package/DOCS/PATTERNS_AND_TYPED_VALUES.md +665 -0
- package/DOCS/REPL.md +125 -125
- package/DOCS/TEXTUAL_VALUES.md +108 -0
- package/DOCS/TYPE_DERIVATION.md +230 -0
- package/DOCS/book/06-values-in-memory.md +1085 -0
- package/DOCS/book/07-the-partial-function-abi.md +143 -0
- package/DOCS/book/08-operational-semantics-and-execution.md +201 -0
- package/DOCS/book/09-compiler-architecture.md +160 -0
- package/DOCS/book/10-from-ast-to-intermediate-representation.md +167 -0
- package/DOCS/book/14-canonical-serialization.md +176 -0
- package/DOCS/book/appendix-b-incorporating-external-predefined-types-and-functions.md +155 -0
- package/DOCS/book/overview.txt +68 -0
- package/README.md +71 -27
- package/codecs/README.md +8 -0
- package/codecs/pandoc.css +71 -0
- package/codecs/runtime/test-codec.mjs +89 -0
- package/package.json +2 -2
- package/{codecs/k-pattern.mjs → patterns/from-k.mjs} +3 -3
- package/repl2.mjs +259 -50
- package/scripts/generate-dependency-graph.mjs +121 -0
- package/test-repl2.mjs +55 -8
- package/DOCS/book-combined.html +0 -3798
- package/DOCS/book-combined.md +0 -3745
- package/DOCS/book-combined.pdf +0 -0
- package/DOCS/book-combined.tex +0 -5682
- package/DOCS/book.html +0 -65
- package/DOCS/book.pdf +0 -0
- package/DOCS/book.tex +0 -150
package/AGENTS.md
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Repository Guidelines
|
|
2
|
+
|
|
3
|
+
## Project Structure & Module Organization
|
|
4
|
+
- Root `.mjs` files hold the core runtime and library entry points (e.g., `index.mjs`, `run.mjs`, `codes.mjs`, `typing.mjs`).
|
|
5
|
+
- `k_compiler/` contains the Python prototype compiler and supporting modules.
|
|
6
|
+
- `type_registry/` houses registry experiments and grammar artifacts.
|
|
7
|
+
- `Examples/` and `DOCS/` provide sample programs and design notes.
|
|
8
|
+
- Tests live in top-level `test*.mjs`, `Code-derivation-tests/`, and `tests.sh`.
|
|
9
|
+
|
|
10
|
+
## Build, Test, and Development Commands
|
|
11
|
+
- `npm install` installs dev dependencies and runs `prepare` to generate parsers.
|
|
12
|
+
- `npm run prepare` regenerates `parser.mjs` and `valueParser.mjs` from `.jison` grammars.
|
|
13
|
+
- `npm test` runs the full suite (`test.mjs`, `Code-derivation-tests/*.mjs`, `test-fingerprint.mjs`, `tests.sh`).
|
|
14
|
+
- `node repl2.mjs` (or `k-repl`) starts the interactive REPL.
|
|
15
|
+
- `./k.mjs <file.k>` executes a k script on binary pattern+value stream input from stdin or a file; use `k-parse` and `k-print` at the boundaries.
|
|
16
|
+
|
|
17
|
+
## Coding Style & Naming Conventions
|
|
18
|
+
- JavaScript uses ES modules (`.mjs`) with 2-space indentation and semicolons.
|
|
19
|
+
- Prefer `const`/`let`, small focused modules, and descriptive lower-case filenames.
|
|
20
|
+
- Python modules in `k_compiler/` follow 4-space indentation and standard snake_case.
|
|
21
|
+
- Grammar sources are `parser.jison` and `valueParser.jison`; generated files should not be hand-edited.
|
|
22
|
+
|
|
23
|
+
## Testing Guidelines
|
|
24
|
+
- Add runtime tests alongside `test.mjs` or new `test-*.mjs` files.
|
|
25
|
+
- Add type-derivation coverage in `Code-derivation-tests/` (one case per file is typical).
|
|
26
|
+
- Verify locally with `npm test` before opening a PR.
|
|
27
|
+
|
|
28
|
+
## Commit & Pull Request Guidelines
|
|
29
|
+
- Commits use short, imperative, sentence-case summaries (e.g., "Improve type derivation").
|
|
30
|
+
- PRs should include: a clear description, the commands run (`npm test` or specific subsets), and any relevant examples or docs updates.
|
|
31
|
+
- If changes touch parsing or grammars, note whether `npm run prepare` was run.
|
|
32
|
+
|
|
33
|
+
## Configuration Notes
|
|
34
|
+
- Requires Node.js 18+.
|
|
35
|
+
- The REPL and CLI binaries are `k-repl` and `k` (see `package.json` for mappings).
|
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented here.
|
|
4
|
+
|
|
5
|
+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
|
6
|
+
This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## [6.0.0] — 2026-05-12 — First public release
|
|
11
|
+
|
|
12
|
+
### Language & Runtime
|
|
13
|
+
|
|
14
|
+
- First-class partial functions over algebraic data types (products and tagged unions).
|
|
15
|
+
- Three combinators: composition `(f g)`, merge `<f, g>`, product `{f l1, g l2}`.
|
|
16
|
+
- Pattern-based type derivation with filter expressions (`?<...>`, `?{...}`).
|
|
17
|
+
- Canonical (hash-addressed) code names — two structurally equivalent codes always produce the same hash.
|
|
18
|
+
- Content-addressed object files (`.ko` / `.klib`) for compiled modules.
|
|
19
|
+
|
|
20
|
+
### Type System
|
|
21
|
+
|
|
22
|
+
- Finite tree automaton type representation (`codes.mjs`).
|
|
23
|
+
- Graph-based constraint propagation via `TypePatternGraph`.
|
|
24
|
+
- SCC-aware convergence with configurable strategy (`auto` / `single_pass` / `fixed_point`).
|
|
25
|
+
- `compileStats` API for inspecting convergence behaviour per SCC.
|
|
26
|
+
|
|
27
|
+
### Toolchain
|
|
28
|
+
|
|
29
|
+
- `k` — CLI executor: reads binary pattern+value stream, applies a k script, writes result.
|
|
30
|
+
- `k-repl` / `k-repl2` — interactive interpreter with tab-completion.
|
|
31
|
+
- `k-pattern` — extract the canonical root pattern from a k script.
|
|
32
|
+
- `k-compile-object` / `k-decompile-object` — object file compilation and decompilation.
|
|
33
|
+
|
|
34
|
+
### Codec Pipeline
|
|
35
|
+
|
|
36
|
+
- Binary format: serialised pattern graph followed by value encoded under that pattern.
|
|
37
|
+
- JSON codec, UTF-8/UTF-16 string codec, IEEE 754 codec.
|
|
38
|
+
- Polymorphic codec streams: pattern carried in-memory through projections and constructors.
|
|
39
|
+
|
|
40
|
+
### Node.js Library API
|
|
41
|
+
|
|
42
|
+
- `k.compile(source, options?)` — returns a runnable JS function.
|
|
43
|
+
- `k.annotate(source, options?)` — type-checks and returns annotated AST with `compileStats`.
|
|
44
|
+
- `k.run(expression, value)` — evaluate a single expression against a value.
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
[6.0.0]: https://github.com/fraczak/k/releases/tag/v6.0.0
|
package/CLAUDE.md
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
+
|
|
5
|
+
## Project Overview
|
|
6
|
+
|
|
7
|
+
k-language is an experimental programming language for building and manipulating algebraic data types ("codes")—tree-shaped values built from product and tagged union. The native k-like notation is canonical; a JSON-like notation is available as a convenience for newcomers.
|
|
8
|
+
|
|
9
|
+
## Essential Commands
|
|
10
|
+
|
|
11
|
+
**Build/Parser Generation:**
|
|
12
|
+
```bash
|
|
13
|
+
npm run prepare # Generate parsers from Jison grammar files
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
**Testing:**
|
|
17
|
+
```bash
|
|
18
|
+
npm test # Run comprehensive test suite (unit tests, derivation tests, shell tests)
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
**Running k programs:**
|
|
22
|
+
```bash
|
|
23
|
+
./k.mjs <file.k> # Execute k script
|
|
24
|
+
./repl2.mjs # Start interactive REPL (or: k-repl)
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## Architecture
|
|
28
|
+
|
|
29
|
+
### Core Language Concepts
|
|
30
|
+
- **Partial Functions**: Functions may be undefined for some inputs (e.g., `.field` projection)
|
|
31
|
+
- **Function Composition**: `(f g h)` - sequential application
|
|
32
|
+
- **Union/Merge**: `<f, g>` - try f first, fallback to g
|
|
33
|
+
- **Product**: `{f label1, g label2}` - parallel application building records
|
|
34
|
+
- **Types (Codes)**: Finite tree automata defining value sets, no built-in types
|
|
35
|
+
- **Notation**: Prefer native k-like for products `{A l, ...}` and unions `<A t, ...>`; JSON-like `{l: A, ...}` and `<t: A, ...>` is supported for convenience
|
|
36
|
+
- **Variant value literals**: Use single-field product notation. Unit: `{{} tag}`; with payload `v`: `{ v tag }`. Angle brackets are for types/merge, not value literals.
|
|
37
|
+
|
|
38
|
+
### Key Implementation Files
|
|
39
|
+
- `parser.mjs` / `parser.jison` - Language parser (generated from Jison grammar)
|
|
40
|
+
- `run.mjs` - Runtime execution engine for partial functions
|
|
41
|
+
- `codes.mjs` - Type/code management and canonicalization
|
|
42
|
+
- `TypePatternGraph.mjs` - Core pattern graph data structure for type derivation
|
|
43
|
+
- `typing.mjs` - Type compression and advanced type system operations
|
|
44
|
+
- `Value.mjs` - Value representation and operations
|
|
45
|
+
- `index.mjs` - Main library entry point
|
|
46
|
+
|
|
47
|
+
### Multi-Stage Processing Pipeline
|
|
48
|
+
1. **Parsing**: Text → AST via Jison-generated parser
|
|
49
|
+
2. **Type Derivation**: Pattern-based type inference across expressions
|
|
50
|
+
3. **Canonicalization**: Types/functions get hash-based canonical names
|
|
51
|
+
4. **Execution**: Runtime evaluation with partial function semantics
|
|
52
|
+
|
|
53
|
+
### Type System
|
|
54
|
+
- **Product types**: `{type1 label1, type2 label2}` with projections `.label1`
|
|
55
|
+
- **Union types**: `<type1 label1, type2 label2>` with variant selection
|
|
56
|
+
- **Pattern matching**: Sophisticated constraint derivation system
|
|
57
|
+
- **Canonical names**: Hash-based universal registry for content-addressed modules
|
|
58
|
+
|
|
59
|
+
## Development Notes
|
|
60
|
+
|
|
61
|
+
- No external runtime dependencies (only jiwson for parser generation)
|
|
62
|
+
- Uses ES6 modules (.mjs files) throughout
|
|
63
|
+
- Extensive test suite in `Code-derivation-tests/` for type system validation
|
|
64
|
+
- REPL supports commands like `--C name` (show canonical type) and `--R name` (show type derivation)
|
package/DOCS/BIG_PICTURE.md
CHANGED
|
@@ -255,14 +255,10 @@ cdr = list? /cons .cdr;
|
|
|
255
255
|
For example, the filter annotation for `car` is:
|
|
256
256
|
|
|
257
257
|
```k-repl
|
|
258
|
-
|
|
259
|
-
{
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
def: '?<{X1 car, X0 cdr} cons, $KL nil>=X0 .cons .car ?X1'
|
|
263
|
-
}
|
|
264
|
-
--C KL
|
|
265
|
-
$ KL = {}; -- $C0={};
|
|
258
|
+
> :t car
|
|
259
|
+
car : ?<{X1 car, X0 cdr} cons, $KL nil>=X0 --> ?X1 (@...)
|
|
260
|
+
> :C KL
|
|
261
|
+
$ KL = {}; -- @...
|
|
266
262
|
```
|
|
267
263
|
|
|
268
264
|
## Universal Schema Registry
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# Codec Wire Format Status
|
|
2
|
+
|
|
3
|
+
This note tracks the current `k-parse | k | k-print` pipeline after the move
|
|
4
|
+
to self-hosted binary pattern framing.
|
|
5
|
+
|
|
6
|
+
## Current Shape
|
|
7
|
+
|
|
8
|
+
The only wire format is:
|
|
9
|
+
|
|
10
|
+
```text
|
|
11
|
+
encode($pattern_value : $pattern) encode(value : decoded_pattern)
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
where:
|
|
15
|
+
|
|
16
|
+
- the first segment is a normal k value of type `$pattern`, encoded under the
|
|
17
|
+
fixed singleton pattern for `$pattern` from [`core.k`](../core.k),
|
|
18
|
+
- that decoded pattern becomes the structural authority for the second segment,
|
|
19
|
+
- the second segment is the prefix-free value payload interpreted by that
|
|
20
|
+
pattern.
|
|
21
|
+
|
|
22
|
+
Decoding the stream produces a materialized `Value` that carries the decoded
|
|
23
|
+
pattern as `value.pattern`. The evaluator propagates the carried pattern, and
|
|
24
|
+
encoding uses it by default.
|
|
25
|
+
|
|
26
|
+
There is no second JSON transport format. Property-list JSON may still appear in
|
|
27
|
+
tests or debug output as a readable notation for pattern graphs, but it is not a
|
|
28
|
+
codec container and is not accepted at the command boundary.
|
|
29
|
+
|
|
30
|
+
## Current Pipeline
|
|
31
|
+
|
|
32
|
+
1. `k-parse`: text value plus optional pattern/type script -> binary pattern+value stream
|
|
33
|
+
2. `k`: binary input -> `Value(pattern, tree)` -> evaluate `k` expression -> binary pattern+value stream
|
|
34
|
+
3. `k-print`: binary input -> JSON-like textual k value
|
|
35
|
+
|
|
36
|
+
This keeps the evaluator stage free of formatting concerns while staying close
|
|
37
|
+
to the new semantic codec model.
|
|
38
|
+
|
|
39
|
+
When `k-parse` has no explicit input pattern or type, it builds the pattern from
|
|
40
|
+
the parsed value tree. Empty nodes become closed products, multi-child nodes
|
|
41
|
+
become closed products, and one-child nodes become open unions by default. An
|
|
42
|
+
explicit product pattern can force a one-child node to be treated as a singleton
|
|
43
|
+
product.
|
|
44
|
+
|
|
45
|
+
The derived wire pattern is canonicalized by collapsing finite closed
|
|
46
|
+
subtrees from the leaves upward, starting at `["closed-product", []]`. Open
|
|
47
|
+
pattern nodes remain distinct. This is pattern graph canonicalization, not
|
|
48
|
+
value-payload DAG compression.
|
|
49
|
+
|
|
50
|
+
## Verified Workflow
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
echo '["zebara", "ela", "kupa", ala, owca]' | \
|
|
54
|
+
./codecs/k-parse.mjs | \
|
|
55
|
+
./k.mjs '{.1 0,.3 1}' | \
|
|
56
|
+
./codecs/k-print.mjs
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Expected output:
|
|
60
|
+
|
|
61
|
+
```json
|
|
62
|
+
["ela","ala"]
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Open Work
|
|
66
|
+
|
|
67
|
+
- tighten the pattern/value validation rules,
|
|
68
|
+
- add direct coverage for more non-ASCII pattern labels in CLI fixtures,
|
|
69
|
+
- extend tests and examples for the prefix codec.
|
|
70
|
+
|
|
71
|
+
For the semantic design, see:
|
|
72
|
+
|
|
73
|
+
- [`codecs/README.md`](../codecs/README.md)
|
|
74
|
+
- [`codecs/POLYMORPHIC_BINARY_FORMAT.md`](../codecs/POLYMORPHIC_BINARY_FORMAT.md)
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
# Convergence Strategies
|
|
2
|
+
|
|
3
|
+
This note describes the current structure of the JavaScript type-derivation engine and the places intended for experimentation.
|
|
4
|
+
|
|
5
|
+
## Current default
|
|
6
|
+
|
|
7
|
+
Type derivation operates on the SCC DAG of relation dependencies.
|
|
8
|
+
|
|
9
|
+
- Singleton SCC with no self-reference:
|
|
10
|
+
- use `single_pass`
|
|
11
|
+
- clone already-derived callee boundary patterns once
|
|
12
|
+
- compact once
|
|
13
|
+
- Recursive SCC:
|
|
14
|
+
- use `fixed_point`
|
|
15
|
+
- iterate until signatures stabilize or the iteration budget is exhausted
|
|
16
|
+
|
|
17
|
+
This split keeps acyclic modules fast while preserving the old behavior for recursive polymorphic definitions.
|
|
18
|
+
|
|
19
|
+
## Public tuning hook
|
|
20
|
+
|
|
21
|
+
`annotate`, `compile`, and `run` accept:
|
|
22
|
+
|
|
23
|
+
```js
|
|
24
|
+
{
|
|
25
|
+
convergence: {
|
|
26
|
+
strategy: "auto" | "single_pass" | "fixed_point",
|
|
27
|
+
maxIterations?: number
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The returned `annotate(...)` object includes:
|
|
33
|
+
|
|
34
|
+
```js
|
|
35
|
+
{
|
|
36
|
+
compileStats: {
|
|
37
|
+
sccCount,
|
|
38
|
+
sccs: [
|
|
39
|
+
{
|
|
40
|
+
members: ["relA", "relB"],
|
|
41
|
+
strategy: "fixed_point",
|
|
42
|
+
iterations: 3,
|
|
43
|
+
converged: true
|
|
44
|
+
}
|
|
45
|
+
]
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
This is the supported inspection surface for convergence experiments.
|
|
51
|
+
|
|
52
|
+
## Why the current fast path works
|
|
53
|
+
|
|
54
|
+
For an acyclic singleton SCC, every callee has already been fully derived by the time the current relation is processed. There is no need for repeated fixed-point propagation because:
|
|
55
|
+
|
|
56
|
+
1. Callee boundary patterns are stable.
|
|
57
|
+
2. The current relation only needs those stable input/output boundaries.
|
|
58
|
+
3. One propagation pass plus one compaction is sufficient.
|
|
59
|
+
|
|
60
|
+
`Examples/ieee.k` is exactly this shape: a large acyclic dependency graph. The major speedup comes from avoiding unnecessary fixed-point iteration there.
|
|
61
|
+
|
|
62
|
+
## Current implementation choices
|
|
63
|
+
|
|
64
|
+
Two implementation changes matter for performance:
|
|
65
|
+
|
|
66
|
+
1. Compaction now starts from the quotient graph of live representatives instead of cloning the full historical union-find forest.
|
|
67
|
+
2. Singleton-pattern registration is incremental and no longer re-finalizes the entire global code repository on every compaction.
|
|
68
|
+
|
|
69
|
+
These two changes are what make `Examples/ieee.k` practical to type-check.
|
|
70
|
+
|
|
71
|
+
## Where to experiment next
|
|
72
|
+
|
|
73
|
+
If you want to continue convergence work, these are the main seams:
|
|
74
|
+
|
|
75
|
+
- `convergence.mjs`
|
|
76
|
+
- strategy selection
|
|
77
|
+
- recursive SCC processing
|
|
78
|
+
- future divergence detection
|
|
79
|
+
- `compression.mjs`
|
|
80
|
+
- quotient-graph construction
|
|
81
|
+
- equivalence partitioning
|
|
82
|
+
- alternative canonicalization schemes
|
|
83
|
+
- `typing.mjs`
|
|
84
|
+
- clone/unify behavior
|
|
85
|
+
- pattern-graph data structure
|
|
86
|
+
|
|
87
|
+
Promising next experiments:
|
|
88
|
+
|
|
89
|
+
- Replace signature-string stabilization in `fixed_point` with structural change tracking.
|
|
90
|
+
- Add divergence heuristics for recursive polymorphic SCCs.
|
|
91
|
+
- Cache cloned callee boundary graphs for repeated call sites within one relation.
|
|
92
|
+
- Compare the current partition refinement in `compression.mjs` with a hash-consed canonical-form approach.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Divergence detection (sketch)
|
|
2
|
+
|
|
3
|
+
We can detect divergence when there is a cycle of definitions:
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
X1 = D1;
|
|
7
|
+
X2 = D2;
|
|
8
|
+
...
|
|
9
|
+
Xn = Dn;
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
such that for each i, `Xi` is in `HEAD(Di-1)` (with indices modulo n).
|
|
13
|
+
|
|
14
|
+
Define `HEAD` as the set of direct first ref calls:
|
|
15
|
+
|
|
16
|
+
```
|
|
17
|
+
HEAD(ref(X)) = {X}
|
|
18
|
+
HEAD(?F E) = HEAD(E)
|
|
19
|
+
HEAD($T E) = HEAD(E)
|
|
20
|
+
HEAD((E1 E2 ...)) = HEAD(E1)
|
|
21
|
+
HEAD({ E1 f1, E2 f2, ... }) = HEAD(E1) union HEAD(E2) union ...
|
|
22
|
+
HEAD(<E1, E2, ...>) = {} -- heads of a union is empty set
|
|
23
|
+
HEAD(/ tag) = HEAD(. lab) = HEAD(| tag) = {}
|
|
24
|
+
```
|
package/DOCS/Makefile
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# Dependencies: pandoc, dejavu-fonts-all, texlive-xetex
|
|
2
|
+
|
|
3
|
+
CHAPTERS = book/01-introduction.md \
|
|
4
|
+
book/02-syntax-and-values.md \
|
|
5
|
+
book/03-types-as-finite-automata.md \
|
|
6
|
+
book/04-partial-functions-and-composition.md \
|
|
7
|
+
book/05-typing-filters-and-normalization.md \
|
|
8
|
+
book/06-values-in-memory.md \
|
|
9
|
+
book/07-the-partial-function-abi.md \
|
|
10
|
+
book/08-operational-semantics-and-execution.md \
|
|
11
|
+
book/09-compiler-architecture.md \
|
|
12
|
+
book/10-from-ast-to-intermediate-representation.md \
|
|
13
|
+
book/11-llvm-basics.md \
|
|
14
|
+
book/12-code-generation.md \
|
|
15
|
+
book/13-linking-and-execution.md \
|
|
16
|
+
book/14-canonical-serialization.md \
|
|
17
|
+
book/15-optimization-and-folding.md \
|
|
18
|
+
book/16-toward-a-universal-schema-registry.md \
|
|
19
|
+
book/appendix-a-implementing-a-prototype-compiler-in-python.md \
|
|
20
|
+
book/appendix-b-incorporating-external-predefined-types-and-functions.md
|
|
21
|
+
|
|
22
|
+
all: book-combined.pdf book-combined.html book-combined.tex book.pdf book.html book.tex
|
|
23
|
+
|
|
24
|
+
# Combined book from individual chapters
|
|
25
|
+
book-combined.md: $(CHAPTERS)
|
|
26
|
+
echo "---" > book-combined.md
|
|
27
|
+
echo "title: \"k-language\"" >> book-combined.md
|
|
28
|
+
echo "author: \"W. Fraczak\"" >> book-combined.md
|
|
29
|
+
echo "---" >> book-combined.md
|
|
30
|
+
echo "" >> book-combined.md
|
|
31
|
+
echo "<div align=\"center\">" >> book-combined.md
|
|
32
|
+
echo "" >> book-combined.md
|
|
33
|
+
echo "\`\`\`" >> book-combined.md
|
|
34
|
+
echo " { } { }" >> book-combined.md
|
|
35
|
+
echo " / \\ {.lang name} |" >> book-combined.md
|
|
36
|
+
echo " lang by --> name" >> book-combined.md
|
|
37
|
+
echo " | | |" >> book-combined.md
|
|
38
|
+
echo " 'k' 'wojtek' 'k'" >> book-combined.md
|
|
39
|
+
echo "\`\`\`" >> book-combined.md
|
|
40
|
+
echo "" >> book-combined.md
|
|
41
|
+
echo "</div>" >> book-combined.md
|
|
42
|
+
echo "" >> book-combined.md
|
|
43
|
+
echo "# Table of Contents" >> book-combined.md
|
|
44
|
+
echo "" >> book-combined.md
|
|
45
|
+
echo "1. [Introduction](#chapter-1-introduction)" >> book-combined.md
|
|
46
|
+
echo "2. [Syntax and Values](#chapter-2-syntax-and-values)" >> book-combined.md
|
|
47
|
+
echo "3. [Types as Finite Automata](#chapter-3-types-as-finite-automata)" >> book-combined.md
|
|
48
|
+
echo "4. [Partial Functions and Composition](#chapter-4-partial-functions-and-composition)" >> book-combined.md
|
|
49
|
+
echo "5. [Typing, Filters, and Normalization](#chapter-5-typing-filters-and-normalization)" >> book-combined.md
|
|
50
|
+
echo "6. [Values in Memory](#chapter-6-values-in-memory)" >> book-combined.md
|
|
51
|
+
echo "7. [The Partial Function ABI](#chapter-7-the-partial-function-abi)" >> book-combined.md
|
|
52
|
+
echo "8. [Operational Semantics and Execution](#chapter-8-operational-semantics-and-execution)" >> book-combined.md
|
|
53
|
+
echo "9. [Compiler Architecture](#chapter-9-compiler-architecture)" >> book-combined.md
|
|
54
|
+
echo "10. [From AST to Intermediate Representation](#chapter-10-from-ast-to-intermediate-representation)" >> book-combined.md
|
|
55
|
+
echo "11. [LLVM Basics](#chapter-11-llvm-basics)" >> book-combined.md
|
|
56
|
+
echo "12. [Code Generation](#chapter-12-code-generation)" >> book-combined.md
|
|
57
|
+
echo "13. [Linking and Execution](#chapter-13-linking-and-execution)" >> book-combined.md
|
|
58
|
+
echo "14. [Canonical Serialization](#chapter-14-canonical-serialization)" >> book-combined.md
|
|
59
|
+
echo "15. [Optimization and Folding](#chapter-15-optimization-and-folding)" >> book-combined.md
|
|
60
|
+
echo "16. [Toward a Universal Schema Registry](#chapter-16-toward-a-universal-schema-registry)" >> book-combined.md
|
|
61
|
+
echo "" >> book-combined.md
|
|
62
|
+
echo "## Appendices" >> book-combined.md
|
|
63
|
+
echo "" >> book-combined.md
|
|
64
|
+
echo "- [Appendix A: Implementing a Prototype Compiler in Python](#appendix-a-implementing-a-prototype-compiler-in-python)" >> book-combined.md
|
|
65
|
+
echo "- [Appendix B: Incorporating External Predefined Types and Functions](#appendix-b-incorporating-external-predefined-types-and-functions)" >> book-combined.md
|
|
66
|
+
echo "" >> book-combined.md
|
|
67
|
+
echo "---" >> book-combined.md
|
|
68
|
+
echo "" >> book-combined.md
|
|
69
|
+
for chapter in $(CHAPTERS); do \
|
|
70
|
+
echo "" >> book-combined.md; \
|
|
71
|
+
cat $$chapter >> book-combined.md; \
|
|
72
|
+
echo "" >> book-combined.md; \
|
|
73
|
+
echo "---" >> book-combined.md; \
|
|
74
|
+
echo "" >> book-combined.md; \
|
|
75
|
+
done
|
|
76
|
+
|
|
77
|
+
book-combined.tex: book-combined.md
|
|
78
|
+
pandoc book-combined.md -t latex -s -o book-combined.tex
|
|
79
|
+
|
|
80
|
+
book-combined.pdf: book-combined.md
|
|
81
|
+
pandoc book-combined.md --pdf-engine=xelatex -V monofont="DejaVu Sans Mono" -V mainfont="DejaVu Serif" -o book-combined.pdf
|
|
82
|
+
|
|
83
|
+
book-combined.html: book-combined.md
|
|
84
|
+
pandoc book-combined.md -o book-combined.html
|
|
85
|
+
|
|
86
|
+
# Master file (table of contents only)
|
|
87
|
+
book.tex: book.md
|
|
88
|
+
pandoc book.md -t latex -s -o book.tex
|
|
89
|
+
|
|
90
|
+
book.pdf: book.md
|
|
91
|
+
pandoc book.md --pdf-engine=xelatex -V monofont="DejaVu Sans Mono" -V mainfont="DejaVu Serif" -o book.pdf
|
|
92
|
+
|
|
93
|
+
book.html: book.md
|
|
94
|
+
pandoc book.md -o book.html
|
|
95
|
+
|
|
96
|
+
clean:
|
|
97
|
+
rm -f book.pdf book.html book.tex book-combined.* *.aux *.log *.out
|
|
98
|
+
|
|
99
|
+
.PHONY: all clean
|
|
@@ -0,0 +1,217 @@
|
|
|
1
|
+
# Object File And Pattern Type
|
|
2
|
+
|
|
3
|
+
This note records the representation boundary after parsing and type derivation.
|
|
4
|
+
The codec boundary is no longer a JSON object. A serialized value is one binary
|
|
5
|
+
stream:
|
|
6
|
+
|
|
7
|
+
```text
|
|
8
|
+
encode($pattern_value : $pattern) encode(value : decoded_pattern)
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
The pattern is the framing protocol. There is no separate envelope type.
|
|
12
|
+
|
|
13
|
+
## Pattern Schema
|
|
14
|
+
|
|
15
|
+
[`core.k`](../core.k) defines the canonical self-hosted pattern type:
|
|
16
|
+
|
|
17
|
+
```k
|
|
18
|
+
$ edge = { string label, bits target };
|
|
19
|
+
$ edges = < {} nil, { edge car, edges cdr } cons >;
|
|
20
|
+
|
|
21
|
+
$ pattern-node = <
|
|
22
|
+
{} any,
|
|
23
|
+
edges open-product,
|
|
24
|
+
edges open-union,
|
|
25
|
+
edges closed-product,
|
|
26
|
+
edges closed-union
|
|
27
|
+
>;
|
|
28
|
+
|
|
29
|
+
$ pattern = < {} nil, { pattern-node car, pattern cdr } cons >;
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
In this schema:
|
|
33
|
+
|
|
34
|
+
- the first pattern-list element is node `0`, the root,
|
|
35
|
+
- an edge target is a node index encoded as `$bits`,
|
|
36
|
+
- labels are `$string` values and therefore Unicode scalar-value strings,
|
|
37
|
+
- edge lists are sorted by label,
|
|
38
|
+
- `any` carries no edges,
|
|
39
|
+
- the four product/union node kinds carry their edge list directly.
|
|
40
|
+
|
|
41
|
+
The fixed singleton pattern of `$pattern` is the first decoder context for all
|
|
42
|
+
k wire values. Decoding any value first decodes a `$pattern` value under that
|
|
43
|
+
constant, then decodes the remaining bits under the pattern just obtained.
|
|
44
|
+
|
|
45
|
+
## Debug Notation
|
|
46
|
+
|
|
47
|
+
Documentation and tests often show patterns as property-list JSON because it is
|
|
48
|
+
compact to read:
|
|
49
|
+
|
|
50
|
+
```json
|
|
51
|
+
[
|
|
52
|
+
["open-union", [["a", 1]]],
|
|
53
|
+
["closed-product", [["b", 2], ["c", 3]]],
|
|
54
|
+
["open-union", [["x", 3]]],
|
|
55
|
+
["closed-product", []]
|
|
56
|
+
]
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
That notation is not a transport format. It is only a host-side rendering of
|
|
60
|
+
the same abstract graph.
|
|
61
|
+
|
|
62
|
+
## Object File Direction
|
|
63
|
+
|
|
64
|
+
The parser plus type-derivation phase should eventually emit an object value
|
|
65
|
+
rather than executable JavaScript structures. A minimal first object shape is:
|
|
66
|
+
|
|
67
|
+
```k
|
|
68
|
+
$ object-file = {
|
|
69
|
+
pattern input-pattern,
|
|
70
|
+
pattern output-pattern,
|
|
71
|
+
pattern value-pattern
|
|
72
|
+
};
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
The concrete input payload is then stored in the same binary stream form:
|
|
76
|
+
`encode(value-pattern) encode(input : value-pattern)`. The exact object-file
|
|
77
|
+
type still needs design work because the concrete input value is typed by the
|
|
78
|
+
decoded pattern. The important boundary is that object files should carry
|
|
79
|
+
ordinary k values and patterns, not a JSON codec object.
|
|
80
|
+
|
|
81
|
+
Later versions can add relation IR, constant tables, symbol dictionaries,
|
|
82
|
+
relocation/linking records, optimization metadata, and optional indexes.
|
|
83
|
+
|
|
84
|
+
## Executable Object Container v1
|
|
85
|
+
|
|
86
|
+
The current executable object format is a small binary container:
|
|
87
|
+
|
|
88
|
+
```text
|
|
89
|
+
"KOBJ" 0x00 0x01 0x0a
|
|
90
|
+
uint32-be payload-byte-length
|
|
91
|
+
utf8-json-payload
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
The payload stores:
|
|
95
|
+
|
|
96
|
+
- the normalized code repository needed by the object,
|
|
97
|
+
- the compiled relation IR,
|
|
98
|
+
- each relation's type-pattern graph state,
|
|
99
|
+
- the main relation name.
|
|
100
|
+
|
|
101
|
+
Loading an object hydrates the type-pattern graphs back into runtime
|
|
102
|
+
`TypePatternGraph` instances and executes them through `run.mjs`. This avoids
|
|
103
|
+
source parsing and type derivation on the execution path.
|
|
104
|
+
|
|
105
|
+
Object execution through `k.mjs`:
|
|
106
|
+
|
|
107
|
+
```sh
|
|
108
|
+
echo "..." | ./some-codec --parse | ./k.mjs -k path/to/program.ko | ./some-codec --print
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
The `-k` option accepts either an object file or a source `.k` file. `k.mjs`
|
|
112
|
+
tries to load the file as an object first, then falls back to source parsing and
|
|
113
|
+
compilation.
|
|
114
|
+
|
|
115
|
+
Standalone helpers live in `objects/`:
|
|
116
|
+
|
|
117
|
+
```sh
|
|
118
|
+
./objects/compile.mjs path/to/program.k path/to/program.ko
|
|
119
|
+
./objects/decompile.mjs path/to/program.ko path/to/program.decompiled.k
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
With no arguments, both helpers read from stdin and write to stdout:
|
|
123
|
+
|
|
124
|
+
```sh
|
|
125
|
+
cat path/to/program.k | ./objects/compile.mjs > path/to/program.ko
|
|
126
|
+
cat path/to/program.ko | ./objects/decompile.mjs > path/to/program.decompiled.k
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
The decompiled source is canonical source regenerated from object IR. It uses
|
|
130
|
+
shortest unique prefixes of the object's hash-based names for type and relation
|
|
131
|
+
definitions, with a minimum prefix length of `4`. It is intended to be runnable
|
|
132
|
+
and inspectable, not a byte-for-byte reconstruction of the original source.
|
|
133
|
+
|
|
134
|
+
## Small Object Example
|
|
135
|
+
|
|
136
|
+
For a tiny source file:
|
|
137
|
+
|
|
138
|
+
```k
|
|
139
|
+
-- id.k
|
|
140
|
+
()
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
compile it with:
|
|
144
|
+
|
|
145
|
+
```sh
|
|
146
|
+
./objects/compile.mjs id.k id.ko
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
The first bytes of `id.ko` are the binary container header:
|
|
150
|
+
|
|
151
|
+
```text
|
|
152
|
+
4b 4f 42 4a 00 01 0a 00 00 03 0b ...
|
|
153
|
+
K O B J \0 v1 \n payload-length
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
After the header, the payload is UTF-8 JSON. Pretty-printed and shortened, the
|
|
157
|
+
payload has this shape:
|
|
158
|
+
|
|
159
|
+
```json
|
|
160
|
+
{
|
|
161
|
+
"format": "k-object",
|
|
162
|
+
"version": 1,
|
|
163
|
+
"codes": {
|
|
164
|
+
"@NiDZqYggx3VZ6b8quBZKTfkgJztWctkesuX4CrhTxM5c": {
|
|
165
|
+
"code": "product",
|
|
166
|
+
"product": {},
|
|
167
|
+
"def": "$C0={};"
|
|
168
|
+
}
|
|
169
|
+
},
|
|
170
|
+
"main": "__main__",
|
|
171
|
+
"defs": {
|
|
172
|
+
"rels": {
|
|
173
|
+
"__main__": {
|
|
174
|
+
"def": {
|
|
175
|
+
"op": "filter",
|
|
176
|
+
"filter": { "type": null, "open": true, "fields": {}, "name": "X0" },
|
|
177
|
+
"patterns": [0, 0],
|
|
178
|
+
"start": { "line": 1, "column": 1 },
|
|
179
|
+
"end": { "line": 1, "column": 3 }
|
|
180
|
+
},
|
|
181
|
+
"typePatternGraph": {
|
|
182
|
+
"patterns": {
|
|
183
|
+
"nodes": [{ "pattern": "(...)", "fields": [] }],
|
|
184
|
+
"parent": []
|
|
185
|
+
},
|
|
186
|
+
"edges": [{}],
|
|
187
|
+
"codeId": {}
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
},
|
|
191
|
+
"representatives": {},
|
|
192
|
+
"relAlias": { "__main__": "..." },
|
|
193
|
+
"compileStats": { "sccs": [...], "sccCount": 1 }
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
The `codes` field is the type repository snapshot needed by this object. The
|
|
199
|
+
`defs.rels.__main__.def` field is the executable relation IR. The
|
|
200
|
+
`typePatternGraph` field is the compiled pattern graph used by `run.mjs` for
|
|
201
|
+
input-envelope refinement. Loading the object reconstructs live
|
|
202
|
+
`TypePatternGraph` instances from this JSON state and then evaluates the input
|
|
203
|
+
value directly against the compiled IR.
|
|
204
|
+
|
|
205
|
+
Decompiling this object produces canonical source with generated prefix names:
|
|
206
|
+
|
|
207
|
+
```k
|
|
208
|
+
----- codes -----
|
|
209
|
+
$ NiDZ = {};
|
|
210
|
+
----- rels -----
|
|
211
|
+
QfyL = ?X0;
|
|
212
|
+
----- main -----
|
|
213
|
+
?(...) QfyL ?(...)
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
In the `rels` section, relation definitions are printed in the same SCC order
|
|
217
|
+
recorded during compilation. Non-empty SCC groups are separated by a blank line.
|