mt-lang 0.3.26 → 0.3.27

data/docs/self-host-plan.md

@@ -0,0 +1,768 @@
+# Self-Host Plan
+
+Status: **Planning** — see `docs/self-host-progress.md` for step-by-step tracking.
+
+## 0. Goal
+
+Port the Milk Tea compiler from Ruby to Milk Tea. The result must achieve a
+**fixed point**: stage2-generated C is byte-identical to stage3-generated C,
+proving the compiler is self-consistent.  The working reference commit from the
+previous self-host incarnation that achieved this is `a28545a7`.
+
+Ruby host: `v0.3.26`, ~86K lines across 256 files.
+Scope: core compiler pipeline (lex → parse → sema → lower → C backend) plus
+build tooling, bootstrap, and test suite.  LSP, DAP, bindgen, and the packages
+subsystem are deferred to post-fixed-point phases.
+
+## 1. Architecture
+
+### 1.1 Pipeline
+
+```
+Source (.mt) → Lexer → Parser → Module Loader → Semantic Analyzer → Control Flow → Lowering → IR → C Backend → C code
+                                    ↑                                  ↑
+                              Module Binder                     Compile-Time Eval
+                              Type System                       Intrinsics
+                              AST / Token / IR
+```
+
+| Stage | Ruby source | Self-host module | Lines (Ruby) |
+|-------|-------------|-----------------|-------------|
+| AST / Token / CST / Keywords | `ast.rb` (369), `token.rb` (43), `cst.rb` (64), `keywords.rb` (97) | `mtc/parser/ast.mt`, `mtc/lexer/token.mt`, `mtc/lexer/keywords.mt` | 573 |
+| Lexer | `lexer.rb` (427) + `lexer/` (949) | `mtc/lexer/` | 1,376 |
+| Parser | `parser.rb` (442) + `parser/` (2,788) | `mtc/parser/` | 3,230 |
+| Type System | `types.rb` (1,727) + `types/` (1,448) | `mtc/semantic/types.mt` | 3,175 |
+| Module Loader & Resolver | `module_loader.rb` (717), `module_path_resolver.rb` (152), `module_roots.rb` (86) | `mtc/loader/` | 955 |
+| Module Binder & Bindings | `module_binder.rb` (187), `bindings.rb` (13) + `bindings/` (148) | `mtc/loader/binder.mt` | 348 |
+| Semantic Analyzer | `semantic_analyzer.rb` (355) + `semantic_analyzer/` (10,262) | `mtc/semantic/` | 10,617 |
+| Control Flow | `control_flow.rb` (34) + `control_flow/` (1,148) | `mtc/semantic/control_flow/` | 1,182 |
+| Compile-Time Evaluator | `compile_time.rb` (533) | `mtc/semantic/compile_time.mt` | 533 |
+| Intrinsics | `intrinsics.rb` (87), `flow_scope.rb` (13) | `mtc/semantic/intrinsics.mt` | 100 |
+| Lowering | `lowering.rb` (378) + `lowering/` (15,201) | `mtc/lowering/` | 15,579 |
+| IR | `ir.rb` (88) | `mtc/ir.mt` | 88 |
+| C Backend | `c_backend.rb` (322) + `c_backend/` (5,826) | `mtc/c_backend/` | 6,148 |
+| Pretty Printer | `pretty_printer.rb` (17) + `pretty_printer/` (1,292) | `mtc/pretty_printer/` | 1,309 |
+| S-expression Dumper | `sexpr_dumper.rb` (344) | `mtc/sexpr_dumper.mt` | 344 |
+| Prelude / Async Installers | `prelude_installer.rb` (29), `async_runtime_installer.rb` (26) | merged into module loader / lowering | 55 |
+| Debug Info Formatter | `debug_info_formatter.rb` (454) | `mtc/debug_info.mt` | 454 |
+| CLI Args + Shared Helpers | in `cli.rb` (900+) | folded into `mtc/main.mt` | ~900 |
+| CLI sub-commands | `cli/commands/` (22 files) | folded into `mtc/main.mt` | ~2,500 |
+| Build / Run / Cache | `build.rb`, `build_cache.rb`, `run.rb` | `mtc/build.mt`, `mtc/build_cache.mt` | ~2,500 |
+| **Core total** | **45,268** | | |
+
+### 1.2 Module Structure
+
+```
+projects/mtc/
+  src/mtc/
+    main.mt              — CLI entrypoint and command dispatch
+    build.mt             — build orchestration (compile → C → binary)
+    build_cache.mt       — incremental compilation cache
+    version_info.mt      — git revision stamp placeholder
+    ast.mt               — AST node definitions (shared by parser, sema, lowering, C backend)
+    ir.mt                — IR data types (pure data, no logic)
+    sexpr_dumper.mt      — S-expression serializer for tokens, AST, types, IR
+    lexer/
+      token_kinds.mt     — TokenKind enum
+      token.mt           — Token struct, LexDiagnostic
+      keywords.mt        — keyword lookup table
+      lexer.mt           — core lexer (lines, tokens, indentation)
+    parser/
+      token_stream.mt    — SyntaxTokenStream wrapper
+      state.mt           — parser state, diagnostics
+      literal_parsing.mt — number/string/char literal parsing
+      parser.mt          — recursive-descent parser
+    loader/
+      path_resolver.mt   — module path resolution with platform variants
+      errors.mt          — error types
+      binder.mt          — ModuleBinding construction from Analysis
+      module_loader.mt   — transitive import loading, cycle handling, sema orchestration
+    semantic/
+      types.mt           — Type variant, predicates, layout helpers, rendering
+      scope.mt           — lexical scope tracking
+      expressions.mt     — expression type inference helpers
+      diagnostics.mt     — diagnostic struct
+      emit_expansion.mt  — emit statement expansion
+      type_compatibility.mt — assignability, coercion, narrowing rules
+      intrinsics.mt      — built-in callable handler (fatal, ref_of, read, size_of, etc.)
+      compile_time.mt    — const/const function evaluator
+      analyzer.mt        — semantic analysis (declarations + function bodies)
+      control_flow/
+        builder.mt       — CFG construction
+        definite_assignment.mt — definite-assignment analysis
+        reachability.mt  — return-on-all-paths, termination
+        nullability.mt   — nullable flow narrowing
+        constprop.mt     — constant propagation
+    lowering/
+      utils.mt           — C name mangling (~20 functions), type classification helpers, range/match utilities
+      async.mt           — async CPS lowering + root-main synthesis
+      lowering.mt        — AST → IR transformation (main lowering pass, single module initially)
+    c_backend/
+      c_backend.mt       — C code generation from IR; includes C keyword sanitization (C_KEYWORDS + sanitize_c_identifier)
+    pretty_printer/
+      ast_formatter.mt   — AST → formatted text output
+      ir_formatter.mt    — IR → text output
+  test/
+    lexer_test.mt
+    parser_test.mt
+    semantic_test.mt
+    lowering_test.mt
+    c_backend_test.mt
+    module_loader_test.mt
+    path_resolver_test.mt
+    ast_formatter_test.mt
+    scope_test.mt
+    linter_test.mt
+    fix_engine_test.mt
+    completion_test.mt
+```
+
+### 1.3 Key Data Representations
+
+**Type system** — a `variant` (tagged union) mirroring Ruby's `Types::Base` hierarchy:
+
+```mt
+public variant Type:
+    ty_primitive(name: str)
+    ty_str
+    ty_error
+    ty_nullable(base: ptr[Type])
+    ty_named(name: str, module_name: str)
+    ty_imported(module_name: str, name: str, args: span[Type])
+    ty_generic(name: str, args: span[Type])
+    ty_function(params: span[Type], return_type: ptr[Type], variadic: bool, is_proc: bool)
+    ty_tuple(elements: span[Type], field_names: Option[span[str]])
+    ty_var(name: str)                    — unresolved type variable
+    ty_dyn(iface: str)                   — dyn[InterfaceName]
+    ty_opaque(module_name, name)
+    ty_literal_int(value: long)          — compile-time integer type arg
+```
+
+**AST** — structs for non-recursive decls + `ptr` indirection for recursive nodes:
+
+- `SourceFile` struct: module name, kind, directives, declarations span
+- `TypeRef` variant: named type, generic type, function type, pointer type, etc.
+- Expression variants: identifier, literal, binary, call, member access, etc.
+- Statement variants: let/var decl, if, while, for, match, return, etc.
+
+**IR** — a decoupled variant between Lowering and CBackend (mirrors `ir.rb`):
+
+- `Expr` variant: name, member, index, call, unary, binary, conditional, literal, etc.
+- `Stmt` variant: local, assignment, block, if, switch, while, for, return, etc.
+- Program struct: includes, constants, globals, opaques, structs, enums, variants, functions
+
+**Symbol tables** — `std.map.Map[str, ...]` for every lookup: functions, struct fields, type aliases, member keys, method signatures, value types, interface bindings.  Keys are always `str`.
+
+## 2. Memory Management Strategy
+
+The Ruby compiler has GC.  The self-host must manage memory explicitly.  Three
+allocation tiers map to three lifetimes:
+
+| Tier | Lifetime | Mechanism | Usage |
+|------|----------|-----------|-------|
+| Per-compilation | Entire `build` invocation | `std.mem.arena` (bump allocator) | Source text, AST nodes, IR nodes, type objects, most temporaries |
+| Per-module | Single module parse/check | Arena mark/reset | Per-file lexer output, parser AST, sema Analysis |
+| Persistent | Cross-module, whole program | `std.map.Map` + `std.vec.Vec` (owning heap) | Loader's `Program`, retained analyses, binding maps |
+
+**Arena discipline:** At the start of each module's parse → sema → lower pass,
+mark the arena.  Before emitting IR for the next module, reset to that mark.
+The loader retains per-module `Analysis` values in Vec/Map storage (owning
+heap); everything else is arena-allocated and bulk-freed.
+
+`std.mem.arena.Arena` provides:
+- `create(capacity_bytes)` / `create_aligned(capacity_bytes, alignment)` — initialize
+- `alloc_bytes(size)` / `alloc_bytes_aligned(size, alignment)` — bump-allocate
+- `mark()` → `ptr_uint` — save position
+- `reset(mark)` — rewind to saved mark
+
+**String ownership:** The lexer produces token lexemes as `str` slices into the
+source `string.String`.  The source buffer is arena-allocated and outlives all
+slices.  No string copies during lexing or parsing.  Diagnostic messages and
+cross-module names use `string.String` (owning heap).
+
+**Vec and Map lifecycle:** Dynamic collections used as symbol tables and IR
+span sources require explicit `release()`.  The convention is: the loader owns
+the top-level `Program` and calls `release()` at teardown; all intermediate
+Vec/Map values are arena-allocated and their release is handled by arena reset.
+
+## 3. Bootstrap Architecture
+
+### 3.1 Stage Model
+
+```
+Stage0 (Ruby host or pre-built snapshot binary)
+    ↓ compiles
+Stage1 (build/stage1/mtc)        — carries stage0 code-gen artifacts, not reproducible
+    ↓ compiles
+Stage2 (build/stage2/mtc)        — clean, distributable artifact
+    ↓ compiles
+Stage3 (build/stage3/mtc)        — verification only
+    ↓
+diff stage2.c stage3.c           — must be empty (fixed point)
+```
+
+All stages read the same in-tree `std/` sources.  There is no stage-specific
+standard library — stdlib is source-only.
+
+### 3.2 Stage0 Resolution
+
+Resolution order in `tools/bootstrap.sh`:
+
+1. `--bootstrap PATH` (explicit CLI argument, highest priority)
+2. `$MTC_BOOTSTRAP` environment variable
+3. `bin/bootstrap-mtc` in the repo root (user-placed binary)
+4. `ruby -Ilib bin/mtc` (Ruby host — the development fallback)
+5. Error: "No bootstrap compiler found"
+
+### 3.3 Bootstrap Script
+
+```sh
+tools/bootstrap.sh [OPTIONS]
+
+Options:
+  --bootstrap PATH     Path to stage0 mtc binary (default: auto-detect)
+  --stage {1,2,3}      Build target stage (default: 3)
+  --no-verify          Skip stage3 fixed-point check
+  --profile {debug,release}  Build profile (default: debug)
+  --keep-c             Save generated C files alongside binaries
+  -j N                 Parallel jobs for C compilation (default: nproc)
+```
+
+Key invariants:
+- `--no-cache` on every stage build — prevents cache poisoning across stages
+- `-I .` to make repo-root `std/` visible as a module root
+- Git revision injected into `version_info.mt` via `sed` before each build
+- Fixed-point verification via `diff stage2.c stage3.c`
+- Stage2 self-test: stage2 builds itself, producing a working binary
+
+### 3.4 Development Shortcuts
+
+```sh
+# Fast iteration: just stage1, no verification
+tools/bootstrap.sh --stage 1 --no-verify
+
+# Verify only: assumes stage2 already built
+tools/bootstrap.sh --verify-only
+
+# Self-test: stage2 builds itself
+tools/bootstrap.sh --self-test
+```
+
+## 4. Implementation Plan
+
+### Phase 0: Foundation (types, IR, AST, token definitions)
+
+**Goal:** Data structures and utilities that every subsequent phase depends on.
+
+| Step | Module | Lines (Ruby) | Verify |
+|------|--------|-------------|--------|
+| 0a | `lexer/token_kinds.mt` — TokenKind enum (all token types) | — | Compiles |
+| 0b | `lexer/token.mt` — Token struct, TriviaToken, LexDiagnostic | 43 | Compiles |
+| 0c | `lexer/keywords.mt` — keyword string → TokenKind lookup table | 97 | Unit tests |
+| 0d | `ast.mt` — full AST: SourceFile, all Decl variants, TypeRef, Expr, Stmt | 369 | Compiles |
+| 0e | `semantic/types.mt` — Type variant, classification predicates, rendering, layout helpers | 3,175 | Unit tests |
+| 0f | `ir.mt` — IR Program, Expr, Stmt, all declaration structs. Imports `mtc.ast` (for `ast.ModuleKind`) and `mtc.semantic.types` (for `types.Type` in field/param definitions). | 88 | Compiles (no logic) |
+| 0g | `sexpr_dumper.mt` — S-expression serialization for tokens, AST, types, and IR; byte-identical output vs Ruby. Used by `mtc lex --sexpr`, `mtc parse --sexpr`, and `mtc lower` for diff-based verification at every phase. | 344 | Diff vs Ruby S-expr output |
+
+**Exit criteria:** All types compile under `mtc check`.  The types module handles
+the full type model: primitive lookup tables (integer widths, float widths),
+type classification (`is_numeric`, `is_bool`, `is_void`, `contains_error`),
+rendering (`type_to_string`), and layout helpers from `types/layout.rb`.
+
+### Phase 1: Lexer
+
+**Goal:** Byte-identical token stream vs Ruby lexer.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 1a | `lexer.mt` — character classification, newline/indent handling, line continuation | `lexer_test.mt` |
+| 1b | Identifier and keyword lexing, operator recognition | `lexer_test.mt` |
+| 1c | Number literals (decimal, hex, binary, float, suffixes) | `lexer_test.mt` |
+| 1d | String literals, c-strings, character literals, escapes | `lexer_test.mt` |
+| 1e | Heredocs (plain, c-prefixed, f-prefixed), format strings | `lexer_test.mt` |
+| 1f | Indent/dedent stack, EOF emission, `lex_reporting` with error collection | `lexer_test.mt` |
+
+**Exit criteria:** `mtc lex <file>` produces byte-identical token output vs
+Ruby for `examples/language_baseline.mt` and all test fixtures.
+
+### Phase 2: Parser
+
+**Goal:** Byte-identical AST vs Ruby parser.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 2a | `token_stream.mt` — token stream abstraction | Compiles |
+| 2b | `state.mt` — parser state, diagnostic collection | Compiles |
+| 2c | `literal_parsing.mt` — number/string/char/heredoc literal AST construction | Parser tests |
+| 2d | `ast.mt` — full AST node definitions (SourceFile, Decl, TypeRef, Expr, Stmt) | Compiles |
+| 2e | `parser.mt` — type parsing, attribute parsing | Parser tests |
+| 2f | Expression parsing (precedence climbing), call, member access, specialization | Parser tests |
+| 2g | Statement parsing (let, var, if, while, for, match, return, defer, etc.) | Parser tests |
+| 2h | Declaration parsing (function, struct, enum, variant, interface, etc.) | Parser tests |
+| 2i | Top-level parsing, import resolution, `parse_source_file` | Parser tests |
+| 2j | Error recovery, `parse_collecting_errors` | Parser tests |
+
+**Exit criteria:** `mtc parse <file>` produces byte-identical AST vs Ruby for
+all examples and test fixtures.  Error recovery matches Ruby for malformed input.
+
+### Phase 3: Path Resolver
+
+**Goal:** Platform-specific file resolution, import path → filesystem path
+mapping.  This module has **no dependency on the semantic analyzer** — it only
+uses `std.fs`, `std.path`, `std.string`.  It can be ported immediately after the
+parser, before the full sema stack.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 3a | `path_resolver.mt` — `Platform` enum, `platform_suffix`, `resolve_source_path`, `resolve_module_path`, `infer_module_name` | `path_resolver_test.mt` |
+
+**Exit criteria:** Platform-specific file resolution (`.linux.mt`, `.wasm.mt`)
+works; import path `a.b.c` resolves to correct filesystem path.
+
+### Phase 4: Semantic Analyzer
+
+**Goal:** Type-check all declarations and function bodies.  Error diagnostics
+match Ruby for all examples and test fixtures.  Ruby reference: `semantic_analyzer.rb` (355) + 16 subfiles (10,262) = 10,617 lines.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 4a | `scope.mt` — lexical scope stack, name resolution helpers | `scope_test.mt` |
+| 4b | `diagnostics.mt` — diagnostic struct, formatting | Compiles |
+| 4c | `type_compatibility.mt` — assignability, coercion, narrowing, widening rules | Semantic tests |
+| 4d | `expressions.mt` — type inference for expressions (read/write target resolution) | Semantic tests |
+| 4e | `intrinsics.mt` — built-in callable handler: classifies calls to `fatal`, `ref_of`, `ptr_of`, `read`, `size_of`, `align_of`, `offset_of`, `reinterpret`, `zero`, `default`, `hash`, `equal`, `order`, `get`, `adapt`, etc. and resolves their types | Semantic tests |
+| 4f | `compile_time.mt` — const/const function evaluator: evaluates integer/float/boolean/string literals, unary/binary ops, if-expressions, const function calls, size_of/align_of/offset_of, handles `ReturnValue` early exit | Semantic tests |
+| 4g | `analyzer.mt` — top-level declaration pass (structs, enums, functions, interfaces, events) | Semantic tests |
+| 4h | Function body checking (statements, return type, `?` propagation, guard narrowing) | Semantic tests |
+| 4i | Interface conformance checking, generic instantiation, constraints | Semantic tests |
+| 4j | `emit_expansion.mt` — emit statement lowering within const functions | Semantic tests |
+
+**Exit criteria:** `mtc check <file>` produces byte-identical error diagnostics
+vs Ruby for `examples/language_baseline.mt` and all test fixtures.  Clean
+programs produce zero diagnostics.
+
+### Phase 4a: Control Flow Analysis
+
+**Goal:** Validate control flow correctness.  Ruby reference: `control_flow.rb` (34) +
+`control_flow/` (9 subfiles, 1,148 lines) = 1,182 lines.  Runs after semantic
+analysis on each function body.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 4a1 | `builder.mt` — CFG construction from AST function bodies, block sequencing | Control flow tests |
+| 4a2 | `termination.mt` / `reachability.mt` — return-on-all-paths verification, unreachable code detection | Control flow tests |
+| 4a3 | `definite_assignment.mt` — mutable/immutable tracking, let vs var, guard narrowing with `else:` | Control flow tests |
+| 4a4 | `nullability_flow.mt` — nullable flow narrowing: `if p != null` narrows to non-null in then-branch, `if p == null` narrows in else-branch | Control flow tests |
+| 4a5 | `constant_propagation.mt` — constant propagation for `when`/`inline if` discriminant evaluation | Control flow tests |
+
+**Exit criteria:** Byte-identical control flow diagnostics (missing return,
+unreachable code) vs Ruby for all examples.
+
+### Phase 4b: Module Binder
+
+### Phase 4b: Module Binder
+
+**Goal:** Construct `ModuleBinding` from a completed `Analysis`, filtering
+public/private visibility.  Depends on `analyzer.Analysis` being fully built
+(Phase 4) and control flow validated (Phase 4a).  Ruby reference:
+`module_binder.rb` (187) + `bindings/` (148) = 335 lines.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 4b1 | `binder.mt` — ModuleBinding construction: copies public FnSig, FieldEntry, type aliases, method sigs, interface info, static members, match case names, and implemented interfaces from Analysis into the binding; constructs private_* counterparts from Analysis fields not marked public | Semantic tests |
+
+**Exit criteria:** `ModuleBinding` correctly filters public declarations.
+Cross-module tests produce correct import resolution.
+
+### Phase 4c: Module Loader
+
+**Goal:** Transitive import resolution, dependency ordering, cycle handling.
+Depends on the parser (Phase 2), semantic analyzer (Phase 4), path resolver
+(Phase 3), and binder (Phase 4b).  Ruby reference: `module_loader.rb` (717 lines).
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 4c1 | `module_loader.mt` — `LoadedModule` struct, `LoadDiagnostic` struct, `Program` struct, `check_program()` entry point (parse all imports → topo sort → sema each → bind each → return) | `module_loader_test.mt` |
+| 4c2 | Circular import handling (forward bindings → real analysis → re-check) | `module_loader_test.mt` |
+
+**Exit criteria:** Multi-file programs with circular imports parse and order
+correctly.  `check_program` returns a valid `Program` with all modules analyzed.
+
+### Phase 5: Lowering
+
+**Goal:** AST → IR transformation.  Ruby reference: `lowering.rb` (378) +
+`lowering/` (16 subfiles, 15,201 lines) = 15,579 lines.  Keep as a single
+module initially — sub-file splitting is a code-organization concern that can
+be deferred to post-fixed-point.  The previous working self-host (commit
+`a28545a7`) proved this approach: a 16,498-line `lowering.mt` was correct and
+maintainable.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 5a | `utils.mt` — C name mangling (~20 functions: module, function, enum member, field, value, imported, external C names), type qualification, struct/enum member C name lookup | Lowering tests |
+| 5b | `lowering.mt` — constant and global lowering | Lowering tests |
+| 5c | Type declaration lowering (struct, enum, union, variant, opaque) | Lowering tests |
+| 5d | Expression lowering (literals, binary, call, member access, index, conditional, reinterpret, specialization) | Lowering tests |
+| 5e | Statement lowering (let, var, if, while, for, match, return, defer, unsafe, `?` propagation) | Lowering tests |
+| 5f | Function definition lowering, entry-point synthesis, foreign function boundary | Lowering tests |
+| 5g | `async.mt` — CPS transformation, frame builder, root-main synthesis for async entrypoints | Lowering tests |
+| 5h | Proc closure lowering, dyn interface dispatch, str_buffer lowering, format string lowering | Lowering tests |
+
+**Exit criteria:** `mtc lower <file>` produces byte-identical IR vs Ruby for
+all examples.  The lowering handles `resolved_expr_types` and
+`resolved_call_kinds` from the retained Analysis for precise type resolution.
+
+### Phase 6: C Backend
+
+**Goal:** IR → C source.  Ruby reference: `c_backend.rb` (322) + `c_backend/`
+(12 subfiles, 5,826 lines) = 6,148 lines.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 6a | Preamble emission (includes, conditional helpers, runtime helpers), C keyword table (`C_KEYWORDS`) and `sanitize_c_identifier` — every field/member name emitted to C passes through this; must be called at struct/union/variant field declarations, member access, `offsetof`, designated initializers, and equality helpers — all 18 sites (see `type_declaration.rb`, `expressions.rb`, `runtime_helpers.rb`) | C backend tests |
+| 6b | Type declaration emission (enum, flags, struct, union, variant, opaque, type aliases) | C backend tests |
+| 6c | Expression emission (name, literal, binary, call, member, index, reinterpret) | C backend tests |
+| 6d | Statement emission (block, if, switch/match, while, for, return, defer, goto/label) | C backend tests |
+| 6e | Function definition emission, format string lowering, simd lowering | C backend tests |
+| 6f | Prelude type emission (Option/Result structs, nullable opt types), aggregate field construction | C backend tests |
+| 6g | String literal constant section, per-type equality helpers, variant equality helpers | C backend tests |
+
+**Exit criteria:** `mtc emit-c <file>` produces byte-identical C vs Ruby for
+`examples/language_baseline.mt` and all examples.
+
+### Phase 6a: Pretty Printer
+
+**Goal:** AST → formatted text and IR → text for debugging and testing.  Ruby
+reference: `pretty_printer.rb` (17) + `pretty_printer/` (3 subfiles, 1,292
+lines) = 1,309 lines.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 6a1 | `ast_formatter.mt` — AST → Milk Tea source text (round-trip formatting) | `ast_formatter_test.mt` |
+| 6a2 | `ir_formatter.mt` — IR → text for `mtc lower` output | Diff vs Ruby `mtc lower` |
+
+**Exit criteria:** `mtc parse file.mt | mtc format` round-trips identically.
+`mtc lower` IR text output is byte-identical vs Ruby.
+
+### Phase 7: Build System and CLI
+
+**Goal:** A working compiler binary that builds itself.  Ruby reference:
+`tooling/cli.rb` (~900 lines), `cli/commands/` (22 files), `build.rb`,
+`build_cache.rb`, `run.rb`.
+
+**CLI dispatch** — a flat `if cmd == ...` chain in `main.mt` routing to handler
+functions.  Manual argv iteration, no option-parsing library.  Each sub-command
+(`lex`, `parse`, `check`, `lower`, `emit-c`, `build`, `run`, `test`, `format`,
+`debug`, `lint`) is a function that parses its arguments from a `span[str]`.
+
+| Step | Module | Verify |
+|------|--------|--------|
+| 7a | `main.mt` — CLI dispatch for `lex --sexpr`, `parse --sexpr`, `check`, `lower`, `emit-c` (no-build verification commands) | Diff S-expr output vs Ruby |
+| 7b | `build.mt` — process spawning for CC, cache integration, linker flags, platform detection | Self-build |
+| 7c | `build_cache.mt` — incremental compilation cache, content-hash keying, artifact storage | Unit tests |
+| 7d | `main.mt` (continued) — `build`, `run`, `test`, `format`, `debug`, `lint` sub-commands | CLI smoke tests |
+| 7e | `version_info.mt` — git revision stamp placeholder (replaced by `sed` in bootstrap script) | Bootstrap script |
+
+**Exit criteria:** Stage1 builds and produces working `build/stage1/mtc`.
+
+### Phase 8: Bootstrap and Fixed-Point
+
+**Goal:** The compiler is self-consistent.
+
+| Step | Verify |
+|------|--------|
+| 8a | `tools/bootstrap.sh` — 3-stage build script | Stage2 builds |
+| 8b | Fixed-point check: `diff stage2.c stage3.c` is empty | Byte-identical |
+| 8c | Self-test: stage2 builds itself to a working binary | Binary runs |
+| 8d | Test suite passes under stage2: `build/stage2/mtc test projects/mtc` | All green |
+
+**Exit criteria:** Stage2.c == stage3.c.  Stage2 self-builds.  Stage2 passes
+all self-host tests.
+
+### Phase 9+: LSP, DAP, Bindgen, Linter (Post-Fixed-Point)
+
+These subsystems can be added incrementally without breaking the fixed point:
+
+| Subsystem | Modules | Lines (est.) |
+|-----------|---------|-------------|
+| LSP server | `mtc/lsp/` (15–20 modules) | ~5,000 |
+| DAP server | `mtc/dap/` (5–7 modules) | ~1,000 |
+| Bindgen | `mtc/bindgen.mt` | ~1,000 |
+| Imported bindings | `mtc/imported_bindings/` (5–7 modules) | ~3,000 |
+| Linter | `mtc/linter/` (6–8 modules) | ~5,000 |
+| Pretty printer (AST/IR) | `mtc/pretty_printer/` | ~2,000 |
+| Packages subsystem | `mtc/packages/` | ~3,000 |
+
+## 5. Key Technical Decisions
+
+### 5.1 Lowering: Single Module Initially
+
+The lowering pass is the largest and most interconnected component (15,579 lines
+in 16 Ruby sub-files).  Start as a single `lowering/lowering.mt` module.
+Sub-file splitting is a code-organization concern that can be deferred to
+post-fixed-point.  The previous working self-host (commit `a28545a7`) proved
+this approach: a 16,498-line `lowering.mt` was correct and maintainable.
+
+### 5.2 Type System: Conservative Analyzer, Structural Lowering
+
+The semantic analyzer is intentionally conservative: anything it cannot resolve
+degrades to `ty_error` (compatible with everything).  The lowering walks the
+AST directly with access to retained `Analysis` data (function signatures,
+`resolved_expr_types`, `resolved_call_kinds`).  When the analyzer's permissive
+types are insufficient, the lowering resolves types structurally.
+
+This two-tier approach means the analyzer catches structural errors (duplicate
+declarations, immutable-assignment violations, module inconsistencies) while
+the lowering handles the precise type resolution needed for C code generation.
+
+### 5.3 IR as Pure Data
+
+The IR (`ir.mt`) contains no lowering or emission logic.  It is a frozen shape
+of variant types that both the Lowering and CBackend depend on.  This
+decoupling means neither stage depends on the other's internals — they share
+only the IR contract.
+
+### 5.4 String Slices, Not Copies
+
+The lexer produces token lexemes as `str` slices borrowing from the source
+buffer.  The parser constructs AST identifiers and string literals as `str`
+slices into the same buffer.  No string heap-allocation during lexing or
+parsing.  Only cross-module names (diagnostic messages, binding keys) need
+owning `string.String` values.
+
+### 5.5 No Exception Handling — Result Types
+
+Milk Tea has no exceptions.  Every fallible operation returns `Result[T, E]`
+or `Option[T]`.  The `?` operator propagates failures.  `let ... else:`
+provides early-exit guard clauses.  This is more verbose than Ruby's
+`raise`/`rescue` but structurally identical — every `raise` in Ruby maps to
+`return Result.failure`, and every `rescue` maps to `match` on the result.
+
+Error collection (e.g., module_loader's collect-and-continue pattern) uses
+explicit `Vec[Diagnostic]` accumulation rather than rescued exceptions.
+
+### 5.6 No Ruby Metaprogramming Equivalents
+
+The Ruby compiler uses:
+- `Data.define` → Milk Tea `struct` + `extending` with default parameter values
+- `case`/`when` on AST types → `match` on variant arms
+- `Hash` with symbol keys → `Map[str, V]` with string keys
+- `.each`/`.map`/`.select` → `for` loops or `iter` chains
+- `require`/`require_relative` → `import` statements
+- `gsub`/`split` → `std.str` operations and PCRE2 for regex cases (~59 occurrences)
+- String interpolation `"#{x}"` → `f"#{x}"` format strings
+
+### 6.7 Process Spawning
+
+`std.process.mt` provides `ChildProcess` for spawning CC and linker
+invocations.  `std.stdio.mt` wraps C `FILE*` for reading/writing files.
+Platform-specific file-system operations use `std/fs.linux.mt` /
+`std/fs.windows.mt` (resolved at import time by the platform suffix mechanism).
+
+### 6.8 CLI Dispatch
+
+The Ruby CLI (`tooling/cli.rb`, ~900 lines) uses a flat `case command` dispatch
+with manual `$stdin`-less argument parsing.  Each sub-command module (22 files
+under `cli/commands/`) is included into the CLI class.  The self-host flattens
+this into `main.mt`: a single `main(args: span[str])` function with an
+`if cmd == "lex": ... else if cmd == "parse": ...` dispatch chain, each branch
+calling into a dedicated handler function.
+
+**Argument parsing** iterates through `args` manually with a mutable counter —
+no option-parsing library:
+
+```mt
+function build_command(args: span[str]) -> int:
+    var roots = vec.Vec[str].create()
+    defer: roots.release()
+    var c_compiler = "cc"
+    var output_override: Option[str] = Option[str].none
+    var use_cache = true
+    var ai: ptr_uint = 1
+    while ai < args.len:
+        let arg = args[ai]
+        if arg == "-o":
+            output_override = Option[str].some(value = args[ai + 1])
+            ai += 2
+            continue
+        if arg == "--no-cache":
+            use_cache = false
+            ai += 1
+            continue
+        # positional operand
+        roots.push(arg)
+        ai += 1
+    # ... use parsed values ...
+```
+
+Every `Vec`, `String`, `Map` allocation uses `defer x.release()` for cleanup.
+Error return paths use `let x = expr else: return 1` guards.  Format output uses
+`stdio.print_format(c"format %.*s\n", int<-(s.len), s.data)` — the `%.*s`
+C-printf format for non-null-terminated `str` values.
+
+The shared CLI conventions are:
+- `-I DIR` / `--root DIR` — module search roots (repeatable)
+- `--platform linux|windows|wasm` — target platform
+- `--sexpr` — S-expression output (lex, parse, lower commands); this replaces
+  the `--machine` flag from the previous self-host at commit `a28545a7`
+- `--no-cache` / `--keep-c PATH` / `-o OUTPUT` / `--cc CC` — build flags
+- `--clean` / `--sanitize` / `--profile debug|release` — build options
+- `--timeout SECONDS` / `--mem MB` / `--format human|tap|junit` — test flags
+- Trailing `--` separates compiler flags from program arguments (run command)
+- `-Werror` — treat warnings as errors (check command)
+- `--locked` / `--frozen` — accepted for CLI compatibility but no-ops until
+  package-graph resolution is implemented
+
+**Build flow** (`build.mt`): `module_loader.check_program()` → `lowering.lower()`
+→ `c_backend.generate_c()` → `fs.write_text()` → build CC argument list as a
+`Vec[str]` → `process.capture(command.as_span())` → check exit code.
+
+**Test flow** (`main.mt` test_command): discover `@[test]` functions in
+discovered `.mt` files → build each test file in-process → spawn each test
+binary sandboxed with memory/timeout limits → parse exit code (0 = pass,
+non-zero = fail, 124 = timeout, 137 = OOM).
+
+### 6.9 S-expression Verification Format
+
+The self-host's primary verification mechanism is byte-identical S-expression
+output against the Ruby compiler.  The `mtc lex --sexpr`, `mtc parse --sexpr`,
+and `mtc lower` commands produce S-expression text that must match Ruby's
+`SexprDumper` output character-for-character.
+
+Key serialization invariants that must be preserved:
+
+```mtc lex --sexpr file.mt
+(token :type :identifier :lexeme "main" :literal nil :line 1 :column 0 ...)
+```
+
+```mtc parse --sexpr file.mt
+(SourceFile :module_name "test" :module_kind :module
+  :declarations ((FunctionDef :name "main" ...)))
+```
+
+```mtc lower file.mt
+(Program :module_name "test"
+  :includes ((Include :header "...")))
+```
+
+Critical formatting rules:
+- Token types use hyphens (`plus-equal`, `shift-left`) not underscores
+- Float literals use the lexeme string (the raw source text, not the parsed
+  `double` value), to preserve precision and avoid platform-specific rounding
+- String escaping: `\\` for backslash, `\"` for quote, `\n` `\t` `\r` for control chars
+- Types objects render as `(Types::ShortName :field value ...)`
+- Symbols render as `:name` with hyphens
+- Hashes and Sets are silently skipped (not serialized)
+- `nil`, `true`, `false` render as bare atoms
+
+## 6. Risk Register
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|-----------|--------|------------|
+| Arena size underestimation causing OOM | Medium | High | Start with 16 MB arenas; add growth/fallback to heap allocation |
+| Type resolution bugs surfaced only at C emission | High | Medium | `ty_error` is permissive; structural lowering resolves at emit time |
+| Map/Vector iterator invalidation during mutable iteration | Medium | High | Two-pass algorithms: collect keys first, then mutate |
+| Order-dependent lowering (module load order affects type resolution) | Medium | High | Always look up types in the owning module first; fall back to global scan |
+| C keyword sanitization gaps | High | Low | Centralize sanitization in `c_backend.mt` as a single `sanitize_c_identifier` function + `C_KEYWORDS` constant; every emission site (declarations, member access, offsetof, initializers, equality helpers) must call it — the Ruby compiler at commit `8f743bc6` defines the exhaustive call-site list of 18 locations across `type_declaration.rb`, `expressions.rb`, and `runtime_helpers.rb` |
+| Fixed-point breakage from non-deterministic map iteration | Medium | Critical | Use `LinkedMap` where insertion order matters; sort keys otherwise |
+| Compiler performance degradation vs Ruby | High | Medium | Arena allocation is faster than GC; string slices eliminate copies; profile after fixed point |
+| Debugging C output from compiler bugs | High | Medium | `--keep-c` saves C for inspection; `diff` against Ruby C output pinpoints divergence |
+
+## 7. Verification Strategy
+
+### 7.1 Per-Phase Verification
+
+Every phase produces testable output via `--sexpr`.  The Ruby compiler serves as
+the oracle.  The previous self-host at `a28545a7` used a `--machine` flag for
+human-readable token tables; the new self-host uses Ruby's current `--sexpr`
+S-expression format, which produces deterministic, diffable output:
+
+| Phase | Test Command | Oracle |
+|-------|-------------|--------|
+| Lexer | `diff <(ruby mtc lex --sexpr file.mt) <(stage1/mtc lex --sexpr file.mt)` | Ruby SexprDumper |
+| Parser | `diff <(ruby mtc parse --sexpr file.mt) <(stage1/mtc parse --sexpr file.mt)` | Ruby SexprDumper |
+| Semantic | `diff <(ruby mtc check file.mt 2>&1) <(stage1/mtc check file.mt 2>&1)` | Ruby sema diagnostics |
+| Lowering | `diff <(ruby mtc lower file.mt) <(stage1/mtc lower file.mt)` | Ruby SexprDumper IR |
+| C Backend | `diff <(ruby mtc emit-c file.mt) <(stage1/mtc emit-c file.mt)` | Ruby C backend |
+| Build | `diff stage2.c stage3.c` | Stage2 C output |
+| Runtime | Run compiled examples; compare stdout and exit code | Ruby-compiled binary |
+
+### 7.2 Language Baseline
+
+`examples/language_baseline.mt` (1,935 lines) exercises the complete language
+surface.  It is the primary integration test.  Every phase must produce
+byte-identical output vs Ruby for this file.
+
+### 7.3 Test Suite
+
+The self-host test suite mirrors the Ruby test suite's coverage:
+
+| Test file | Scope |
+|-----------|-------|
+| `lexer_test.mt` | Token stream correctness, error recovery, edge cases |
+| `parser_test.mt` | AST structure, error recovery, all declaration/expression/statement forms |
+| `semantic_test.mt` | Type checking, interface conformance, generics, control flow |
+| `lowering_test.mt` | IR structure, C name mangling, lowering passes |
+| `c_backend_test.mt` | C output correctness, preamble, type emission |
+| `module_loader_test.mt` | Multi-file programs, circular imports, platform resolution |
+| `path_resolver_test.mt` | Platform-specific file resolution |
+| `ast_formatter_test.mt` | AST → text round-trip |
+| `scope_test.mt` | Lexical scope resolution |
+| `linter_test.mt` | Lint rule detection and auto-fixes |
+| `fix_engine_test.mt` | Text edit engine |
+| `completion_test.mt` | LSP completion |
+
+## 8. Standard Library Dependencies
+
+The self-host compiler depends on the following `std` modules (all present in
+the current standard library):
+
+| Module | Used for |
+|--------|---------|
+| `std.mem.arena` | Per-compilation arena allocation |
+| `std.mem.heap` | Owning heap allocation (Vec backing, Map backing) |
+| `std.vec` | Dynamic arrays (token lists, declaration spans, argument lists) |
+| `std.map` | Symbol tables, type caches, binding maps |
+| `std.string` | Owned string building (diagnostics, C output, cross-module names) |
+| `std.str` | String view operations (equality, starts_with, ends_with, slicing) |
+| `std.fmt` | Format string building, integer/float rendering |
+| `std.fs` | File system (read source, write C output, discover test files) |
+| `std.path` | Path manipulation (join, basename, extension stripping) |
+| `std.process` | Process spawning (CC invocation, linker) |
+| `std.stdio` | C FILE* I/O, printf for debug output |
+| `std.terminal` | Colored terminal output for diagnostics |
+| `std.hash` | `hash[str]` and `equal[str]` for Map keying |
+| `std.iter` | Composable iteration over collections |
+| `std.json` | LSP/DAP protocol serialization (post-fixed-point) |
+| `std.pcre2` | Regular expressions for version parsing (post-fixed-point) |
+| `std.testing` | Test framework (`@[test]`, assertions) |
+
+## 9. Configuration
+
+### 9.1 Package Manifest
+
+```toml
+# projects/mtc/package.toml
+[package]
+name = "mtc"
+version = "0.1.0"
+source_root = "src"
+
+[profile]
+default = "debug"
+
+[platform]
+default = "linux"
+
+[build]
+entry = "src/mtc/main.mt"
+```
+
+### 9.2 Bootstrap Environment
+
+| Variable | Purpose | Default |
+|----------|---------|---------|
+| `MTC_BOOTSTRAP` | Path to stage0 mtc binary | auto-detect |
+| `CC` | C compiler for native builds | `cc` |
+| `MTC_BUILD_DIR` | Build output directory | `build/` |
+| `XDG_CACHE_HOME` | Cache root for build cache | `~/.cache` |