mt-lang 0.3.26 → 0.3.27

data/docs/self-host-contract.md

@@ -0,0 +1,1091 @@
+# Self-Host Contract
+
+The exact public API surface this document describes is **inferred from the Ruby
+host compiler** (`lib/milk_tea/core/`).  A previous working self-host (commit
+`a28545a7`) demonstrated that this contract produces a correct, fixed-point
+compiler.  Do not deviate from it without verifying that the Ruby host does the
+same thing.
+
+---
+
+## 0. Import Dependency Graph
+
+Modules are listed in **topological dependency order** — a module may only
+import modules that appear earlier in the graph.  This guarantees zero circular
+imports within the compiler itself.
+
+```
+                      ┌─────────────────────┐
+                      │  token_kinds (enum) │  ← zero imports, always the root
+                      └────────┬────────────┘
+         ┌─────────────────────┼─────────────────────┐
+         │                     │                     │
+    ┌────┴────┐          ┌─────┴─────┐       ┌───────┴───────┐
+    │  token  │          │ keywords  │       │  ast (types)  │
+    │ structs │          │ lookup    │       │  SourceFile,  │
+    └────┬────┘          └───────────┘       │  Decl, Expr,  │
+         │                                   │  Stmt, ...    │
+         │                                   └───────┬───────┘
+         │                                           │
+    ┌────┴──────────────────────────────────────┐    │
+    │              lexer                        │    │
+    │  source → tokens, indent/dedent, heredocs │    │
+    └───────────────────┬───────────────────────┘    │
+                        │                            │
+    ┌───────────────────┼────────────────────────────┘
+    │                   │
+    │           ┌───────┴───────┐
+    │           │ parser.state  │  (diagnostic types)
+    │           └───────┬───────┘
+    │                   │
+    │           ┌───────┴──────────┐
+    │           │    parser        │
+    │           │ tokens → ast     │
+    │           └───────┬──────────┘
+    │                   │
+    │    ┌──────────────┼─────────────┐
+    │    │              │             │
+    │    │    ┌─────────┴─────────┐   │
+    │    │    │  semantic.types   │   │
+    │    │    │  Type variant     │   │
+    │    │    └─────────┬─────────┘   │
+    │    │              │             │
+    │    │    ┌─────────┴─────────┐   │
+    │    │    │  ir (IR types)    │   │  ← depends on semantic.types
+    │    │    └─────────┬─────────┘   │
+    │    │              │             │
+    │    │    ┌─────────┴─────────┐   │
+    │    │    │  sexpr_dumper     │   │  ← depends on ast, token, types, ir
+    │    │    └─────────┬─────────┘   │
+    │    │              │             │
+    │    │    ┌─────────┴─────────────┴┐
+    │    │    │   semantic.scope       │
+    │    │    └─────────┬──────────────┘
+    │    │              │
+    │    │    ┌─────────┴──────────────┐
+    │    │    │   path_resolver        │
+    │    │    │   platform resolution  │
+    │    │    └─────────┬──────────────┘
+    │    │              │
+    │    │    ┌─────────┴───────────────────────────────┐
+    │    │    │           semantic.analyzer             │
+    │    │    │  check_module, Analysis, ModuleBinding  │
+    │    │    └─────────┬───────────────────────────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  control_flow.*   │  (depends on analyzer.Analysis)
+    │    │    └─────────┬─────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  loader.binder    │
+    │    │    └─────────┬─────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  module_loader    │
+    │    │    │  orchestration    │
+    │    │    └─────────┬─────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  compile_time     │  const/const-func eval
+    │    │    └─────────┬─────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  lowering.utils   │  C name mangling, type helpers (needs types, ast, ir)
+    │    │    └─────────┬─────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  lowering.main    │
+    │    │    │  AST → IR         │
+    │    │    └─────────┬─────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  lowering.async   │
+    │    │    └─────────┬─────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  c_backend        │
+    │    │    │  IR → C source    │
+    │    │    └─────────┬─────────┘
+    │    │              │
+    │    │    ┌─────────┴─────────┐
+    │    │    │  pretty_printer   │
+    │    │    └───────────────────┘
+    │    │
+    │    │    ┌───────────────────────┐
+    │    └────┤  build / build_cache  │
+    │         └───────────┬───────────┘
+    │                     │
+    │       ┌─────────────┴─────────────┐
+    │       │  main (CLI dispatch)      │
+    │       │  lex / parse / check /    │
+    │       │  lower / emit-c / build   │
+    │       │  run / test / format      │
+    │       └───────────────────────────┘
+```
+
+### 0.1 Ownership Cheat Sheet
+
+| Data structure | Allocator | Passed as | Freed by |
+|---------------|-----------|-----------|----------|
+| `Token` | arena | value (copy by stack) | arena reset |
+| `TokenKind` | — (enum) | value | — |
+| `Ast.SourceFile` | arena (heap nodes via `ptr[]`) | `ref` or `ptr` | arena reset |
+| `Ast.Expr` / `Ast.Stmt` (recursive) | arena (`ptr[Expr]`) | `ptr` | arena reset |
+| `Types.Type` | arena (`ptr[Type]`) | `ptr` | arena reset |
+| `IR.Program` | arena (heap nodes) | ref or owned | arena reset |
+| `IR.Expr` / `IR.Stmt` (recursive) | arena (`ptr[Expr]`) | `ptr` | arena reset |
+| `Analysis` (per-module sema result) | loader's `Vec` (heap) | `ptr` (borrowed from loader) | loader's `Program.release()` |
+| `ModuleBinding` (per-module export surface) | loader's `Map` (heap) | embedded in Analysis | loader's `Program.release()` |
+| `Program` (loader result) | heap (`Vec` + `Map`) | value/ref | caller's `release()` |
+| `string.String` (diagnostics, C output) | heap | value, `ref` to extend | caller's `release()` |
+| `Vec[T]` (token lists, decl spans) | heap | value, `ref` to push/iter | caller's `release()` |
+| `Map[K,V]` (symbol tables) | heap | value, `ref` to lookup/insert | caller's `release()` |
+| `str` (source slices, keys) | borrows from source buffer | value (copy of fat ptr) | outlives source buffer |
+| `span[T]` (slices of IR / decls) | borrows from owning Vec | value (copy of fat ptr) | outlives owning Vec |
+
+### 0.2 Module Naming Convention
+
+| Convention | Meaning |
+|-----------|---------|
+| `mtc.lexer.token` | Module path, same as `import` statement |
+| `token.Token` | Type `Token` from module aliased as `token` |
+| `vec.Vec[T]` | Generic `Vec` from module aliased as `vec` |
+
+---
+
+## 1. Root-Level Data Modules
+
+These modules contain **zero logic** — only struct, variant, enum, and flag
+definitions.  They can be ported by copying the type definitions exactly.
+
+### 1.1 `mtc.lexer.token_kinds` (TokenKind enum)
+
+Imports: **none**.
+
+The enum matches the Ruby compiler's token types exactly.  **All keyword
+members use the `tk_` prefix** to avoid collisions with Milk Tea's own reserved
+words (e.g., `tk_if` not `if`, `tk_function` not `function`).  The parser
+dispatches on `TokenKind` values.
+
+See `lib/milk_tea/core/token.rb` `KEYWORDS` and the old self-host's
+`projects/mtc/src/mtc/lexer/token_kinds.mt` at commit `a28545a7` for the full
+member list.  The exact numeric values are not reproduced here — they are
+irrelevant to the contract.
+
+### 1.2 `mtc.lexer.token` (Token types)
+
+Imports: `mtc.lexer.token_kinds`.
+
+```mt
+public struct TriviaToken:
+    kind: token_kinds.TokenKind
+    text: str           # slice into source
+    line: ptr_uint
+    column: ptr_uint
+    start_offset: ptr_uint
+    end_offset: ptr_uint
+
+public struct Token:
+    kind: token_kinds.TokenKind
+    lexeme: str         # slice into source
+    literal: Option[TokenLiteral]
+    line: ptr_uint
+    column: ptr_uint
+    start_offset: ptr_uint
+    end_offset: ptr_uint
+    leading_trivia: span[TriviaToken]
+    trailing_trivia: span[TriviaToken]
+
+public variant TokenLiteral:
+    integer_literal(value: long, raw: str)
+    float_literal(value: double, raw: str, suffix: Option[str])
+    string_literal(value: str)
+    char_literal(value: ubyte)
+    boolean_literal(value: bool)
+
+public struct LexDiagnostic:
+    message: cstr
+    line: ptr_uint
+    column: ptr_uint
+```
+
+### 1.3 `mtc.lexer.keywords` (Keyword lookup)
+
+Imports: `mtc.lexer.token_kinds`.
+
+```mt
+# Returns token_kinds.TokenKind for a keyword string, or Option.none
+public function lookup(word: str) -> Option[token_kinds.TokenKind]:
+```
+
+### 1.4 `mtc.ast` (AST types)
+
+Imports: **none** (no other mtc modules — may use `std.vec`, `std.str`).
+
+The AST is a direct 1:1 translation of `lib/milk_tea/core/ast.rb`.  Every
+`Data.define` becomes a `struct`; every `case`/`when` discrimination on the
+Ruby class type becomes a `variant` arm.  Recursive nodes (`Expr`, `Stmt`,
+`TypeRef`) use `ptr[Expr]` / `ptr[Stmt]` / `ptr[TypeRef]` for heap indirection
+within the arena.
+
+Key top-level types:
+
+```mt
+public struct QualifiedName:
+    module_name: Option[str]
+    name: str
+
+public struct SourceFile:
+    module_name: str
+    module_kind: ModuleKind
+    directives: span[Directive]
+    declarations: span[Decl]
+
+public enum ModuleKind: ubyte
+    module_normal = 0
+    module_external = 1
+
+public variant Directive:
+    importing(module_path: QualifiedName, alias: Option[str])
+    directive_include(header: str)
+    directive_link(library: str)
+    directive_compiler_flag(flag: str)
+
+# Decl, Expr, Stmt, TypeRef, Param, Field, MethodDef, etc. variants
+# match ast.rb node-for-node — see that file for the complete list.
+```
+
+**Ownership:** Every `ptr[Expr]` / `ptr[Stmt]` / `ptr[TypeRef]` points into the
+arena.  The parser allocates them via the arena; the lowering reads them; the
+arena reset frees them.  All `span[Decl]`, `span[Field]`, `span[Param]` are
+slices borrowed from arena-allocated Vecs.
+
+### 1.5 `mtc.ir` (IR types)
+
+Imports: `mtc.semantic.types` (for `types.Type`), `mtc.ast` (for `ast.ModuleKind`).
+
+Pure data — no logic.  Mirrors `lib/milk_tea/core/ir.rb` exactly.
+
+```mt
+public struct Include:
+    header: str
+
+public struct Field:
+    name: str
+    ty: types.Type
+
+public struct Param:
+    name: str
+    linkage_name: str
+    ty: types.Type
+    pointer: bool
+
+public struct EnumMember:
+    name: str
+    linkage_name: str
+    value: ptr[Expr]
+
+public struct AggregateField:
+    name: str
+    value: ptr[Expr]
+
+public struct VariantArm:
+    name: str
+    linkage_name: str
+    fields: span[Field]
+
+public struct SwitchCase:
+    is_default: bool
+    value: ptr[Expr]?
+    body: span[Stmt]
+
+# Expression variant — ~30 arms, match ir.rb exactly
+public variant Expr:
+    expr_name(name: str, ty: types.Type, pointer: bool)
+    expr_member(receiver: ptr[Expr], member: str, ty: types.Type)
+    expr_call(callee: str, arguments: span[Expr], ty: types.Type)
+    expr_binary(operator: str, left: ptr[Expr], right: ptr[Expr], ty: types.Type)
+    expr_integer_literal(value: long, ty: types.Type)
+    expr_string_literal(value: str, ty: types.Type, cstring: bool)
+    # ... (complete list in ir.rb)
+
+# Statement variant — ~15 arms, match ir.rb exactly
+public variant Stmt:
+    stmt_block(body: span[Stmt])
+    stmt_if(condition: ptr[Expr], then_body: span[Stmt], else_body: span[Stmt])
+    stmt_return(value: ptr[Expr]?, line: ptr_uint, source_path: str)
+    # ... (complete list in ir.rb)
+
+public struct Program:
+    module_name: str
+    module_kind: ast.ModuleKind
+    includes: span[Include]
+    constants: span[Constant]
+    globals: span[Global]
+    opaques: span[OpaqueDecl]
+    structs: span[StructDecl]
+    unions: span[UnionDecl]
+    enums: span[EnumDecl]
+    variants: span[VariantDecl]
+    static_asserts: span[StaticAssert]
+    type_aliases: span[TypeAlias]
+    functions: span[Function]
+```
+
+### 1.6 `mtc.sexpr_dumper` (S-expression serializer)
+
+Imports: `mtc.lexer.token`, `mtc.ast`, `mtc.semantic.types`, `mtc.ir`.
+
+```mt
+public function dump_tokens(tokens: span[token.Token]) -> str:
+    # Returns "(token :type :plus :lexeme "+" ...)" per token
+
+public function dump_ast(source_file: ast.SourceFile) -> str:
+    # Returns "(SourceFile :module_name ...)" S-expression
+
+public function dump_ir(program: ir.Program) -> str:
+    # Returns "(Program ...)" S-expression
+
+public function dump_type(type: types.Type) -> str:
+    # Returns "(Types::Primitive :name "int")" S-expression
+
+# Internal helpers (not public, but must exist for the above):
+#  emit_token(token, buf)
+#  emit_data(node, buf)  — dispatches on Data members
+#  emit_value(value, buf) — boolean, integer, float, string, symbol, nil
+#  emit_string(value, buf) — escapes \ " \n \t \r
+#  format_float(value) — sprintf("%g") with ".0" suffix for whole numbers
+#  TYPES_SHORT_NAMES — Map[str, str] for type class → short name
+#  emit_types_object(type, buf) — dispatches on type variant arms
+#  emit_types_fields(type, buf) — per-arm field emission
+```
+
+**Critical invariants** (must match Ruby byte-for-byte):
+- Token kinds use hyphens (`plus-equal` not `plus_equal`)
+- Float literals use raw lexeme (not parsed `double`)
+- String escaping: `\\` `\"` `\n` `\t` `\r`
+- Symbols: `:name` with hyphens
+- Hashes and Sets: silently skipped (not serialized)
+- `nil`, `true`, `false`: bare atoms, no quoting
+
+---
+
+## 2. Lexer (`mtc.lexer.lexer`)
+
+Imports: `mtc.lexer.token_kinds`, `mtc.lexer.token`, `mtc.lexer.keywords`, `std.vec`.
+
+No other mtc modules.  Self-contained: `source → Vec[Token]`.
+
+### 2.1 Public API
+
+```mt
+# Lex source text, calling fatal() on hard errors
+public function lex(source: str, path: str) -> vec.Vec[token.Token]:
+
+# Lex source text, collecting recoverable errors in diagnostics
+public function lex_reporting(source: str, diagnostics: ref[vec.Vec[token.LexDiagnostic]], path: str) -> vec.Vec[token.Token]:
+```
+
+### 2.2 Internal Architecture
+
+The old self-host used a `LexSession` struct holding mutable state:
+
+```mt
+struct LexSession:
+    tokens: vec.Vec[token.Token]
+    indent_stack: vec.Vec[ptr_uint]
+    diagnostics: ptr[vec.Vec[token.LexDiagnostic]]?
+    continuation_pending: bool
+    grouping_depth: int
+    source: str
+    src_len: ptr_uint
+    pos: ptr_uint
+```
+
+Key internal functions:
+- `is_identifier_start_byte(b: ubyte) -> bool`
+- `is_identifier_part_byte(b: ubyte) -> bool`
+- `is_digit_byte(b: ubyte) -> bool`
+- `is_hex_digit_byte(b: ubyte) -> bool`
+- `lex_lines(session: ref[LexSession]) -> void` — main loop
+- `lex_line(session: ref[LexSession]) -> void` — single line
+- `lex_whitespace(session: ref[LexSession]) -> void` — leading whitespace + indent/dedent
+- `lex_identifier_or_keyword(session: ref[LexSession]) -> void`
+- `lex_number(session: ref[LexSession]) -> void`
+- `lex_string(session: ref[LexSession], kind: CStringKind) -> void`
+- `lex_char_literal(session: ref[LexSession]) -> void`
+- `lex_heredoc(session: ref[LexSession], prefix: HeredocPrefix) -> void`
+- `lex_format_interpolation(session: ref[LexSession]) -> void`
+- `lex_operator(session: ref[LexSession]) -> void`
+- `lex_comment(session: ref[LexSession]) -> void`
+- `emit_eof(session: ref[LexSession]) -> vec.Vec[token.Token]`
+- `add_token(session: ref[LexSession], kind: token_kinds.TokenKind, lexeme: str, literal: Option[token.TokenLiteral]) -> void`
+
+### 2.3 Ownership
+
+- `source: str` — borrows from caller's arena or `string.String`
+- Returned `vec.Vec[token.Token]` — **owned**.  Caller must `release()`.
+- `token.Token.lexeme` — `str` slice into the source buffer (no copy)
+- `token.Token.leading_trivia` / `trailing_trivia` — `span[token.TriviaToken]` slices into the trivia Vec owned by the lexer session
+- `trivia` Vec data — arena-allocated
+- Diagnostic `cstr` messages — static string literals (lifetime = program)
+
+---
+
+## 3. Parser (`mtc.parser.parser`)
+
+Imports: `mtc.lexer.token_kinds`, `mtc.lexer.token`, `mtc.ast`, `mtc.parser.state`.
+
+### 3.1 `mtc.parser.state` (Parser state types)
+
+Imports: `mtc.lexer.token_kinds`, `mtc.lexer.token`, `std.vec`.
+
+```mt
+public struct ParseDiagnostic:
+    line: ptr_uint
+    column: ptr_uint
+    lexeme: str
+    kind: str
+    message: str
+
+public struct ParserState:
+    tokens: span[token.Token]
+    pos: ptr_uint
+    diagnostics: vec.Vec[ParseDiagnostic]
+
+# Token stream helpers
+public function current(state: ref[ParserState]) -> token.Token:
+public function advance(state: ref[ParserState]) -> void:
+public function peek(state: ref[ParserState]) -> token.Token:
+public function check(state: ref[ParserState], kind: token_kinds.TokenKind) -> bool:
+public function expect(state: ref[ParserState], kind: token_kinds.TokenKind) -> Result[token.Token, ParseDiagnostic]:
+public function skip_newlines(state: ref[ParserState]) -> void:
+```
+
+### 3.2 `mtc.parser.parser` (Parser)
+
+```mt
+# Parse to AST only — no error recovery (for tools that need just the tree)
+public function parse(source: str, path: str) -> ast.SourceFile:
+
+# Parse with error collection — continues past errors
+public function parse_collecting_errors(source: str, path: str) -> ParseResult:
+
+public struct ParseResult:
+    ast: ast.SourceFile
+    diagnostics: vec.Vec[state.ParseDiagnostic]
+```
+
+Key internal parsing functions (each returns a struct/ptr for its grammar rule):
+- `parse_source_file(state: ref[ParserState]) -> ast.SourceFile`
+- `parse_declaration(state: ref[ParserState]) -> ast.Decl`
+- `parse_function_def(state, is_public, is_async, is_const, attributes) -> ast.Decl`
+- `parse_struct_def(state, attributes) -> ast.Decl`
+- `parse_enum_def(state, attributes) -> ast.Decl`
+- `parse_type_ref(state) -> ptr[ast.TypeRef]`
+- `parse_expression(state) -> ptr[ast.Expr]`
+- `parse_expression_precedence(state, min_precedence) -> ptr[ast.Expr]`
+- `parse_primary(state) -> ptr[ast.Expr]`
+- `parse_call(state, callee) -> ptr[ast.Expr]`
+- `parse_statement(state) -> ptr[ast.Stmt]`
+- `parse_if_statement(state) -> ptr[ast.Stmt]`
+- `parse_match_statement(state) -> ptr[ast.Stmt]`
+- `parse_while_loop(state) -> ptr[ast.Stmt]`
+- `parse_for_loop(state) -> ptr[ast.Stmt]`
+- `parse_block(state) -> span[ast.Stmt]`
+- `parse_literal(state) -> ptr[ast.Expr]`
+- `parse_string_literal(state) -> ptr[ast.Expr]`
+
+### 3.3 Ownership
+
+- `ParserState.tokens` — `span[token.Token]` borrowed from the lexer's Vec (the lexer's Vec must outlive the parser)
+- Returned `ast.SourceFile` — all `ptr[Expr]`/`ptr[Stmt]` point into arena
+- `ParseDiagnostic` strings — arena-allocated (or static for kind names)
+- AST nodes have zero runtime ownership — they are pure data in the arena
+
+---
+
+## 4. Module Loader (`mtc.loader`)
+
+### 4.1 `mtc.loader.path_resolver` (Path resolution)
+
+Imports: `std.fs`, `std.string`, `std.path`.
+
+```mt
+public enum Platform: ubyte
+    linux = 0
+    windows = 1
+    wasm = 2
+
+# Returns "" for the default platform, ".linux"/".windows"/".wasm" otherwise
+public function platform_suffix(platform: Platform) -> str:
+
+# Given a platform-less path, checks if a platform-specific variant exists;
+# returns the detected platform (or Option.none for the generic file)
+public function platform_suffix_for_path(path: str) -> Option[Platform]:
+
+# Resolves `a.b.c` → `a/b/c.mt` (preferring `.linux.mt` etc.)
+public function resolve_source_path(path: str, platform: Platform) -> string.String:
+
+# Resolves import path `a.b.c` → filesystem path
+public function resolve_module_path(
+    module_path: str, roots: span[str], platform: Platform
+) -> Option[string.String]:
+
+# Infers module name from filesystem path relative to roots
+public function infer_module_name(path: str, roots: span[str]) -> string.String:
+```
+
+**Ownership:** All returned `string.String` values are **owned** — caller must `release()`.
+
+### 4.2 `mtc.loader.module_loader` (Loader orchestration)
+
+Imports: `mtc.parser.ast`, `mtc.parser.parser`, `mtc.parser.state`,
+`mtc.lexer.lexer` (for C backend — transient, used only in check), `mtc.semantic.analyzer`,
+`mtc.semantic.types`, `mtc.loader.binder`, `mtc.loader.path_resolver`,
+`std.map`, `std.vec`, `std.string`, `std.fs`.
+
+```mt
+public struct LoadedModule:
+    module_name: string.String
+    path: string.String
+    source: string.String
+    source_file: ast.SourceFile
+    parse_diagnostics: vec.Vec[state.ParseDiagnostic]
+
+public struct LoadDiagnostic:
+    path: string.String
+    line: ptr_uint
+    column: ptr_uint
+    message: string.String
+    severity: str
+    code: str
+
+public struct Program:
+    modules: vec.Vec[LoadedModule]
+    order: span[ptr_uint]            # dependency-first ordering (indices → modules)
+    analyses: vec.Vec[analyzer.Analysis]  # one per module, in dependency order
+    diagnostics: vec.Vec[LoadDiagnostic]
+
+# Full pipeline: parse all imports → topo sort → sema each in order → return
+public function check_program(
+    root_path: str, roots: span[str], platform: path_resolver.Platform
+) -> Program:
+
+# Single-file check (no imports)
+public function check_single(
+    root_path: str, roots: span[str], platform: path_resolver.Platform
+) -> Program:
+```
+
+**Ownership:**
+- `Program` is **owned** by the caller — caller must `program.release()`
+- `Program.modules[]` source buffers own the `string.String` — AST slices borrow from these
+- `Program.analyses[]` are arena-allocated (embedded in the Program's arena)
+- `Program.diagnostics[]` strings are heap-allocated — freed by `Program.release()`
+- The `LoadedModule.source_file` AST nodes are arena-allocated
+- All `Vec` and `Map` fields must be `release()`d in `Program.release()`
+
+### 4.3 `mtc.loader.binder` (Module binding)
+
+Imports: `mtc.semantic.analyzer`.
+
+```mt
+# Constructs a ModuleBinding from an Analysis, filtering public/private
+public function bind_module(analysis: analyzer.Analysis) -> analyzer.ModuleBinding:
+```
+
+**Ownership:** Returns a `ModuleBinding` with `Map` fields — owned by the caller
+(embedded in the loader's `Program`).
+
+---
+
+## 5. Semantic Analyzer (`mtc.semantic`)
+
+### 5.1 `mtc.semantic.types` (Type model)
+
+Imports: `std.string`, `std.str`, `std.fmt`, `std.mem.heap` (for alloc_type).
+
+```mt
+public variant Type:
+    ty_primitive(name: str)
+    ty_str
+    ty_error
+    ty_type_meta
+    ty_nullable(base: ptr[Type])
+    ty_named(name: str, module_name: str)
+    ty_imported(module_name: str, name: str, args: span[Type])
+    ty_var(name: str)
+    ty_dyn(iface: str)
+    ty_opaque(module_name: str, name: str)
+    ty_generic(name: str, args: span[Type])
+    ty_function(params: span[Type], return_type: ptr[Type], variadic: bool, is_proc: bool)
+    ty_literal_int(value: long)
+    ty_tuple(elements: span[Type], field_names: Option[span[str]])
+
+# Allocates a ptr[Type] on the heap for recursive type embedding
+public function alloc_type(value: Type) -> ptr[Type]:
+
+# Convenience constructors
+public function primitive(name: str) -> Type:
+public function literal_int(value: long) -> Type:
+
+# Predicates (~30 functions — see types.rb for the full list)
+public function is_numeric_name(name: str) -> bool:
+public function is_integer_name(name: str) -> bool:
+public function is_bool(t: Type) -> bool:
+public function is_void(t: Type) -> bool:
+public function is_numeric(t: Type) -> bool:
+public function is_error(t: Type) -> bool:
+public function contains_error(t: Type) -> bool:
+public function is_raw_pointer(t: Type) -> bool:
+public function is_own_type(t: Type) -> bool:
+public function is_ref_type(t: Type) -> bool:
+public function is_nullable_type(t: Type) -> bool:
+public function is_integer_type(t: Type) -> bool:
+public function is_float_type(t: Type) -> bool:
+public function is_char_type(t: Type) -> bool:
+public function is_nominal_type(t: Type) -> bool:
+public function is_generic_type(t: Type) -> bool:
+public function lossless_integer_assignable(target_name: str, source_name: str) -> bool:
+public function definitely_different(a: Type, b: Type) -> bool:
+public function unwrap_nullable(t: Type) -> Type:
+public function nominal_key(t: Type) -> str:
+
+# Rendering (for diagnostics)
+public function type_to_string(t: Type) -> str:
+
+# Equality (structural, handles recursive types with by-identity caching)
+public function type_equals(a: Type, b: Type) -> bool:
+```
+
+**Ownership:** Types are arena-allocated.  `ptr[Type]` for recursive references
+(e.g., nullable base, function params/return) is allocated via `alloc_type` on
+the arena.  `span[Type]` for type arguments borrows from a Vec.
+
+### 5.2 `mtc.semantic.scope` (Lexical scope)
+
+Imports: `std.map`.
+
+```mt
+# Scope stack — a Map per nesting level, pushed/popped on block entry/exit
+public struct Scope:
+    stack: vec.Vec[map_mod.Map[str, bool]]
+
+public function push_scope(scope: ref[Scope]) -> void:
+public function pop_scope(scope: ref[Scope]) -> void:
+public function define(scope: ref[Scope], name: str) -> void:
+public function resolve(scope: ref[Scope], name: str) -> bool:
+public function shadow_check(scope: ref[Scope], name: str) -> bool:
+```
+
+### 5.3 `mtc.semantic.analyzer` (Type checker)
+
+Imports: `mtc.parser.ast`, `mtc.semantic.types`, `mtc.semantic.scope`,
+`mtc.semantic.diagnostics`, `mtc.semantic.type_compatibility`,
+`mtc.semantic.expressions`, `mtc.semantic.emit_expansion`,
+`std.map`, `std.vec`, `std.string`, `std.fmt`.
+
+```mt
+public struct SemanticDiagnostic:
+    line: ptr_uint
+    column: ptr_uint
+    message: str
+
+public struct ParamEntry:
+    name: str
+    ty: types.Type
+
+public struct FnSig:
+    name: str
+    params: span[ParamEntry]
+    return_type: types.Type
+    method_kind: ast.MethodKind
+    is_async: bool
+    is_variadic: bool
+    is_extern: bool
+
+public struct FieldEntry:
+    name: str
+    ty: types.Type
+
+public struct ModuleBinding:
+    functions: map_mod.Map[str, FnSig]
+    structs: map_mod.Map[str, span[FieldEntry]]
+    value_types: map_mod.Map[str, types.Type]
+    type_aliases: map_mod.Map[str, bool]
+    type_alias_types: map_mod.Map[str, types.Type]
+    static_member_types: map_mod.Map[str, bool]
+    member_keys: map_mod.Map[str, bool]
+    method_sigs: map_mod.Map[str, FnSig]
+    interfaces: map_mod.Map[str, span[ast.InterfaceMethod]]
+    implemented: map_mod.Map[str, span[ast.QualifiedName]]
+    match_case_names: map_mod.Map[str, span[str]]
+    types: map_mod.Map[str, bool]
+    # ... and private_* counterparts for each
+
+public struct Analysis:
+    module_name: str
+    module_kind: ast.ModuleKind
+    functions: map_mod.Map[str, FnSig]
+    structs: map_mod.Map[str, span[FieldEntry]]
+    value_types: map_mod.Map[str, types.Type]
+    type_aliases: map_mod.Map[str, types.Type]
+    enums: map_mod.Map[str, types.Type]
+    variants: map_mod.Map[str, span[str]]  # arm names
+    resolved_expr_types: map_mod.Map[ptr_uint, types.Type]   # expr ptr → type
+    resolved_call_kinds: map_mod.Map[ptr_uint, FnSig]        # call ptr → resolved function
+    diagnostics: vec.Vec[SemanticDiagnostic]
+    source_path: str
+
+# Single-file analysis (no imports)
+public function check_source_file(file: ast.SourceFile) -> Analysis:
+
+# Multi-file analysis (with import bindings)
+public function check_module(
+    file: ast.SourceFile,
+    imported_modules: ptr[map_mod.Map[str, ModuleBinding]]?,
+    module_name: str
+) -> Analysis:
+```
+
+**Ownership:**
+- `Analysis` is arena-allocated; returned to the loader which stores it in a
+  `vec.Vec[Analysis]`
+- All `Map` fields in `Analysis` are heap-allocated — freed when the vec is
+  released
+- `FnSig.params` — `span[ParamEntry]` borrows from an arena Vec
+- `resolved_expr_types` keys are `ptr_uint` (pointer addresses) — used as
+  identity keys for expression nodes allocated in the arena
+
+### 5.4 `mtc.semantic.compile_time` (Const evaluator)
+
+Imports: `mtc.parser.ast`, `mtc.semantic.types`, `std.vec`, `std.map`.
+
+```mt
+# Evaluate a const expression to a compile-time value
+public function eval_const_expression(
+    expr: ptr[ast.Expr],
+    env: ref[ConstEnv]
+) -> ConstValue:
+
+public variant ConstValue:
+    integer(value: long)
+    float(value: double)
+    string(value: str)
+    boolean(value: bool)
+    type_value(ty: types.Type)
+    void_value
+
+public struct ConstEnv:
+    # Maps const/let name → ConstValue
+    bindings: map_mod.Map[str, ConstValue]
+    # Stack of called function names (for recursion detection)
+    call_stack: vec.Vec[str]
+```
+
+### 5.5 `mtc.semantic.intrinsics` (Built-in callable resolver)
+
+Imports: `mtc.semantic.types`.
+
+```mt
+# Recognized built-in names (fatal, ref_of, ptr_of, read, size_of, etc.)
+# Returns Option.none if not a builtin; else returns the result type.
+# `env` is the analyzer's internal Context (defined in analyzer.mt).
+public function resolve_intrinsic(
+    name: str, arguments: span[ptr[ast.Expr]], env: ref[Context]
+) -> Option[types.Type]:
+```
+
+---
+
+## 6. Control Flow Analysis (`mtc.semantic.control_flow`)
+
+Imports: `mtc.parser.ast`, `mtc.semantic.types`.
+
+### 6.1 `mtc.semantic.control_flow.builder`
+
+```mt
+public struct CFG:
+    blocks: vec.Vec[BasicBlock]
+    entry: ptr_uint
+    exit: ptr_uint
+
+public struct BasicBlock:
+    statements: span[ptr[ast.Stmt]]
+    successors: span[ptr_uint]
+
+public function build_cfg(body: span[ast.Stmt]) -> CFG:
+```
+
+### 6.2 `mtc.semantic.control_flow.definite_assignment`
+
+```mt
+public struct AssignmentState:
+    # Maps local name → assigned?
+    locals: map_mod.Map[str, bool]
+
+public function check_definite_assignment(body: span[ast.Stmt]) -> vec.Vec[Diagnostic]:
+```
+
+### 6.3 `mtc.semantic.control_flow.reachability`
+
+```mt
+# Returns diagnostics for: missing return, unreachable code after return/break
+public function check_reachability(body: span[ast.Stmt], expected_return_type: types.Type) -> vec.Vec[Diagnostic]:
+```
+
+### 6.4 `mtc.semantic.control_flow.nullability`
+
+```mt
+# Tracks nullable → non-null narrowing through if-guards, else-guards, match arms
+public function analyze_nullability(
+    body: span[ast.Stmt], initial_state: ref[NullabilityState]
+) -> NullabilityResult:
+```
+
+---
+
+## 7. Lowering (`mtc.lowering`)
+
+### 7.1 `mtc.lowering.utils` (C name mangling + helpers)
+
+Imports: `mtc.semantic.types`, `mtc.parser.ast`, `mtc.ir`.
+
+```mt
+# C name mangling — produces `linkage_name` values for IR declarations.
+# Module-qualified names become underscore-joined C identifiers.
+public function module_function_c_name(module_name: str, name: str, type_arguments: span[types.Type]) -> str:
+public function function_binding_c_name(binding: analyzer.FnSig, module_name: str) -> str:
+public function external_function_c_name(binding: analyzer.FnSig) -> str:
+public function enum_member_c_name(type: types.Type, member_name: str) -> str:
+public function value_c_name(name: str) -> str:
+public function imported_value_c_name(imported_module: str, name: str) -> str:
+public function module_value_c_name(module_name: str, name: str) -> str:
+public function struct_field_c_name(struct_name: str, field_name: str) -> str:
+
+# Type classification helpers (~25 functions — see lowering/utils.rb)
+public function range_iterable?(expression) -> bool:
+public function array_type?(type) -> bool:
+public function array_element_type(type) -> types.Type:
+public function array_length(type) -> ptr_uint:
+public function char_array_text_type?(type) -> bool:
+public function str_buffer_type?(type) -> bool:
+public function str_buffer_capacity(type) -> ptr_uint:
+public function collection_loop_type(type) -> types.Type:
+```
+
+### 7.2 `mtc.lowering.lowering` (AST → IR)
+
+Imports: `mtc.parser.ast`, `mtc.ir`, `mtc.semantic.types`,
+`mtc.semantic.analyzer`, `mtc.loader.module_loader`,
+`mtc.lowering.utils`, `std.vec`, `std.map`, `std.string`, `std.mem.heap`.
+
+```mt
+public struct LoweringError:
+    message: str
+    line: ptr_uint
+    column: ptr_uint
+    path: str
+
+# The single public entry point: Analysis → IR.Program
+public function lower(program: loader.Program) -> ir.Program:
+```
+
+Key internal functions (these are in the single `lowering.mt` file, not separate modules):
+- `lower_module(loaded: loader.LoadedModule, analysis: analyzer.Analysis) -> ir.Program`
+- `lower_constant_decl(top_level_decl, env) -> void`
+- `lower_function_def(top_level_decl, env) -> void`
+- `lower_struct_def(top_level_decl, env) -> void`
+- `lower_enum_def(top_level_decl, env) -> void`
+- `lower_expression(expr: ptr[ast.Expr], env) -> ptr[ir.Expr]`
+- `lower_statement(stmt: ptr[ast.Stmt], env) -> span[ir.Stmt]`
+- `lower_call(callee_expr, arguments, env) -> ptr[ir.Expr]`
+- `lower_binary_expression(left, op, right, env) -> ptr[ir.Expr]`
+- `lower_member_access(receiver, member, env) -> ptr[ir.Expr]`
+- `lower_match_statement(scrutinee, arms, env) -> span[ir.Stmt]`
+- `lower_if_statement(condition, then_body, else_body, env) -> span[ir.Stmt]`
+- `lower_while_loop(condition, body, env) -> span[ir.Stmt]`
+- `lower_for_loop(iterable, bindings, body, env) -> span[ir.Stmt]`
+
+### 7.3 `mtc.lowering.async` (Async CPS transformation)
+
+Imports: `mtc.ir`, `mtc.semantic.types`, `mtc.lowering.utils`.
+
+```mt
+# Transforms an async function's IR body into CPS continuation-passing style
+# with a heap-allocated frame struct
+public function lower_async_function(
+    func: ir.Function, module_name: str
+) -> ir.Function:
+
+# Synthesizes a root main() entrypoint for async-main programs
+public function build_root_main_entrypoint(program: ir.Program) -> ir.Program:
+```
+
+**Ownership:**
+- `ir.Program` returned by `lower()` is arena-allocated
+- `ir.Program.functions[]` — `span[ir.Function]` borrowed from arena Vec
+- Each `ir.Function.body` — `span[ir.Stmt]` borrowed from arena
+- `ptr[ir.Expr]` / `ptr[ir.Stmt]` — heap-allocated in the arena
+- `LoweringError` is raised (fatal) — not returned as Result
+
+---
+
+## 8. C Backend (`mtc.c_backend`)
+
+Imports: `mtc.ir`, `mtc.semantic.types`, `mtc.lowering.utils`,
+`std.string`, `std.str`, `std.fmt`, `std.map`, `std.vec`, `std.mem.heap`.
+
+```mt
+public struct CBackendError:
+    message: str
+    line: ptr_uint
+    column: ptr_uint
+    path: str
+
+# The single public entry point: IR → C source text.
+# Returns an owned string.String containing the complete .c file.
+# Merge (`include`) lines, forward declarations, constants, globals,
+# struct/enum/union/variant/opaque type definitions, type aliases,
+# and all function definitions.
+public function generate_c(program: ir.Program) -> string.String:
+
+# C keyword table and sanitizer — call from every emission site
+# (18 call sites across type declarations, expressions, runtime helpers)
+# Static string comparison table; no Map allocation needed at runtime.
+static C_KEYWORDS: array[str, 44] = array[str, 44](
+    "auto", "break", "case", "char", "const", "continue", "default", "do",
+    "double", "else", "enum", "extern", "float", "for", "goto", "if",
+    "inline", "int", "long", "register", "return", "short", "signed",
+    "sizeof", "static", "struct", "switch", "typedef", "union", "unsigned",
+    "void", "volatile", "while", "_Bool", "_Complex", "_Imaginary",
+    "alignas", "alignof", "bool", "complex", "imaginary",
+    "noreturn", "static_assert", "thread_local",
+)
+public function sanitize_c_identifier(name: str) -> str:
+```
+
+Key internal emission functions:
+- `emit_preamble(program, emitter) -> void`
+- `emit_forward_declarations(program, emitter) -> void`
+- `emit_constant(decl, emitter) -> void`
+- `emit_global(decl, emitter) -> void`
+- `emit_struct_declaration(decl, emitter) -> void`
+- `emit_enum_declaration(decl, emitter) -> void`
+- `emit_variant_declaration(decl, emitter) -> void`
+- `emit_function_definition(func, emitter) -> void`
+- `emit_expression(expr: ptr[ir.Expr]) -> str`  # appends to emitter buffer
+- `emit_statement(stmt: ptr[ir.Stmt]) -> void`
+- `emit_switch_statement(expr, cases, emitter) -> void`
+- `c_type(type: types.Type) -> str`
+- `c_declaration(type: types.Type, name: str) -> str`
+- `c_field_declaration(type: types.Type, name: str) -> str`
+- `render_expression(expr) -> str`  # returns str slice into emitter buffer
+
+---
+
+## 9. Build System (`mtc.build`, `mtc.build_cache`)
+
+### 9.1 `mtc.build` (Build driver)
+
+Imports: `mtc.ir`, `mtc.c_backend`, `mtc.loader.module_loader`,
+`std.fs`, `std.path`, `std.process`, `std.string`, `std.vec`.
+
+```mt
+# Lower + generate C + compile C → binary
+# Returns the output path on success, error message on failure
+public function build(
+    program: loader.Program,
+    ir_program: ir.Program,
+    output_path: str,
+    c_compiler: str,
+    roots: span[str],
+    sanitize: bool
+) -> Result[string.String, string.String]:
+```
+
+Internal flow:
+1. Call `c_backend.generate_c(ir_program)` → C source `string.String`
+2. Write C source to temp file via `fs.write_text`
+3. Build CC argument list as `Vec[str]`
+4. Append vendor lib link flags, binding flags, sanitizer flags
+5. `process.capture(command_vec.as_span())` → check exit code
+6. Return output path or error
+
+### 9.2 `mtc.build_cache` (Incremental cache)
+
+Imports: `mtc.loader.module_loader`, `std.string`.
+
+```mt
+public function compute_key(
+    program: ref[loader.Program], c_compiler: str
+) -> string.String:
+
+public function lookup(key: str) -> Option[string.String]:
+public function store(key: str, binary_path: str) -> void:
+public function materialize(cached_path: str, output_path: str) -> bool:
+```
+
+---
+
+## 10. CLI (`mtc.main`)
+
+Imports: every module above, plus `std.stdio`, `std.terminal`, `std.process`.
+
+```mt
+# Entry point — the C main() function
+function main(args: span[str]) -> int:
+
+# Command dispatch:
+#   if cmd == "lex":      lex_command(args)
+#   if cmd == "parse":    parse_command(args)
+#   if cmd == "check":    check_command(args)
+#   if cmd == "lower":    lower_command(args)
+#   if cmd == "emit-c":   emit_c_command(args)
+#   if cmd == "build":    build_command(args)
+#   if cmd == "run":      run_command(args)
+#   if cmd == "test":     test_command(args)
+#   if cmd == "format":   format_command(args)
+#   if cmd == "debug":    debug_command(args)
+
+# Each command handler parses its own argv portion manually
+# (while loop with mutable ai: ptr_uint counter)
+# Every Vec/String allocation uses defer: x.release()
+```
+
+---
+
+## 11. Porting Order & Verification Checkpoints
+
+| Order | Module | Depends on | Verification |
+|-------|--------|-----------|-------------|
+| 1 | `mtc.lexer.token_kinds` | nothing | Compiles |
+| 2 | `mtc.lexer.token` | token_kinds | Compiles |
+| 3 | `mtc.lexer.keywords` | token_kinds | Unit test |
+| 4 | `mtc.ast` | nothing | Compiles |
+| 5 | `mtc.semantic.types` | nothing (except std) | Unit test for predicates |
+| 6 | `mtc.ir` | semantic.types, ast | Compiles |
+| 7 | `mtc.sexpr_dumper` | token, ast, types, ir | Diff vs Ruby S-expr |
+| 8 | `mtc.lexer.lexer` | token, token_kinds, keywords | `diff <(ruby mtc lex --sexpr f) <(self lex --sexpr f)` |
+| 9 | `mtc.parser.state` | token, token_kinds | Compiles |
+| 10 | `mtc.parser.parser` | ast, state, lexer | `diff <(ruby mtc parse --sexpr f) <(self parse --sexpr f)` |
+| 11 | `mtc.loader.path_resolver` | (stdlib only) | Unit test |
+| 12 | `mtc.semantic.scope` | (stdlib only) | Unit test |
+| 13 | `mtc.semantic.intrinsics` | types | Compiles |
+| 14 | `mtc.semantic.compile_time` | ast, types | Unit test |
+| 15 | `mtc.semantic.type_compatibility` | types | Unit test |
+| 16 | `mtc.semantic.expressions` | types | Unit test |
+| 17 | `mtc.semantic.diagnostics` | (stdlib only) | Compiles |
+| 18 | `mtc.semantic.analyzer` | all semantic.* | `diff <(ruby mtc check f) <(self check f)` |
+| 19 | `mtc.semantic.control_flow.*` | analyzer, ast | Unit test |
+| 20 | `mtc.loader.binder` | analyzer | Unit test |
+| 21 | `mtc.loader.module_loader` | all above | Multi-file test |
+| 22 | `mtc.lowering.utils` | types, ast, ir | Unit test |
+| 23 | `mtc.lowering.lowering` | all above | `diff <(ruby mtc lower f) <(self lower f)` |
+| 24 | `mtc.lowering.async` | ir, lowering | Async example test |
+| 25 | `mtc.c_backend` | ir, types, lowering.utils | `diff <(ruby mtc emit-c f) <(self emit-c f)` |
+| 26 | `mtc.pretty_printer` | ast, ir | AST round-trip test |
+| 27 | `mtc.build` | c_backend, loader | Self-build |
+| 28 | `mtc.build_cache` | build | Unit test |
+| 29 | `mtc.main` | all above | `diff stage1.c stage2.c` = empty |