@jhlagado/azm 0.2.8 → 0.2.10

package/docs/codebase/02-source-loading-and-parsing.md

@@ -0,0 +1,263 @@
+---
+layout: default
+title: 'Chapter 2 - Source Loading and Parsing'
+parent: 'AZM Engineering Manual'
+nav_order: 2
+---
+
+[<- Orientation and Repository Layout](01-orientation-and-repository-layout.md) | [Assembly and Z80 Emission ->](03-assembly-and-z80-emission.md)
+
+# Chapter 2 - Source Loading and Parsing
+
+Source loading and parsing turn entry files into typed source items. This
+chapter follows the path from a filename to the structured data that assembly,
+tooling and register contracts consume.
+
+The loading boundary lives in `src/node/source-host.ts`. The parser is
+orchestrated by `parseNextSourceItems()` in `src/core/compile.ts`, with
+single-line parsing in `src/syntax/parse-line.ts`. Expression and declaration
+parsing is split across tokenizer, token-expression, directive and layout
+modules in `src/syntax/`.
+
+## Entry Files and Source Text
+
+The public tooling and compile APIs enter loading through `loadProgramNext()` in
+`src/tooling/api.ts`. That function calls `expandSourceForTooling()` and then
+passes the expanded logical lines to `parseNextSourceItems()`.
+
+`expandSourceForTooling()` accepts:
+
+```ts
+export interface LoadProgramNextOptions {
+  readonly entryFile: string;
+  readonly includeDirs?: readonly string[];
+  readonly directiveAliasFiles?: readonly string[];
+  readonly preloadedText?: string;
+  readonly signal?: AbortSignal;
+}
+```
+
+The entry file is normalised and checked for a source extension. AZM source
+entries use `.asm` or `.z80`. `preloadedText` lets editor integrations parse an
+unsaved buffer for the entry file while included files still come from disk.
+`signal` lets an editor cancel stale work when a newer buffer arrives.
+
+The loader keeps the full text of every loaded source file in `sourceTexts`.
+Later stages use parsed source items for compiler logic, but several features
+need original text:
+
+- register contract annotation rewrites exact source lines
+- tooling reads source text for diagnostics and code actions
+- D8 map generation needs file names and line provenance
+- case-style linting inspects original token case
+
+Logical lines drive parsing. Source texts support tools that need to point back
+into the user's files.
+
+## Source Loading Directives
+
+The tooling loader recognises two source-loading directives before parsing:
+`.include` and `.import`.
+
+`.include` is textual inclusion. The loader reads the entry file, scans it into
+logical lines and recursively expands include directives. Include paths resolve
+relative to the including source file first, then through configured include
+directories.
+
+`.import` uses the same path resolution rule, but it starts a new source
+ownership unit for tooling. Parsed items from the imported file still join the
+same flattened logical line stream, though their spans record the imported file
+as the owning unit. This lets tools distinguish entry-owned source, text pulled
+in by `.include` and routines introduced by imported modules.
+
+Repeated imports of the same resolved file are idempotent. The first import
+loads and emits the module at the import point; later imports of that same
+resolved file are skipped. Repeated includes remain textual and repeatable.
+Recursive include or import stacks are diagnosed before parsing, with the
+diagnostic naming the recursive source relation.
+
+Labels in imported source keep their physical source locations. Imported
+`@Name:` labels are public exports visible to outside source as `Name`. Plain
+labels in an imported file are private to that import unit. Text included from
+inside an imported file remains part of that imported unit, so its plain labels
+are private unless they are also public `@` labels.
+
+That rule keeps library files portable. A library can include a sibling file and
+still assemble when the entry file is run from another directory. Include
+directories then act as project-level search paths for shared headers, vendor
+source and imported modules.
+
+The loader returns:
+
+```ts
+export interface ExpandedNextSource {
+  readonly entryFile: string;
+  readonly lines: readonly LogicalLine[];
+  readonly sourceTexts: ReadonlyMap<string, string>;
+  readonly sourceLineComments: ReadonlyMap<string, ReadonlyMap<number, string>>;
+}
+```
+
+`lines` is the flattened source stream for parsing. `sourceTexts` keeps the
+original file text. `sourceLineComments` keeps comments indexed by file and line
+so register contract analysis can reconstruct AZMDoc contract blocks after routines have
+been identified.
+
+## Logical Lines and Comments
+
+`src/source/logical-lines.ts` scans a `SourceFile` into `LogicalLine` objects. A
+logical line records the source name, line number and original text. Tooling
+loads can also attach `sourceUnit` and `sourceRelation`:
+
+- `sourceUnit` is the owning file for the current tooling unit
+- `sourceRelation` is `entry`, `include` or `import`
+
+This thin structure gives every later diagnostic a stable location and enough
+provenance for tooling features that need to reason about module ownership.
+
+The source helpers are small and important:
+
+| File                      | Role                                                   |
+| ------------------------- | ------------------------------------------------------ |
+| `source-file.ts`          | Wraps source text with a source name.                  |
+| `logical-lines.ts`        | Splits text into line records.                         |
+| `source-span.ts`          | Defines the common span shape.                         |
+| `line-comment-scanner.ts` | Finds line comments while respecting quoted text.      |
+| `strip-line-comment.ts`   | Removes semicolon comments through the shared scanner. |
+
+`strip-line-comment.ts` is used by source-loading directive recognition, layout parsing,
+conditional assembly and single-line parsing. Shared comment handling prevents
+each stage from inventing a slightly different rule for semicolons inside
+strings and character literals.
+
+## Directive Aliases
+
+Directive aliases are loaded during `loadProgramNext()`:
+
+```ts
+const directiveAliasProfiles = await Promise.all(
+  (options.directiveAliasFiles ?? []).map((path) => readDirectiveAliasProfile(path)),
+);
+const directiveAliasPolicy = buildDirectiveAliasPolicy(directiveAliasProfiles);
+```
+
+`src/syntax/directive-aliases.ts` owns the alias policy. Built-in aliases and
+project alias files are normalised before line parsing. The parser then
+receives canonical directive forms and emits canonical source items.
+
+Aliases are a syntax boundary. They affect directive recognition before parsing.
+The assembler-time model receives canonical source items.
+
+## Source Items
+
+The parser is the first place where AZM source becomes compiler data. Before
+this point, a line is text with a file name and line number. After this point, a
+line is a label, instruction, directive, layout declaration or comment item.
+
+`src/model/source-item.ts` defines the parser output. The model includes:
+
+- labels
+- `.org`, `.equ`, `.db`, `.dw`, `.ds`, `.align`, string directives and `.end`
+- instructions
+- record and union layout declarations
+- type aliases
+- enums
+- op-expanded items
+- comments
+
+Each item carries a source span where appropriate. Tooling spans now preserve
+optional `sourceUnit` and `sourceRelation` fields when the loader attached them.
+Assembly uses item kind to decide size and emission. Register contract analysis
+uses instruction, label and comment items to build routines. D8 map output uses
+spans to connect emitted bytes back to files and lines.
+
+## Top-Level Parse Order
+
+`parseNextSourceItems()` handles structural forms before ordinary line parsing:
+
+1. `applyConditionalAssembly()` in `src/core/conditional-assembly.ts` filters
+   the logical line stream.
+2. `collectOps()` records top-level `op` definitions and marks their body lines.
+3. Name-left `.typealias` declarations are parsed.
+4. Record and union headers collect `.field` declarations until `.endtype` or
+   `.endunion`.
+5. Visible op invocations expand into ordinary source items.
+6. `parseLogicalLine()` handles single-line labels, directives, data and
+   instructions.
+
+This order matters. Ops must be collected before invocation expansion. Layout
+declarations must collect their body lines as one source item. Ordinary
+instruction parsing should see the lines that remain after those structural
+forms have been handled.
+
+## Layout and Declaration Parsing
+
+Name-left layout syntax is parsed in `parseNextSourceItems()` because a record
+or union body spans multiple lines:
+
+```asm
+Sprite .type
+x      .field byte
+y      .field byte
+tile   .field byte
+flags  .field byte
+       .endtype
+```
+
+Fields are parsed as `LayoutField` values. Each field has a name and a type
+expression. The parser checks declaration shape. `address-planning.ts` later
+checks duplicate field names, layout size and type references.
+
+Type aliases are parsed as named bindings:
+
+```asm
+SpriteArray .typealias Sprite[16]
+```
+
+The parser stores the alias target as a type expression. Assembly resolves the
+target against scalar layout names, record names, union names and other type
+aliases.
+
+The parser also distinguishes address labels from declarations. An address
+label uses a colon and becomes a label item. Name-left declarations become
+equate, enum, type, union or type-alias items.
+
+```asm
+Start:
+        ret
+
+COUNT .equ 8
+```
+
+A label contributes an address based on placement. An equate contributes an
+assembler-time value based on expression evaluation.
+
+## Expressions and Conditionals
+
+`src/syntax/expression-tokenizer.ts` tokenizes expression text.
+`parse-token-expression.ts` builds expression trees from tokens.
+`parse-expression.ts` is the public syntax wrapper used by line parsing.
+`parse-layout-expression.ts` parses layout type expressions used by `.ds`,
+`.field`, `.typealias`, `sizeof(...)`, `offset(...)` and layout casts.
+`parse-directive-statement.ts` parses directive statements that need more than
+single-token recognition.
+
+The parser produces expression trees from `src/model/expression.ts`.
+`src/semantics/expression-evaluation.ts` evaluates those trees when the
+assembler-time environment is available.
+
+Conditional assembly is handled before final line parsing. The conditional pass
+keeps the active lines and removes inactive branches from the stream seen by
+later stages. Ordinary parsing then receives one effective source program.
+
+## Parse Diagnostics
+
+`src/syntax/parse-diagnostics.ts` contains shared helpers for syntax errors.
+Diagnostic IDs come from `src/model/diagnostic.ts`. Use those helpers when
+adding parse failures so source positions, severity and code shape stay
+consistent.
+
+Parser recovery matters for editor tooling. A user may have a half-written line
+while typing. Tooling still needs symbols, diagnostics and register contract hints
+for surrounding source, so parse errors should usually report a diagnostic and
+let parsing continue.

package/docs/codebase/03-assembly-and-z80-emission.md

@@ -0,0 +1,251 @@
+---
+layout: default
+title: 'Chapter 3 - Assembly and Z80 Emission'
+parent: 'AZM Engineering Manual'
+nav_order: 3
+---
+
+[<- Source Loading and Parsing](02-source-loading-and-parsing.md) | [Ops and Register Contracts ->](04-ops-and-register-contracts.md)
+
+# Chapter 3 - Assembly and Z80 Emission
+
+Assembly turns source items into facts, bytes, fixups and source segments. This
+chapter combines the assembler-time fact model with the Z80 emission path
+because those two stages are tightly coupled: planning decides addresses and
+sizes, then emission writes bytes using those facts.
+
+The central files are:
+
+- `src/assembly/address-planning.ts`
+- `src/assembly/address-symbols.ts`
+- `src/assembly/placement.ts`
+- `src/assembly/program-emission.ts`
+- `src/assembly/fixup-emission.ts`
+- `src/assembly/assemble-program.ts`
+- `src/semantics/expression-evaluation.ts`
+- `src/semantics/constant-operators.ts`
+- `src/semantics/layout-evaluation.ts`
+- `src/z80/parse-instruction.ts`
+- `src/z80/instruction.ts`
+- `src/z80/encode.ts`
+- `src/z80/effects.ts`
+
+## Assembly Orchestration
+
+`src/assembly/assemble-program.ts` is the entry point:
+
+```ts
+export function assembleProgram(items: readonly SourceItem[]): AssembleProgramResult;
+```
+
+It builds the address state, emits the program image and returns diagnostics,
+symbols, byte maps and source segments. The function is intentionally small.
+Detailed decisions live in the modules below it.
+
+## Address Planning
+
+`src/assembly/address-planning.ts` walks source items and builds the facts
+needed to place and evaluate the program:
+
+- labels and their addresses
+- `.equ` constants and enum members
+- record and union layouts
+- type aliases
+- `.org` placement state
+- sizes for data, storage and instructions
+
+The returned `AddressState` contains symbol facts, equate records, layout
+records, enum names, diagnostics and placement information. This is where parsed
+syntax becomes assembler knowledge. A label item becomes an address. A type
+declaration becomes a layout record. A `.ds` directive becomes a byte count.
+
+## Placement
+
+`src/assembly/placement.ts` owns placement state. `.org` sets the active
+address. Instructions advance by encoded size. `.db` advances by emitted byte
+count. `.dw` advances by two bytes per expression. `.ds` advances by calculated
+storage size. `.align` advances to the next aligned address.
+
+Placement is separate from byte emission so the assembler can resolve symbols
+before every byte has been written. A branch to a later label can be encoded
+after address planning has seen the label definition.
+
+## Symbols and Layouts
+
+Labels define addresses. `.equ` declarations define assembler-time values.
+Enums define qualified constants. `src/assembly/address-symbols.ts` contains
+the symbol helpers that define labels, equates and enum members, enforce
+duplicate rules and record spans for diagnostics and output metadata.
+
+Record and union declarations become layout records. A record field advances the
+offset by its byte size. A union field starts at offset zero and the union size
+is the largest field size.
+
+For a record:
+
+```asm
+Sprite .type
+x      .field byte
+y      .field byte
+tile   .field byte
+flags  .field byte
+       .endtype
+```
+
+the layout record stores `x = 0`, `y = 1`, `tile = 2`, `flags = 3` and total
+size `4`.
+
+Type aliases bind a name to another type expression:
+
+```asm
+SpriteArray .typealias Sprite[16]
+```
+
+`sizeof(SpriteArray)` resolves through the alias to the size of `Sprite[16]`,
+with the same field paths as the underlying array expression.
+
+## Expression Evaluation
+
+`src/semantics/expression-evaluation.ts` evaluates expression trees against the
+assembler-time environment. It coordinates literal arithmetic, symbol lookup,
+constant operators, byte functions, layout functions and layout casts that fold
+to constant addresses.
+
+The operator and layout rules live in focused modules:
+
+| File                    | Role                                                                |
+| ----------------------- | ------------------------------------------------------------------- |
+| `constant-operators.ts` | Binary and unary constant-operator dispatch.                        |
+| `binary-operators.ts`   | Binary arithmetic, bitwise, comparison and logical operators.       |
+| `unary-operators.ts`    | Unary numeric operators.                                            |
+| `byte-functions.ts`     | `LSB(...)`, `MSB(...)` and related byte extraction helpers.         |
+| `layout-evaluation.ts`  | `sizeof(...)`, `offset(...)` and layout type expression evaluation. |
+| `layout-path.ts`        | Field and array path resolution for layout casts.                   |
+| `layout-format.ts`      | Human-readable layout names for diagnostics.                        |
+| `diagnostics.ts`        | Shared semantic diagnostic construction.                            |
+
+Expression evaluation is context-sensitive. A symbol in an instruction operand
+may be a label or constant. A type expression in `.ds Sprite[4]` resolves to a
+byte count. A layout cast such as `<SpriteArray>Sprites[3].flags` resolves to an
+address when the base address, type alias, index and field path are all known at
+assembly time.
+
+## Data and Storage Size
+
+Address planning needs the byte length of each directive. It contains size
+helpers for `.db`, `.dw`, `.ds`, strings and alignment. The same string
+directive byte rules are reused during emission so planning and output stay
+aligned.
+
+Initialized data writes bytes through `.db`, `.dw`, `.cstr`, `.pstr` and
+`.istr`. `.ds` reserves space calculated from numbers or layout type
+expressions. Layout type expressions are byte-size expressions in storage and
+field-size positions.
+
+## Program Emission
+
+`src/assembly/program-emission.ts` owns byte writing. It walks source items in
+order and writes emitted bytes to a map keyed by absolute address. It also
+records source segments for the D8 map writer.
+
+The emitted program shape is:
+
+```ts
+export interface EmittedProgram {
+  readonly image: ReadonlyMap<number, number>;
+  readonly sourceSegments: readonly EmittedSourceSegment[];
+  readonly initializedAddresses: readonly number[];
+}
+```
+
+The byte map stores final addresses. If source emits one byte at `$0100` and
+another at `$8000`, the map has two entries. A sparse map fits Z80 programs that
+place code, data and ROM vectors at separate origins.
+
+## Emission Walkthrough
+
+For this source:
+
+```asm
+        .org $0100
+Start:
+        ld      a,42
+        jp      Start
+```
+
+address planning records `Start = $0100`. The instruction parser represents
+`ld a,42` as an immediate load and `jp Start` as an absolute branch. The encoder
+returns literal opcode bytes plus a 16-bit expression fragment for `Start`.
+Fixup emission evaluates `Start` to `$0100` and writes the little-endian operand
+bytes.
+
+The final byte map contains:
+
+```text
+$0100: 3E
+$0101: 2A
+$0102: C3
+$0103: 00
+$0104: 01
+```
+
+The D8 source segment records that those bytes came from the source lines that
+emitted them.
+
+## Z80 Instruction Model
+
+`src/z80/instruction.ts` defines the instruction and operand types.
+`src/z80/parse-instruction.ts` dispatches instruction text into parser families.
+`src/z80/encode.ts` dispatches typed instructions into encoder families.
+`src/z80/effects.ts` describes register and flag effects for register contracts.
+
+The parser and encoder work as a pair:
+
+| File                                                                                           | Question                                                                                     |
+| ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
+| `parse-instruction.ts`                                                                         | Which parser family should handle this instruction head?                                     |
+| `parse-basic.ts`, `parse-branch.ts`, `parse-exchange.ts`, `parse-io-control.ts`, `parse-ld.ts` | Parse specific instruction families.                                                         |
+| `parse-operands.ts`, `parse-conditions.ts`, `operand-split.ts`                                 | Split operands and classify conditions, registers, indexed operands and expression operands. |
+| `instruction.ts`                                                                               | How is that form represented as TypeScript data?                                             |
+| `encode.ts`, `encode-core.ts`, `encode-ld.ts`, `encode-ld-helpers.ts`                          | Which bytes and fixup fragments represent that form?                                         |
+| `effects.ts`                                                                                   | Which registers and flags does that form read or write?                                      |
+
+The instruction model keeps overloaded Z80 mnemonics manageable. `ld` has many
+forms, but the parser classifies operands before encoding. The encoder then
+selects a specific opcode family from typed operands.
+
+## Fixups and Relative Branches
+
+The Z80 encoder returns fragments: literal bytes, 8-bit immediates, 16-bit
+immediates and relative branch targets. `src/assembly/fixup-emission.ts`
+evaluates the expression attached to each fragment and writes the final byte or
+word.
+
+Relative branches such as `jr` and `djnz` emit an 8-bit displacement from the
+next instruction. The parser recognises `jr nz,Loop`. The encoder knows the
+opcode and that the operand is relative. Fixup emission knows the current
+address and final target address.
+
+The same principle applies to absolute branches and immediate operands. The
+encoder chooses the fragment width. Fixup emission evaluates the expression and
+checks that the value fits that width.
+
+## Source Segments
+
+Every emitted byte range can carry source provenance. `program-emission.ts`
+adds `EmittedSourceSegment` records with start address, end address, source
+file, line and segment kind. `writeD8m()` later groups these segments by file
+for Debug80.
+
+Source segments classify emitted ranges as code, data, directive output, label
+context or unknown output. Debug80 uses this metadata to connect source lines to
+addresses. A byte-perfect assembler can still give a poor debugging experience
+when source segments are too broad, absent or attached to a different file.
+
+## Changing Assembly or Encoding
+
+Instruction changes usually touch `parse-instruction.ts`, `instruction.ts`,
+`encode.ts` and `effects.ts`. Assembly changes usually touch
+`address-planning.ts`, `program-emission.ts`, `fixup-emission.ts` or
+`placement.ts`, depending on whether the change affects facts, emitted bytes,
+symbolic references or address movement.