parser-lr 0.4.0 → 0.5.0

package/docs/parser-lr-enhancement-proposals.md

@@ -0,0 +1,548 @@
+# parser-lr enhancement proposals (EduBASIC consumer)
+
+This document is for agents working on [parser-lr](https://github.com/adamrmoss/parser-lr). It lists concrete improvements suggested by the EduBASIC migration (`parser-lr` **0.4.1**, grammar at [`src/lang/parsing/grammars/edu-basic.grammar`](../src/lang/parsing/grammars/edu-basic.grammar)).
+
+**Goal:** let grammar authors write stable `ast` + `transform` rules for statements the same way expressions already work, so downstream bridges do not need `keyword-fallbacks.ts`-style workarounds.
+
+---
+
+## Reference consumer
+
+| Artifact | Path |
+|----------|------|
+| Grammar | [`src/lang/parsing/grammars/edu-basic.grammar`](../src/lang/parsing/grammars/edu-basic.grammar) |
+| Generated table | [`src/lang/parsing/edu-basic.table.json`](../src/lang/parsing/edu-basic.table.json) |
+| AST bridge | [`src/lang/parsing/ast-bridge/`](../src/lang/parsing/ast-bridge/) |
+| Keyword fallbacks (debt) | [`src/lang/parsing/ast-bridge/statements/keyword-fallbacks.ts`](../src/lang/parsing/ast-bridge/statements/keyword-fallbacks.ts) |
+| Dispatch table | [`src/lang/parsing/ast-bridge/statements/dispatch.ts`](../src/lang/parsing/ast-bridge/statements/dispatch.ts) |
+
+Regenerate table after grammar changes:
+
+```bash
+npm run parser:table
+```
+
+Parse smoke test:
+
+```bash
+node -e "
+import { ParseContext } from 'parser-lr';
+import table from './src/lang/parsing/edu-basic.table.json' with { type: 'json' };
+const ctx = ParseContext.fromTableJson(JSON.stringify(table));
+console.log(JSON.stringify(ctx.parseSource('CLS'), null, 2));
+"
+```
+
+---
+
+## Current asymmetry
+
+**Expressions** in EduBASIC have full `ast` + `transform` coverage (`comparison_expr.#binary`, `flatten(expr_list)`, etc.). Bridge code walks predictable node shapes.
+
+**Statements** use `transform` only at the `statement` dispatcher:
+
+```ebnf
+statement ->
+    #cls pass(stmt)
+  | #print pass(stmt)
+  | … ;
+```
+
+Individual `*_stmt` productions have **no** `ast` entries and **no** `transform` rules. When `pass(stmt)` runs and the matched production has a single terminal child, the output often **collapses** to that terminal instead of preserving the production symbol.
+
+That forces ~70 lines of `keyword-fallbacks.ts` plus duplicate handling in domain builders.
+
+---
+
+## Priority 1: `pass()` must preserve production symbol
+
+### Problem
+
+Grammar author binds `[stmt]:cls_stmt` and expects consumers to key off `cls_stmt`. Today bare `CLS` yields `kw_cls`.
+
+**Grammar fragment (reproducer):**
+
+```ebnf
+name "pass-collapse-repro" ;
+
+tokens
+    kw_cls = /CLS/i ;
+    kw_with = /WITH/i ;
+    ident = /[a-z]+/ ;
+
+start program_line ;
+
+grammar
+    program_line =
+        #statement [stmt]:cls_stmt
+      ;
+
+    cls_stmt =
+        kw_cls [ kw_with ident ]
+      ;
+
+transform
+    program_line ->
+        #statement pass(stmt) ;
+```
+
+**Input:** `CLS`
+
+**Actual output today:**
+
+```json
+{
+  "symbol": "kw_cls",
+  "children": [],
+  "text": "CLS",
+  "variant": null
+}
+```
+
+**Required output:**
+
+```json
+{
+  "symbol": "cls_stmt",
+  "children": [
+    { "symbol": "kw_cls", "text": "CLS", "children": [] }
+  ],
+  "variant": null
+}
+```
+
+**Input:** `CLS WITH foo`
+
+**Required output (variant optional but symbol must stay `cls_stmt`):**
+
+```json
+{
+  "symbol": "cls_stmt",
+  "children": [
+    { "symbol": "kw_cls", "text": "CLS", "children": [] },
+    { "symbol": "kw_with", "text": "WITH", "children": [] },
+    { "symbol": "ident", "text": "foo", "children": [] }
+  ],
+  "variant": null
+}
+```
+
+### Semantics proposal
+
+`pass(slot)` where `slot` is bound to production `P`:
+
+1. Always emit an AST node whose `symbol` is `P` (the production name).
+2. Children are the transformed slots of the matched alternative (terminals and subtrees).
+3. Set `variant` from the alternative label (`#program`, `#bare`, …) when present.
+4. **Never** hoist a sole child terminal to replace the `P` node unless the transform explicitly says `pass(thatTerminal)` at the **same** rule (not a parent rule).
+
+### EduBASIC statements affected (bare input → wrong symbol today)
+
+| Input | Expected symbol | Actual symbol today |
+|-------|-----------------|---------------------|
+| `CLS` | `cls_stmt` | `kw_cls` |
+| `PRINT` | `print_stmt` | `kw_print` |
+| `END` | `end_stmt` (`variant: program`) | `kw_end` |
+| `ELSE` | `else_stmt` | `kw_else` |
+| `WEND` | `wend_stmt` | `kw_wend` |
+| `UEND` | `uend_stmt` | `kw_uend` |
+| `TRY` | `try_stmt` | `kw_try` |
+| `CATCH` | `catch_stmt` | `kw_catch` |
+| `FINALLY` | `finally_stmt` | `kw_finally` |
+| `RETURN` | `return_stmt` | `kw_return` |
+| `NEXT` | `next_stmt` | `kw_next` |
+| `DO` | `do_stmt` (`variant: bare`) | `kw_do` |
+| `LOOP` | `loop_stmt` (`variant: bare`) | `kw_loop` |
+| `RANDOMIZE` | `randomize_stmt` | `kw_randomize` |
+
+`END IF` correctly yields `end_stmt` with `variant: if` because multiple children prevent collapse.
+
+### Consumer cleanup after fix
+
+Delete [`keyword-fallbacks.ts`](../src/lang/parsing/ast-bridge/statements/keyword-fallbacks.ts) and remove the fallback call from [`dispatch.ts`](../src/lang/parsing/ast-bridge/statements/dispatch.ts). Remove duplicate `buildNullaryControlFlowStatement` paths that exist only for `*_stmt` vs `kw_*` pairs.
+
+### Suggested parser-lr tests
+
+Add fixtures under `grammars/fixtures/pass-preserves-production/`:
+
+- `bare-terminal.grammar` (reproducer above)
+- `labeled-variant.grammar` (`end_stmt` with `#program` / `#if`)
+- `optional-present.grammar` (`CLS WITH x`)
+- `optional-absent.grammar` (`CLS`)
+
+Assert `node.symbol` after transform, not only parse success.
+
+### Suggested CLI warning
+
+At `table generate` time, warn when:
+
+- Parent transform uses `pass(boundSlot)`
+- Bound production has **no** `transform` rule
+- Production can match with exactly one terminal child
+
+---
+
+## Priority 2: Statement-level `ast` + `transform` (parity with expressions)
+
+### Problem
+
+EduBASIC `ast` section today only describes expression shapes. Statements cannot use `build` / `flatten` / labeled variants.
+
+### Target pattern for EduBASIC
+
+After Priority 1, EduBASIC should be able to add rules like:
+
+```ebnf
+ast
+    cls_stmt =
+        #bare kw_cls
+      | #with kw_cls kw_with [bg]:expr
+      ;
+
+    print_stmt =
+        #bare kw_print
+      | #items kw_print [list]:print_list
+      ;
+
+    print_list =
+        #list [first]:print_segment { [rest]:print_segment }
+      ;
+
+    print_segment =
+        #bare [value]:expr
+      | #trailingComma [value]:expr [comma]:comma
+      | #trailingSemicolon [value]:expr [semi]:semicolon
+      ;
+
+    end_stmt =
+        #program kw_end
+      | #if kw_end kw_if
+      | #unless kw_end kw_unless
+      | #select kw_end kw_select
+      | #sub kw_end kw_sub
+      | #try kw_end kw_try
+      ;
+
+transform
+    cls_stmt ->
+        #bare cls_stmt.#bare(kw_cls)
+      | #with cls_stmt.#with(kw_cls, kw_with, bg) ;
+
+    print_stmt ->
+        #bare print_stmt.#bare(kw_print)
+      | #items print_stmt.#items(kw_print, list) ;
+
+    print_list ->
+        #list flatten(print_list.#list, first, rest) ;
+
+    print_segment ->
+        #bare print_segment.#bare(value)
+      | #trailingComma print_segment.#trailingComma(value, comma)
+      | #trailingSemicolon print_segment.#trailingSemicolon(value, semi) ;
+
+    end_stmt ->
+        #program end_stmt.#program(kw_end)
+      | #if end_stmt.#if(kw_end, kw_if)
+      | … ;
+```
+
+Bridge code then becomes:
+
+```typescript
+export function buildClsStatement(node: AstNode): ClsStatement
+{
+    if (node.variant === 'with')
+    {
+        return new ClsStatement(buildExpression(findChildBySymbol(node, 'bg')!));
+    }
+
+    return new ClsStatement();
+}
+```
+
+No scanning for `kw_cls` at the root, no `findExpressionRoot` guessing.
+
+### Enhancement requests for parser-lr
+
+1. **Document** recommended statement authoring pattern (mirror expression docs).
+2. **Optional shorthand:** if `ast` defines `cls_stmt` but `transform` omits it, apply identity build `cls_stmt.#alt(...)` from grammar bindings (opt-in flag `--infer-identity-transforms`).
+3. **Table validation:** error if `transform` references `type.#variant` not declared in `ast`.
+
+---
+
+## Priority 3: Stable shapes for nested statement helpers
+
+### 3a. `help_topic` collapse
+
+**Grammar:**
+
+```ebnf
+help_stmt = kw_help help_topic ;
+help_topic = #keyword help_kw | #punct help_punct ;
+```
+
+**Input:** `HELP PRINT`
+
+**Observed:** `help_stmt` with children `kw_help`, `kw_print` (no `help_topic` wrapper).
+
+**Desired:** either preserve `help_topic` node:
+
+```json
+{
+  "symbol": "help_stmt",
+  "children": [
+    { "symbol": "kw_help", "text": "HELP" },
+    {
+      "symbol": "help_topic",
+      "variant": "keyword",
+      "children": [{ "symbol": "kw_print", "text": "PRINT" }]
+    }
+  ]
+}
+```
+
+or require explicit transform:
+
+```ebnf
+help_topic ->
+    #keyword help_topic.#keyword(help_kw)
+  | #punct help_topic.#punct(help_punct) ;
+```
+
+**Consumer workaround today:** [`buildHelpStatement`](../src/lang/parsing/ast-bridge/statements/io.ts) scans for any child that is not `kw_help`.
+
+### 3b. `case_selector` and `comparison_op`
+
+**Grammar:**
+
+```ebnf
+case_selector =
+    #value expr
+  | #range expr kw_to expr
+  | #relational kw_is comparison_op expr
+  ;
+
+comparison_op =
+    #less less
+  | #greater_equal greater_equal
+  | … ;
+```
+
+**Input:** `CASE IS < 0`
+
+**Observed `case_selector` (variant `relational`) children:** `kw_is`, `less`, `postfix_expr` — no `comparison_op` node.
+
+**Desired:**
+
+```json
+{
+  "symbol": "case_selector",
+  "variant": "relational",
+  "children": [
+    { "symbol": "kw_is", "text": "IS" },
+    {
+      "symbol": "comparison_op",
+      "variant": "less",
+      "children": [{ "symbol": "less", "text": "<" }]
+    },
+    { "symbol": "postfix_expr", "…": "…" }
+  ]
+}
+```
+
+With transform:
+
+```ebnf
+case_selector ->
+    #value case_selector.#value(expr)
+  | #range case_selector.#range(expr, kw_to, expr)
+  | #relational case_selector.#relational(kw_is, comparison_op, expr) ;
+
+comparison_op ->
+    #less comparison_op.#less(less)
+  | #greater_equal comparison_op.#greaterEqual(greater_equal)
+  | … ;
+```
+
+**Consumer workaround today:** [`buildCaseSelector`](../src/lang/parsing/ast-bridge/statements/control-flow.ts) matches raw token names (`less_equal`) via `isComparisonOpSymbol`.
+
+### 3c. `print_list` / repeat tails
+
+**Grammar:**
+
+```ebnf
+print_stmt = kw_print [ print_list ] ;
+print_list = print_segment { print_segment } ;
+```
+
+Bridge must handle:
+
+- `print_list` wrapper present or absent
+- `print_list$repeat_0` flatten tails
+- bare trailing `postfix_expr` siblings (when list flattening inlines items)
+
+**Desired:** consistent `flatten(print_list.#list, first, rest)` output (same as `expr_list` for expressions).
+
+**Consumer workaround today:** [`buildPrintSegmentsFromStatement`](../src/lang/parsing/ast-bridge/statements/io.ts) accepts `print_list`, `print_segment`, or raw expression children interchangeably.
+
+### 3d. `let_target` / assignment paths
+
+**Grammar:**
+
+```ebnf
+let_stmt = kw_let let_target compound_op expr ;
+let_target = identifier assn_path_suffix ;
+```
+
+Simple `LET x% = 1` often arrives as flat `identifier`, `equal`, `expr` without `let_target` wrapper.
+
+**Desired:** always `let_target` node with `identifier` child and optional `assn_path_suffix` list (possibly empty).
+
+**Consumer workaround today:** [`assignment-helpers.ts`](../src/lang/parsing/ast-bridge/assignment-helpers.ts) handles both `let_target` and bare `identifier` at the statement root.
+
+---
+
+## Priority 4: `flatten()` repeat semantics
+
+### Problem
+
+EduBASIC relies on `flatten` for expression lists, array literals, struct members, and (target) print lists. Bridge code defends against:
+
+- Empty repeat tail nodes (`postfix_expr` with `variant: primary` and zero children) — see [`isEmptyPostfixTail`](../src/lang/parsing/ast-bridge/ast-node-helpers.ts)
+- Repeat segments named `foo$repeat_0` that drop or duplicate items
+
+### Minimal reproducer grammar
+
+```ebnf
+name "flatten-repeat-repro" ;
+
+tokens
+    ident = /[a-z]+/ ;
+    comma = /,/ ;
+
+start list ;
+
+grammar
+    list = item { comma item } ;
+
+ast
+    list = #list [first]:item { [rest]:item } ;
+
+transform
+    list -> flatten(list.#list, first, rest) ;
+```
+
+**Input:** `a,b,c`
+
+**Required:** `list` node with three `item` payloads in order; no spurious empty tail nodes that confuse consumers.
+
+### Migration note
+
+EduBASIC previously carried a local patch to `applyFlatten` in `node_modules/parser-lr` for repeat-list correctness. That fix belongs upstream with fixture tests.
+
+---
+
+## Priority 5: `build()` optional slot handling
+
+### Problem
+
+When a production has optional slots (`[ kw_with expr ]`) and transform uses `build`, absent optionals should be omitted from the child list (or surfaced as explicit `null`), not shift subsequent bindings.
+
+### Minimal reproducer
+
+```ebnf
+cls_stmt = kw_cls [ kw_with ident ] ;
+
+ast
+    cls_stmt =
+        #main kw_cls [withKw]:kw_with [target]:ident
+      ;
+
+transform
+    cls_stmt -> cls_stmt.#main(kw_cls, withKw, target) ;
+```
+
+**Input:** `CLS` — `withKw` and `target` absent.
+
+**Required:** `cls_stmt` node with only `kw_cls` child; optional named slots not required for downstream indexing.
+
+### Migration note
+
+EduBASIC previously patched `applyBuild` optional-slot skip in `node_modules/parser-lr`.
+
+---
+
+## Priority 6: Labeled alternatives on unary productions
+
+### Problem
+
+Productions like `end_stmt` use labeled alternatives:
+
+```ebnf
+end_stmt =
+    #program kw_end
+  | #if kw_end kw_if
+  ;
+```
+
+When Priority 1 is fixed, `variant` on the output `end_stmt` node must reflect `#program` vs `#if` even for bare `END` (single child `kw_end`).
+
+**Input:** `END` → `{ symbol: "end_stmt", variant: "program", children: […] }`
+
+**Input:** `END IF` → `{ symbol: "end_stmt", variant: "if", children: […] }`
+
+Same for `do_stmt` / `loop_stmt` (`#bare`, `#while`, `#until`).
+
+---
+
+## Priority 7: Tooling and packaging
+
+| Item | Rationale |
+|------|-----------|
+| `parser-lr table validate` | Check transform/`ast` consistency, pass-collapse warnings |
+| Fixture corpus | Grammars + inputs + expected AST JSON (like EduBASIC cases above) |
+| CommonJS `require` entry | Jest consumers (see `jest.config.js` `moduleNameMapper` for `parser-lr`) |
+| Document `pass` vs `build` vs `flatten` | Include “do not collapse production symbol” rule |
+
+---
+
+## What “elegant EduBASIC grammar” looks like after fixes
+
+1. **Remove** duplicate `grammar` / `ast` / `transform` blocks (EduBASIC currently duplicates statement sections ~lines 611–1400 and ~1694–2123).
+2. **Add** `ast` + `transform` for all `*_stmt` productions (or rely on Priority 1 + identity build).
+3. **Delete** [`keyword-fallbacks.ts`](../src/lang/parsing/ast-bridge/statements/keyword-fallbacks.ts).
+4. **Shrink** bridge helpers that only exist for flat/collapsed trees (`buildPrintSegmentsFromStatement` multi-path scan, `buildHelpStatement` flat child scan, `isComparisonOpSymbol` for `CASE IS`).
+5. **Keep** [`dispatch.ts`](../src/lang/parsing/ast-bridge/statements/dispatch.ts) as a thin `symbol → builder` table keyed on stable `*_stmt` names.
+
+Rough expected line reduction in `ast-bridge/`: on the order of **15–25%**, mostly from deleting fallback and defensive tree-walk code.
+
+---
+
+## Suggested implementation order for parser-lr agents
+
+1. Fix `pass()` production-symbol preservation + tests (unblocks everything).
+2. Land `flatten` and `build` optional-slot fixes with fixtures.
+3. Add `table validate` warnings for missing statement transforms.
+4. Document statement authoring guide with full mini-grammar (calc-style, but statements).
+5. Work with EduBASIC to add statement `ast`/`transform` incrementally and delete fallbacks.
+
+---
+
+## EduBASIC verification checklist
+
+After each parser-lr release, run in this repo:
+
+```bash
+npm install parser-lr@<version>
+npm run parser:table
+npm test
+```
+
+Key tests:
+
+- [`specs/units/parsers/grammar-table.spec.ts`](../specs/units/parsers/grammar-table.spec.ts) — bare `CLS`, bare `PRINT`
+- [`specs/units/parsers/ast-bridge.spec.ts`](../specs/units/parsers/ast-bridge.spec.ts) — dispatch and `kw_print` fallback (remove after Priority 1)
+- [`specs/units/parsers/parser-service.spec.ts`](../specs/units/parsers/parser-service.spec.ts) — full statement coverage
+- [`programs/SelectCaseTorture.bas`](../programs/SelectCaseTorture.BAS) — `CASE IS` relational selectors
+
+When Priority 1 ships, add an assertion that `parseSource('CLS').symbol === 'cls_stmt'` and remove `keyword-fallbacks` from the codebase in the same PR that bumps the parser-lr version.