rip-lang 3.16.1 → 3.16.2

package/src/grammar/README.md

@@ -1,27 +1,25 @@
-# Solar & Lunar — Dual Parser Generators
+# Solar — Rip's Parser Generator
 
-One grammar. Two parsers. Two fundamentally different parsing strategies.
+One grammar. One parser. Mathematically derived.
 
 ```
-grammar.rip ──→ Solar ──→ parser.js     (SLR(1) table-driven, 215KB)
-             └→ Lunar ──→ parser-rd.js  (predictive recursive descent, 110KB)
+grammar.rip ──→ Solar ──→ parser.js   (SLR(1) table-driven)
 ```
 
-Both parsers accept the same token stream from the lexer and produce identical
-s-expression ASTs. They share no code at runtime — they are completely
-independent implementations derived from the same grammar specification.
-
-**Test parity:** 1,162 / 1,182 tests passing (98.3%) — 20 files at 100%.
+The parser accepts the token stream from the lexer and produces s-expression
+ASTs — simple arrays like `["=", "x", 42]` — that the code emitter consumes.
 
 ---
 
 ## Files
 
-| File | Lines | Purpose |
-|------|-------|---------|
-| `grammar.rip` | 944 | Grammar specification — defines all syntax rules |
-| `solar.rip` | 929 | SLR(1) parser generator — produces table-driven parsers |
-| `lunar.rip` | 2,412 | Predictive recursive descent generator — produces PRD parsers |
+| File | Purpose |
+|------|---------|
+| `grammar.rip` | Grammar specification — defines all syntax rules |
+| `solar.rip` | SLR(1) parser generator — produces the table-driven parser |
+
+`src/parser.js` is the generated output. Never edit it by hand — regenerate
+with `bun run parser` after any grammar change.
 
 ---
 
@@ -51,6 +49,13 @@ Operation: [
 ]
 ```
 
+Action format:
+
+- Numbers (`1`, `2`, `...3`) reference matched symbols by position
+- `...N` spreads the array at position N
+- String literals become s-expression nodes: `'["if", 2, 3]'`
+- Default action (no action given) returns position 1
+
 ---
 
 ## Solar (`solar.rip`) — SLR(1) Table Parser Generator
@@ -66,169 +71,23 @@ Grammar → Process Rules → Build LR Automaton → Compute FIRST/FOLLOW
 
 A table-driven parser where every parsing decision is a lookup:
 `parseTable[state][symbol]` → shift, reduce, or accept. The parse table
-encodes 801 states with all transitions delta-compressed for minimal size.
+encodes all states with transitions delta-compressed for minimal size.
 
 ### How to use
 
 ```bash
+bun run parser                                     # Regenerate src/parser.js
 bun src/grammar/solar.rip grammar.rip              # → parser.js (SLR table)
-bun src/grammar/solar.rip -r grammar.rip           # → parser-rd.js (PRD via Lunar)
 bun src/grammar/solar.rip --info grammar.rip       # Show grammar statistics
+bun src/grammar/solar.rip --info --conflicts grammar.rip  # Conflict details
 bun src/grammar/solar.rip --sexpr grammar.rip      # Show grammar as s-expression
+bun src/grammar/solar.rip -o out.js grammar.rip    # Custom output path
 ```
 
-### Integration with Lunar
-
-Solar imports Lunar and installs it with one line:
-
-```coffee
-import { install as installLunar } from './lunar.rip'
-# ... (Generator class definition) ...
-installLunar Generator
-```
-
-This adds `generateRD()` to the Generator prototype. When `-r` is passed,
-Solar calls `generator.generateRD()` instead of `generator.generate()`.
-
----
-
-## Lunar (`lunar.rip`) — Predictive Recursive Descent Generator
-
-Lunar analyzes the same grammar that Solar processes and generates a
-hand-rolled-looking recursive descent parser with Pratt expression parsing.
-
-### Architecture
-
-The generated parser has five layers:
-
-1. **Token management** — `advance()`, `expect()`, `match()`, `loc()`, `withLoc()`
-2. **Speculation** — `mark()`, `reset()`, `speculate()` for backtracking
-3. **Pratt expression parser** — `parseExpression(minBP)` with binding powers
-4. **Nonterminal functions** — one `parseX()` function per grammar nonterminal
-5. **Parser shell** — same API as the table parser (`parser.parse()`, exports)
-
-### How it works
-
-Lunar derives everything from the grammar — no hardcoded token or nonterminal
-names in the core generators:
-
-**Expression analysis** (`_analyzeExpressionRules`) — walks the grammar to detect:
-- Which nonterminal is the "expression" (contains Operation as an alternative)
-- Which is the "operation" (has the most `NT OP NT` binary rules with precedence)
-- Which is the "value" (pure choice nonterminal with the most atom alternatives)
-- Which is "code" (FIRST set contains `->` or `=>`)
-- Assignment operators (nonterminals where all rules are `LHS TOKEN RHS`)
-- Prefix starters (keyword tokens that begin expression alternatives)
-- Postfix chains (left-recursive property/index/call rules through Value chain)
-- Atom types (terminals reachable through the Value nonterminal chain)
-- Expression-handled tokens (for skipping redundant choice alternatives)
-
-**Pratt parser generation** (`_generateRDExpression`) — builds the while loop:
-- Prefix starters dispatch to keyword-led parsers (IF→parseIf, FOR→parseFor, etc.)
-- Assignment operators checked at binding power 0
-- Postfix operators from Operation rules (with INDENT/TERMINATOR variants)
-- Postfix chains from property/index/call rules (resolved to FIRST sets)
-- Infix binary operators with correct associativity and control-flow merging
-- Ternary operators
-- Postfix if/unless/while/until and comprehensions
-- Statement tokens (RETURN, STATEMENT) enter the Pratt loop for postfix patterns
-
-**Nonterminal classification** (`_classifyNonterminal`) — detects patterns:
-
-| Pattern | Detection | Example |
-|---------|-----------|---------|
-| `root` | Grammar start symbol | Root |
-| `body-list` | Left-recursive with TERMINATOR | Body, ComponentBody |
-| `comma-list` | Left-recursive with `,` | ArgList, ParamList |
-| `concat-list` | Left-recursive, no separator | Interpolations, Cases |
-| `left-rec-loop` | Self-referential with terminal continuation | IfBlock |
-| `expression` | Contains the operation nonterminal | Expression |
-| `operation` | Has binary operator rules | Operation |
-| `token` | Single rule, single terminal | Identifier, Property |
-| `keyword` | All rules start with unique terminals | Return, Def, Enum |
-| `choice` | All rules are single-nonterminal passthroughs | Value, Line, Statement |
-| `sequence` | Everything else | Assign, Catch, Block |
-
-**Shared prefix disambiguation** (`_generateRDSharedPrefix`) — handles rules
-that share a common beginning:
-- Optional chain detection with rule-length-based action dispatch
-- Same-token grouping with deeper disambiguation
-- Terminal and nonterminal suffix separation with FIRST set grouping
-- Empty-rule-as-default when nonterminal suffixes have FIRST checks
-
-**Speculation** — `mark()`/`reset()`/`speculate()` for grammar ambiguities:
-- Range vs Array: `[1..10]` vs `[1, 2, 3]` — try Range first
-- Slice vs Expression in INDEX_START: `arr[1..3]` vs `arr[0]`
-- Range vs destructuring in For: `for [1..5]` vs `for [a, b] as iter`
-- Terminal/nonterminal overlap: `...` as Splat vs expansion marker
-- Left-rec-loop lookahead: `ELSE IF` vs `ELSE Block` in IfBlock
-
-### Three specialized generators
-
-Three nonterminals (out of 93) have patterns that require multi-token lookahead
-and can't be handled by the generic generators:
-
-- **For** (34 rules) — FORIN/FOROF/FORAS variants with optional WHEN/BY in both orders
-- **Object** (5 rules) — comprehension vs regular object determined after parsing key:value
-- **AssignObj** (6 rules) — key:value vs key=default vs shorthand vs rest after parsing key
-
-90 of 93 nonterminals (96.8%) are generated purely from grammar analysis.
-
-### Remaining 20 test failures (98.3% → 100%)
-
-The remaining failures cluster into a few fixable categories:
-
-| Category | Tests | Root Cause |
-|----------|-------|------------|
-| **Array elisions** | 5 | `[,1]`, `[1,,3]` — ArgElisionList/Elision not parsed |
-| **Trailing comma** | 1 | `[1,2,]` — OptElisions at end of array |
-| **Export/Import edge** | 3 | `export { x }` without FROM, `export x ~= ...`, `import x, * as m` — deeper shared-prefix disambiguation generates duplicate conditions |
-| **Semicolons context** | 3 | `def foo(); 42` — Def/async without block body (CALL_END before Block) |
-| **Class patterns** | 2 | Class expression without name, `@bar:` static in class body |
-| **Postfix ternary** | 1 | `a = x if true else 0` — assignment context for postfix ternary |
-| **Other edge cases** | 5 | `invalid extends`, `array destructuring skip`, type alias, typed runtime |
-
-**Key fix needed for Export/Import:** The shared-prefix handler generates duplicate
-`else if` branches with identical conditions when two rules share a common prefix
-through nonterminals but diverge at a terminal after parsing (e.g., `} FROM` vs `}`).
-The inner terminal group disambiguator needs to check terminal continuation instead
-of generating separate branches.
-
-**Key fix needed for elisions:** The Array parser routes `[` to Range speculation
-then Array. Array uses ArgElisionList which calls Arg → Expression. But leading
-commas `[,1]` and sparse `[1,,3]` need the Elision nonterminal which the generic
-comma-list handler doesn't generate.
-
----
-
-## Comparison
-
-| Aspect | Solar (SLR) | Lunar (PRD) |
-|--------|-------------|-------------|
-| Strategy | Bottom-up table lookup | Top-down predictive |
-| Output size | 215KB (encoded tables) | 110KB (readable code) |
-| Startup | Decode table on load | Zero initialization |
-| Debugging | "State 437" errors | Named function call stacks |
-| Correctness | Mathematically derived | Grammar-derived + 3 specializations |
-| Test parity | 1,235/1,235 (100%) | 1,162/1,182 (98.3%) |
-| Speed | O(n) tight loop | O(n) function calls |
-| Expressions | Shift/reduce with precedence | Pratt with binding powers |
-
-Both produce identical s-expressions for 98.3% of the test suite. Having two
-independent implementations from the same grammar provides cross-validation.
-
----
-
-## The Innovation
-
-Most parser generators commit to one strategy: Yacc produces LALR tables,
-ANTLR produces LL recursive descent, PEG.js produces PEG parsers. Solar and
-Lunar generate **both** from a single grammar specification.
-
-The key insight: the FIRST/FOLLOW sets and operator precedence table that
-Solar computes for SLR(1) contain all the information a Pratt-based recursive
-descent parser needs. The data is the same — it's just read differently.
-Solar reads it as table entries. Lunar reads it as dispatch conditions and
-binding powers.
+### Workflow for grammar changes
 
-Same grammar. Same semantics. Two fundamentally different parsers.
+1. Edit `src/grammar/grammar.rip`
+2. Run `bun run parser`
+3. Verify the new parse shape with `echo 'code' | ./bin/rip -s`
+4. Update codegen in `src/compiler.js` if the node shape changed
+5. Run `bun run test`

package/src/grammar/grammar.rip

@@ -96,6 +96,7 @@ grammar =
     o 'ComputedAssign'
     o 'ReadonlyAssign'
     o 'Effect'
+    o 'Gate'
     o 'If'
     o 'Try'
     o 'While'
@@ -233,6 +234,13 @@ grammar =
     o 'Assignable READONLY_ASSIGN INDENT Expression OUTDENT', '["readonly", 1, 4]'
   ]
 
+  # Render-ready gate (<~) — load-before-render binding; component bodies only.
+  # One form on purpose: the RHS must be a one-line literal stash path, so the
+  # compiler can hoist it into the component's static gate-set.
+  Gate: [
+    o 'Assignable GATE Expression', '["gate", 1, 3]'
+  ]
+
   # Reactive effect (~>) — side effects that run when dependencies change
   Effect: [
     o 'Assignable EFFECT Expression'               , '["effect", 1, 3]'
@@ -592,14 +600,15 @@ grammar =
     o 'INDENT Body OUTDENT', '["block", ...2]'
   ]
 
-  # Parenthetical mark-up: when the inner expression is a single `=` assignment,
-  # tag it with `.parenthesized = true` so the codegen ternary-hoist doesn't
-  # silently turn `(x = "a") if cond else "b"` (conditional assign) into
-  # `x = (cond ? "a" : "b")` (unconditional). The marker is harmless for any
-  # other consumer; it's just a flag the hoist consults.
+  # Parenthetical mark-up: tag a single parenthesized expression with
+  # `.parenthesized = true`. Consumers that respect explicit grouping
+  # consult it: the codegen ternary-hoist (so `(x = "a") if cond else "b"`
+  # stays a conditional assign) and comparison chaining (so
+  # `(a == b) == c` compares the boolean instead of chaining). The marker
+  # is harmless for any other consumer.
   Parenthetical: [
-    o '( Body )'               , '$2.length === 1 ? (Array.isArray($2[0]) && $2[0][0] === "=" && ($2[0].parenthesized = true), $2[0]) : ["block", ...$2]'
-    o '( INDENT Body OUTDENT )', '$3.length === 1 ? (Array.isArray($3[0]) && $3[0][0] === "=" && ($3[0].parenthesized = true), $3[0]) : ["block", ...$3]'
+    o '( Body )'               , '$2.length === 1 ? (Array.isArray($2[0]) && ($2[0].parenthesized = true), $2[0]) : ["block", ...$2]'
+    o '( INDENT Body OUTDENT )', '$3.length === 1 ? (Array.isArray($3[0]) && ($3[0].parenthesized = true), $3[0]) : ["block", ...$3]'
   ]
 
   OptComma: [
@@ -1004,9 +1013,6 @@ grammar =
     o 'Expression || Expression'      , '["||", 1, 3]'
     o 'Expression ?? Expression'      , '["??", 1, 3]'
 
-    # Pipe
-    o 'Expression PIPE Expression'    , '["|>", 1, 3]'
-
     # Relational
     o 'Expression RELATION Expression', '[2, 1, 3]'     # in, of, instanceof
 
@@ -1046,8 +1052,7 @@ operators = """
   left        ^
   left        |
   left        &&
-  left        ||
-  left        PIPE
+  left        || ??
   right       TERNARY
   nonassoc    INDENT OUTDENT
   right       YIELD

package/src/grammar/solar.rip

@@ -10,7 +10,6 @@
 import * as fs from 'fs'
 import * as path from 'path'
 import { fileURLToPath, pathToFileURL } from 'url'
-import { install as installLunar } from './lunar.rip'
 
 VERSION = '1.5.0'
 
@@ -90,11 +89,9 @@ class Generator
     # Build parser
     @timing '💥 Total time', =>
       @timing 'processGrammar'   , => @processGrammar grammar # Process grammar rules
-      unless @options.rd
-        @timing 'buildLRAutomaton' , => @buildLRAutomaton()     # Build LR(0) automaton
+      @timing 'buildLRAutomaton' , => @buildLRAutomaton()     # Build LR(0) automaton
       @timing 'processLookaheads', => @processLookaheads()    # Compute FIRST/FOLLOW and assign lookaheads
-      unless @options.rd
-        @timing 'buildParseTable'  , => @buildParseTable()      # Build parse table with default actions
+      @timing 'buildParseTable'  , => @buildParseTable()      # Build parse table with default actions
 
   # ============================================================================
   # Helper Functions
@@ -656,8 +653,6 @@ class Generator
     # Decoder reconstructs [{key: val}, ...] at runtime
     "(()=>{let d=[#{data.join ','}],t=[],p=0,n,o,k,a;while(p<d.length){n=d[p++];o={};k=0;a=[];while(n--)k+=d[p++],a.push(k);for(k of a)o[k]=d[p++];t.push(o)}return t})()"
 
-  # Recursive descent generation — installed by Lunar (see lunar.rip)
-
   # ============================================================================
   # Runtime Parser
   # ============================================================================
@@ -772,8 +767,6 @@ class Generator
 # Exports
 # ==============================================================================
 
-installLunar Generator
-
 export { Generator }
 
 # ==============================================================================
@@ -805,7 +798,6 @@ if isRunAsScript
         -v, --version           Show version
         -i, --info              Show grammar information
         -s, --sexpr             Show grammar as s-expression
-        -r, --rd                Generate recursive descent parser (parser-rd.js)
         -c, --conflicts         Show conflict details (use with --info)
         -o, --output <file>     Output file (default: parser.js)
 
@@ -843,7 +835,7 @@ if isRunAsScript
           console.log "     Resolution: #{conflict.resolution} (by default)"
 
     # Parse command line
-    options = {help: false, version: false, info: false, sexpr: false, conflicts: false, rd: false, output: 'parser.js'}
+    options = {help: false, version: false, info: false, sexpr: false, conflicts: false, output: 'parser.js'}
     grammarFile = null
     i = 0
 
@@ -854,7 +846,6 @@ if isRunAsScript
         when '-v', '--version'   then options.version   = true
         when '-i', '--info'      then options.info      = true
         when '-s', '--sexpr'     then options.sexpr     = true
-        when '-r', '--rd'        then options.rd        = true
         when '-c', '--conflicts' then options.conflicts = true
         when '-o', '--output'    then options.output    = process.argv[++i + 2]
         else grammarFile = arg unless arg.startsWith('-')
@@ -915,11 +906,7 @@ if isRunAsScript
       if options.info
         showStats generator
       else
-        if options.rd
-          options.output = 'parser-rd.js' if options.output is 'parser.js'
-          parserCode = generator.generateRD()
-        else
-          parserCode = generator.generate()
+        parserCode = generator.generate()
         fs.writeFileSync options.output, parserCode
         console.log "\nParser generated: #{options.output}"
 

package/src/lexer.js

@@ -129,7 +129,7 @@ let IMPLICIT_UNSPACED_CALL = new Set(['+', '-']);
 // Tokens that end an implicit call
 let IMPLICIT_END = new Set([
   'POST_IF', 'POST_UNLESS', 'FOR', 'WHILE', 'UNTIL',
-  'WHEN', 'BY', 'LOOP', 'TERMINATOR', '||', '&&', 'PIPE',
+  'WHEN', 'BY', 'LOOP', 'TERMINATOR', '||', '&&',
 ]);
 
 // Tokens that trigger implicit comma insertion before arrows
@@ -251,7 +251,7 @@ let UNARY_MATH = new Set(['!', '~']);
 // conflict with ?. (optional chaining), ?? (nullish), ?! (presence), ?.( and ?.[
 let IDENTIFIER_RE = /^(?!\d)((?:(?!\s)[$\w\x7f-\uffff])+(?:!|[?](?![.?![(]))?)([^\n\S]*:(?![=:]))?/;
 let NUMBER_RE     = /^0b[01](?:_?[01])*n?|^0o[0-7](?:_?[0-7])*n?|^0x[\da-f](?:_?[\da-f])*n?|^\d+(?:_\d+)*n|^(?:\d+(?:_\d+)*)?\.?\d+(?:_\d+)*(?:e[+-]?\d+(?:_\d+)*)?/i;
-let OPERATOR_RE   = /^(?:<=>|::|\*>|[-=]>|~>|~=|:=|=!|===|!==|\?\!|\?\?|=~|\|>|[-+*\/%<>&|^!?=]=|>>>=?|([-+:])\1|([&|<>*\/%])\2=?|\?\.?|\.{2,3})/;
+let OPERATOR_RE   = /^(?:<=>|<~|::|\*>|[-=]>|~>|~=|:=|=!|===|!==|\?\!|\?\?|=~|[-+*\/%<>&|^!?=]=|>>>=?|([-+:])\1|([&|<>*\/%])\2=?|\?\.?|\.{2,3})/;
 let WHITESPACE_RE = /^[^\n\S]+/;
 let NEWLINE_RE    = /^(?:\n[^\n\S]*)+/;
 let COMMENT_RE    = /^(\s*)###([^#][\s\S]*?)(?:###([^\n\S]*)|###$)|^((?:\s*#(?!##[^#]).*)+)/;
@@ -748,18 +748,6 @@ export class Lexer {
       }
       if (/^\s+#[a-zA-Z_]/.test(this.chunk)) return 0;  // let lineToken handle indentation first
     }
-    // Schema field modifier: `#` adjacent (unspaced) to an identifier acts
-    // as the unique-marker inside schema bodies (e.g. `email!# email`).
-    // Return 0 so literalToken emits a standalone `#` token; rewriteSchema
-    // absorbs it. Outside schema bodies the `#` token is harmless because
-    // nothing else in the grammar accepts `IDENTIFIER #` without a space.
-    if (this.chunk[0] === '#') {
-      let prev = this.prev();
-      if (prev && !prev.spaced && !prev.newLine &&
-          (prev[0] === 'IDENTIFIER' || prev[0] === 'PROPERTY')) {
-        return 0;
-      }
-    }
     let match = COMMENT_RE.exec(this.chunk);
     if (!match) return 0;
     return match[0].length;
@@ -1388,8 +1376,6 @@ export class Lexer {
       this.inTypeAnnotation = false;
       tag = 'TERMINATOR';
     }
-    // Pipe operator
-    else if (val === '|>') tag = 'PIPE';
     // Type operators
     else if (val === '::' && /^[a-zA-Z_$]/.test(this.chunk[2] || '')) {
       // Prototype access: String::trim → String.prototype.trim
@@ -1403,6 +1389,7 @@ export class Lexer {
     else if (val === '~=') tag = 'COMPUTED_ASSIGN';
     else if (val === ':=') tag = 'REACTIVE_ASSIGN';
     else if (val === '<=>') tag = 'BIND';
+    else if (val === '<~') { tag = 'GATE'; this.inTypeAnnotation = false; }
     else if (val === '~>') { tag = 'EFFECT'; this.inTypeAnnotation = false; }
     else if (val === '=!') { tag = 'READONLY_ASSIGN'; this.inTypeAnnotation = false; }
     // Merge assignment: *>config = {a: 1} → Object.assign(config, {a: 1})
@@ -2063,12 +2050,32 @@ export class Lexer {
         } else if (t && t !== 'TERMINATOR' && t !== 'OUTDENT' && t !== ',') j++;
         logicalKeep = tokens[j]?.[0] === ',';
       }
+      // A postfix conditional on the FIRST property line of a multiline
+      // implicit object must bind to the property's value, not close the
+      // object — otherwise `a: 1 if c` followed by more properties splits
+      // the object into separate arguments. Later property lines already
+      // bind to the value (the sameLine flag is false by then), so only
+      // first-line conditionals need the lookahead: keep the object open
+      // when the property list continues after this line.
+      let objectContinues = (j) => {
+        for (let d = 0; j < tokens.length; j++) {
+          let t = tokens[j][0];
+          if (t === '(' || t === '[' || t === '{' || t === 'CALL_START' || t === 'INDEX_START' || t === 'INDENT') d++;
+          else if (t === ')' || t === ']' || t === '}' || t === 'CALL_END' || t === 'INDEX_END' || t === 'OUTDENT') {
+            if (d === 0) return false;
+            d--;
+          }
+          else if (t === 'TERMINATOR' && d === 0) return this.looksObjectish(j + 1);
+        }
+        return false;
+      };
       if ((IMPLICIT_END.has(tag) && !logicalKeep) || (CALL_CLOSERS.has(tag) && newLine)) {
         while (isImplicit(stackTop())) {
           let [stackTag, , {sameLine, startsLine}] = stackTop();
           if (inImplicitCall() && prevTag !== ',') {
             endImplicitCall();
-          } else if (inImplicitObject() && !isLogicalOp && sameLine && tag !== 'TERMINATOR' && prevTag !== ':') {
+          } else if (inImplicitObject() && !isLogicalOp && sameLine && tag !== 'TERMINATOR' && prevTag !== ':' &&
+                     !((tag === 'POST_IF' || tag === 'POST_UNLESS') && objectContinues(i + 1))) {
             endImplicitObject();
           } else if (inImplicitObject() && tag === 'TERMINATOR' && prevTag !== ',' && !(startsLine && this.looksObjectish(i + 1))) {
             endImplicitObject();