rip-lang 3.13.26 → 3.13.28

package/docs/RIP-INTERNALS.md

@@ -1,593 +0,0 @@
-<img src="https://raw.githubusercontent.com/shreeve/rip-lang/main/docs/assets/rip.png" style="width:50px" /> <br>
-
-# Rip Internals
-
-> Architecture, design decisions, and technical reference for the Rip compiler.
-
----
-
-## Table of Contents
-
-1. [Why Rip](#1-why-rip)
-2. [Architecture](#2-architecture)
-3. [S-Expressions](#3-s-expressions)
-4. [Lexer & Rewriter](#4-lexer--rewriter)
-5. [Code Generation](#5-code-generation)
-6. [Compiler](#6-compiler)
-7. [Solar Parser Generator](#7-solar-parser-generator)
-8. [Debug Tools](#8-debug-tools)
-9. [Future Work](#9-future-work)
-
----
-
-# 1. Why Rip
-
-## The Short Version
-
-1. **Simplicity scales** — S-expressions make compilers 50% smaller and 10x easier to maintain
-2. **Zero dependencies** — True autonomy from the npm ecosystem
-3. **Modern output** — ES2022 everywhere, no legacy baggage
-4. **Reactivity as operators** — `:=`, `~=`, `~>` are language syntax, not library imports
-5. **Self-hosting** — Rip compiles itself, including its own parser generator
-
-## Why S-Expressions
-
-Most compilers use complex AST node classes. Rip uses **simple arrays**:
-
-```javascript
-// Traditional AST (CoffeeScript, TypeScript, Babel)
-class BinaryOp {
-  constructor(op, left, right) { ... }
-  compile() { /* 50+ lines */ }
-}
-
-// Rip's S-Expression
-["+", left, right]  // That's it!
-```
-
-**Result:** CoffeeScript's compiler is 17,760 LOC. Rip's is ~10,400 LOC — smaller, yet includes a complete reactive runtime, type system, component system, and source maps.
-
-> **Transform the IR (s-expressions), not the output (strings).**
-
-This single principle eliminates entire categories of bugs. When your IR is simple data (arrays), transformations are trivial and debuggable:
-
-```javascript
-// Debugging: inspect the data directly
-console.log(sexpr);
-// ["comprehension", ["*", "x", 2], [["for-in", ["x"], ["array", 1, 2, 3]]], []]
-
-// vs. string manipulation
-console.log(code);
-// "(() => {\n  const result = [];\n  for (const x of arr) {\n..."
-```
-
-## Rip vs CoffeeScript
-
-| Feature | CoffeeScript | Rip |
-|---------|-------------|------|
-| Optional chaining | 4 soak operators | ES6 `?.` / `?.[]` / `?.()` + shorthand `?[]` / `?()` |
-| Ternary | No | `x ? a : b` and `a if x else b` |
-| Regex features | Basic | Ruby-style (`=~`, indexing, captures in `_`) |
-| Async shorthand | No | Dammit operator (`!`) |
-| Void functions | No | `def fn!` |
-| Reactivity | None | `:=`, `~=`, `~>` |
-| Comprehensions | Always IIFE | Context-aware |
-| Modules | CommonJS | ES6 |
-| Classes | ES5 | ES6 |
-| Dependencies | Multiple | **Zero** |
-| Parser generator | External (Jison) | **Built-in (Solar)** |
-| Self-hosting | No | **Yes** |
-| Total LOC | 17,760 | ~10,400 |
-
-## Design Principles
-
-- **Simplicity scales** — Simple IR, clear pipeline, minimal code, comprehensive tests
-- **Zero dependencies is a feature** — No supply chain attacks, no version conflicts, no `node_modules` bloat
-- **Self-hosting proves quality** — Rip compiles its own parser generator; if it can compile itself, it works
-
----
-
-# 2. Architecture
-
-## The Pipeline
-
-```
-Source Code  →  Lexer  →  emitTypes  →  Parser  →  S-Expressions  →  Codegen  →  JavaScript
-                (1,761)    (types.js)    (359)       (arrays + .loc)     (3,293)      + source map
-                              ↓
-                           file.d.ts (when types: "emit")
-```
-
-## Key Files
-
-| File | Purpose | Lines | Modify? |
-|------|---------|-------|---------|
-| `src/lexer.js` | Lexer + Rewriter | 1,761 | Yes |
-| `src/compiler.js` | Compiler + Code Generator | 3,293 | Yes |
-| `src/types.js` | Type System (lexer sidecar) | 1,099 | Yes |
-| `src/components.js` | Component System (compiler sidecar) | 1,750 | Yes |
-| `src/sourcemaps.js` | Source Map V3 Generator | 189 | Yes |
-| `src/tags.js` | HTML Tag Classification | 62 | Yes |
-| `src/parser.js` | Generated parser | 359 | No (auto-gen) |
-| `src/grammar/grammar.rip` | Grammar specification | 944 | Yes (carefully) |
-| `src/grammar/solar.rip` | Parser generator | 929 | No |
-
-## Example Flow
-
-```coffee
-# Input
-x = 42
-
-# Tokens (from lexer)
-[["IDENTIFIER", "x"], ["=", "="], ["NUMBER", "42"]]
-
-# S-Expression (from parser)
-["program", ["=", "x", 42]]
-
-# Generated Code (from codegen)
-"x = 42;"
-```
-
----
-
-# 3. S-Expressions
-
-S-expressions are simple arrays that serve as Rip's intermediate representation (IR). Each has a **head** (string identifying node type) and **rest** (arguments/children).
-
-## Complete Node Type Reference
-
-### Top Level
-```javascript
-['program', ...statements]
-```
-
-### Variables & Assignment
-```javascript
-['=', target, value]
-['+=', target, value]  // And all compound assigns: -=, *=, /=, %=, **=
-['&&=', target, value]
-['||=', target, value]
-['?=', target, value]   // Maps to ??=
-['??=', target, value]
-```
-
-### Functions
-```javascript
-['def', name, params, body]     // Named function
-['->', params, body]            // Thin arrow (unbound this)
-['=>', params, body]            // Fat arrow (bound this)
-
-// Parameters can be:
-'name'                          // Simple param
-['rest', 'name']                // Rest: ...name
-['default', 'name', expr]      // Default: name = expr
-['expansion']                   // Expansion marker: (a, ..., b)
-['object', ...]                 // Object destructuring
-['array', ...]                  // Array destructuring
-```
-
-### Calls & Property Access
-```javascript
-[callee, ...args]               // Function call
-['await', expr]                 // Await
-['.', obj, 'prop']              // Property: obj.prop
-['?.', obj, 'prop']             // Optional: obj?.prop
-['[]', arr, index]              // Index: arr[index]
-['optindex', arr, index]        // Optional: arr?.[index]
-['optcall', fn, ...args]        // Optional: fn?.(args)
-['new', constructorExpr]        // Constructor
-['super', ...args]              // Super call
-['tagged-template', tag, str]   // Tagged template
-```
-
-### Data Structures
-```javascript
-['array', ...elements]          // Array literal
-['object', ...pairs]            // Object literal (pairs: [key, value])
-['...', expr]                   // Spread (prefix only)
-```
-
-### Operators
-```javascript
-// Arithmetic
-['+', left, right]   ['-', left, right]   ['*', left, right]
-['/', left, right]   ['%', left, right]   ['**', left, right]
-
-// Comparison (== compiles to ===)
-['==', left, right]  ['!=', left, right]
-['<', left, right]   ['<=', left, right]
-['>', left, right]   ['>=', left, right]
-
-// Logical
-['&&', left, right]  ['||', left, right]  ['??', left, right]
-
-// Unary
-['!', expr]          ['~', expr]          ['-', expr]
-['+', expr]          ['typeof', expr]     ['delete', expr]
-['++', expr, isPostfix]                   ['--', expr, isPostfix]
-
-// Special
-['instanceof', expr, type]
-['?', expr]                     // Existence check
-```
-
-### Control Flow
-```javascript
-['if', condition, thenBlock, elseBlock?]
-['unless', condition, body]
-['?:', condition, thenExpr, elseExpr]   // Ternary
-['switch', discriminant, cases, defaultCase?]
-```
-
-### Loops
-```javascript
-['for-in', vars, iterable, step?, guard?, body]
-['for-of', vars, object, guard?, body]
-['for-as', vars, iterable, async?, guard?, body]  // ES6 for-of on iterables
-['while', condition, body]
-['until', condition, body]
-['loop', body]
-['break']          ['continue']
-['break-if', condition]          ['continue-if', condition]
-```
-
-### Comprehensions
-```javascript
-['comprehension', expr, iterators, guards]
-['object-comprehension', keyExpr, valueExpr, iterators, guards]
-```
-
-### Exceptions
-```javascript
-['try', tryBlock, [catchParam, catchBlock]?, finallyBlock?]
-['throw', expr]
-```
-
-### Classes
-```javascript
-['class', name, parent?, ...members]
-```
-
-### Types
-```javascript
-['enum', name, body]              // Enum declaration (runtime JS)
-// Type aliases, interfaces → handled by rewriter, never reach parser
-```
-
-### Ranges & Slicing
-```javascript
-['..', from, to]      // Inclusive range
-['...', from, to]     // Exclusive range
-```
-
-### Blocks & Modules
-```javascript
-['block', ...statements]        // Multiple statements
-['do-iife', expr]               // Do expression (IIFE)
-['import', specifiers, source]
-['export', statement]
-['export-default', expr]
-['export-all', source]
-['export-from', specifiers, source]
-```
-
----
-
-# 4. Lexer & Rewriter
-
-The lexer (`src/lexer.js`) is a clean reimplementation that replaces the old lexer (3,260 lines) with ~1,760 lines producing the same token vocabulary the parser expects.
-
-## Architecture
-
-- **9 tokenizers** in priority order: identifier, comment, whitespace, line, string, number, regex, js, literal
-- **7 rewriter passes**: removeLeadingNewlines, closeOpenCalls, closeOpenIndexes, normalizeLines, tagPostfixConditionals, addImplicitBracesAndParens, addImplicitCallCommas
-- **Token format**: `[tag, val]` array with `.pre`, `.data`, `.loc`, `.spaced`, `.newLine` properties
-
-## Token Properties
-
-| Property | Type | Purpose |
-|----------|------|---------|
-| `.pre` | number | Whitespace count before this token |
-| `.data` | object/null | Metadata: `{await, predicate, quote, invert, parsedValue, ...}` |
-| `.loc` | `{r, c, n}` | Row, column, length |
-| `.spaced` | boolean | Sugar for `.pre > 0` |
-| `.newLine` | boolean | Preceded by a newline |
-
-## Identifier Suffixes
-
-| Suffix | Data flag | Meaning | JS output |
-|--------|-----------|---------|-----------|
-| `!` | `.data.await = true` | Dammit operator | `await` + base name |
-| `?` | `.data.predicate = true` | Existence check | `(expr != null)` |
-
-The `?` suffix is captured only when NOT followed by `.`, `?`, `[`, or `(` — so `?.` (optional chaining), `??` (nullish coalescing), `?.()`, and `?.[i]` remain unambiguous.
-
-The `!` suffix on `as` in for-loops (`as!`) emits `FORASAWAIT` instead of `FORAS`, enabling `for x as! iterable` as shorthand for `for await x as iterable`.
-
-## Language Changes (3.0 Rewrite)
-
-### Removed
-
-| Feature | Old syntax | Replacement |
-|---------|-----------|-------------|
-| Postfix spread/rest | `x...` | `...x` (ES6 prefix only) |
-| Prototype access | `x::y`, `x?::y` | Direct `.prototype` or class syntax; `::` reserved for type annotations |
-| `is not` contraction | `x is not y` | `x isnt y` |
-
-### Added
-
-| Feature | Syntax | Purpose |
-|---------|--------|---------|
-| `for...as` iteration | `for x as iter` | ES6 `for...of` on iterables (replaces `for x from iter`) |
-| `as!` async shorthand | `for x as! iter` | Shorthand for `for await x as iter` |
-
-### Changed
-
-| Item | Old | New |
-|------|-----|-----|
-| Location data | `locationData` (object) | `.loc = {r, c, n}` |
-| `for...from` keyword | `FORFROM` | `FORAS` |
-| Token metadata | `new String(val)` with props | `.data` object on token |
-| Category arrays | `Array` + `indexOf` | `Set` + `.has()` |
-| Variable style | `const`/`let` mix | All `let` |
-| Rewriter passes | 13 | 7 |
-
-### Preserved
-
-All 9 tokenizer methods, full token vocabulary, implicit call/object/brace detection, string interpolation with recursive sub-lexing, heredoc indent processing, arrow function parameter tagging, `do` IIFE support, `for own x of obj`, all reactive operators (`:=`, `~=`, `~>`, `=!`), all Rip aliases (`and`, `or`, `is`, `isnt`, `not`, `yes`, `no`, `on`, `off`).
-
----
-
-# 5. Code Generation
-
-The compiler (`src/compiler.js`) transforms s-expressions into JavaScript. The `CodeGenerator` class is a dispatch table — s-expression heads map to generator methods.
-
-## Context-Aware Generation
-
-Some patterns generate different code based on usage context:
-
-```javascript
-generate(sexpr, context = 'statement') {
-  // context can be 'statement' or 'value'
-}
-```
-
-**Comprehensions** are the primary example:
-
-```coffee
-# Statement context (result discarded) → Plain loop
-console.log x for x in arr
-
-# Value context (result used) → IIFE with array building
-result = (x * 2 for x in arr)
-```
-
-| Parent Node | Child | Context | Reason |
-|-------------|-------|---------|--------|
-| Assignment | RHS | `'value'` | Value assigned to variable |
-| Call | Arguments | `'value'` | Values passed to function |
-| Return | Expression | `'value'` | Value returned from function |
-| Function | Last statement | `'value'` | Implicit return |
-| Function | Non-last statements | `'statement'` | Result discarded |
-| Loop | Body | `'statement'` | Loops don't return values |
-| If/Unless | Branches | Inherit parent | Pass through context |
-| Array | Elements | `'value'` | Values stored in array |
-
-## Variable Scoping
-
-CoffeeScript semantics: function-level scoping with closure access.
-
-- `collectProgramVariables()` — Walks top-level, stops at functions
-- `collectFunctionVariables()` — Walks function body, stops at nested functions
-- Filters out outer variables (accessed via closure)
-- Emits `let` declarations at scope top
-
-## Auto-Detection
-
-Functions automatically become async or generators:
-
-```coffee
-# Contains await or dammit → becomes async
-def loadData(id)
-  user = getUser!(id)
-  user.posts
-
-# Contains yield → becomes generator
-counter = ->
-  yield 1
-  yield 2
-```
-
-## Existence Check
-
-| Syntax | Compiles To |
-|--------|-------------|
-| `x?` | `(x != null)` |
-| `obj.prop?` | `(obj.prop != null)` |
-| `x ?? y` | `x ?? y` |
-| `x ??= 10` | `x ??= 10` |
-
-Optional chaining uses ES6 syntax (both forms supported):
-
-| Syntax | Compiles To |
-|--------|-------------|
-| `obj?.prop` | `obj?.prop` |
-| `arr?.[0]` | `arr?.[0]` |
-| `fn?.(x)` | `fn?.(x)` |
-| `arr?[0]` | `arr?.[0]` |
-| `fn?(x)` | `fn?.(x)` |
-
-## Range Optimization
-
-```coffee
-for i in [1...100]
-  process(i)
-# → for (let i = 1; i < 100; i++) { process(i); }
-```
-
-## String & Regex Processing
-
-String tokens carry metadata in `.data`:
-- `quote`: The quote delimiter (`"`, `'`, `"""`, `'''`, `///`)
-- `quote.length === 3`: Indicates a heredoc
-
-Heredocs use the closing delimiter's column position as the baseline for indentation stripping.
-
-REGEX tokens store `delimiter` and optional `heregex` flags in `token.data`.
-
----
-
-# 6. Compiler
-
-The compiler (`src/compiler.js`) is a clean reimplementation replacing the old compiler (6,016 lines) with ~3,293 lines producing identical JavaScript output.
-
-## Structure
-
-```
-CodeGenerator class
-  - GENERATORS dispatch table (~55 generators)
-  - Variable collection (program + function scope)
-  - Main generate() dispatch
-  - ~55 generate* methods
-  - Body/formatting/utility helpers
-  - Reactive runtime (inline string, ~270 lines)
-Compiler class (with shim adapter for new lexer)
-Convenience exports
-```
-
-## Metadata Bridge
-
-Two one-line helpers isolate all `new String()` awareness:
-
-```javascript
-let meta = (node, key) => node instanceof String ? node[key] : undefined;
-let str  = (node) => node instanceof String ? node.valueOf() : node;
-```
-
-The `Compiler` class's lexer adapter reconstructs `new String()` wrapping from the new lexer's `token.data` property, so grammar actions pass metadata through s-expressions unchanged.
-
-## Removed Generators
-
-| Generator | S-expr | Reason |
-|-----------|--------|--------|
-| `generatePrototype` | `::` | Feature removed from lexer |
-| `generateOptionalPrototype` | `?::` | Feature removed from lexer |
-
-## Renamed: `for-from` → `for-as`
-
-- `GENERATORS['for-as']` replaces `GENERATORS['for-from']`
-- Grammar adds `FORASAWAIT` token: `for x as! iter` → `for await x as iter`
-- Both forms produce the same s-expression: `["for-as", vars, iterable, true, guard, body]`
-
-## Consolidation
-
-| Area | Old lines | New lines | Reduction |
-|------|-----------|-----------|-----------|
-| Total file | 6,016 | ~3,290 | **45%** |
-| Body generation | ~500 | ~200 | 60% |
-| Variable collection | ~230 | ~100 | 57% |
-| Helper methods | ~600 | ~250 | 58% |
-
----
-
-# 7. Solar Parser Generator
-
-**Solar** is a complete SLR(1) parser generator included with Rip — written in Rip, compiled by Rip, zero external dependencies.
-
-**Location:** `src/grammar/solar.rip` (929 lines)
-
-## Grammar Syntax
-
-```coffeescript
-o = (pattern, action, options) ->
-  pattern = pattern.trim().replace /\s{2,}/g, ' '
-  [pattern, action ? 1, options]
-```
-
-**Style 1: Pass-Through** — Omit action, returns first token:
-```coffeescript
-Expression: [
-  o 'Value'
-  o 'Operation'
-]
-```
-
-**Style 2: S-Expression** — Bare numbers become token references:
-```coffeescript
-For: [
-  o 'FOR ForVariables FOROF Expression Block', '["for-of", 2, 4, null, 5]'
-]
-```
-
-**Style 3: Advanced** — `$n` patterns for conditional logic:
-```coffeescript
-Parenthetical: [
-  o '( Body )', '$2.length === 1 ? $2[0] : $2'
-]
-```
-
-## Performance
-
-| Metric | Jison | Solar |
-|--------|-------|-------|
-| Parse time | 12,500ms | ~50ms |
-| Dependencies | Many | Zero |
-| Self-hosting | No | Yes |
-| Code size | 2,285 LOC | 929 LOC |
-
-After modifying `src/grammar/grammar.rip`:
-
-```bash
-bun run parser    # Regenerates src/parser.js
-```
-
----
-
-# 8. Debug Tools
-
-```bash
-# See tokens from lexer
-echo 'x = 42' | ./bin/rip -t
-
-# See s-expressions from parser
-echo 'x = 42' | ./bin/rip -s
-
-# See generated JavaScript
-echo 'x = 42' | ./bin/rip -c
-
-# Interactive REPL with debug modes
-./bin/rip
-rip> .tokens  # Toggle token display
-rip> .sexp    # Toggle s-expression display
-rip> .js      # Toggle JS display
-```
-
-## REPL Architecture
-
-The CLI REPL (`src/repl.js`) uses a `vm`-based sandbox with two eval paths:
-
-- **Primary:** `vm.runInContext` with async IIFE wrapper — handles all code including `await`
-- **Fallback:** `vm.SourceTextModule` — used only when `import` statements are present (required for module linking)
-
-This split works around Bun bug [#22287](https://github.com/oven-sh/bun/issues/22287) where `vm.SourceTextModule` crashes on top-level `await`.
-
-The stdlib is injected once at REPL startup via `getStdlibCode()` — the same function the compiler uses, ensuring a single source of truth. Variables persist across evaluations via a `__vars` object.
-
-The browser REPL (`docs/index.html`) uses a hidden iframe as its sandbox. Code is compiled with `skipPreamble: true` and wrapped in an async IIFE for `await` support. The stdlib and reactive runtime are injected into the iframe context at initialization.
-
----
-
-# 9. Future Work
-
-- Parser update to read `.data` directly instead of `new String()` properties
-- Once parser supports `.data`, the `meta()`/`str()` helpers become trivial to update
-
----
-
-**See Also:**
-- [RIP-LANG.md](RIP-LANG.md) — Language reference (includes reactivity deep dive)
-- [RIP-TYPES.md](RIP-TYPES.md) — Type system specification
-
----
-
-*Rip 3.13 — 1,265 tests — Zero dependencies — Self-hosting — ~13,500 LOC*