rip-lang 1.0.0

package/AGENT.md

@@ -0,0 +1,623 @@
+# AI Agent Guide for Rip
+
+**Purpose:** This document helps AI assistants understand and work with the Rip language compiler.
+
+**What is Rip:** An elegant scripting language that compiles to modern JavaScript (ES2022), featuring zero dependencies, self-hosting capability, and an S-expression intermediate representation.
+
+---
+
+## Quick Reference
+
+### What Rip Does
+
+```
+Rip Source Code → JavaScript (ES2022)
+
+# Input
+def greet(name)
+  "Hello, ${name}!"
+
+# Output
+function greet(name) {
+  return `Hello, ${name}!`;
+}
+```
+
+### The Compilation Pipeline
+
+```
+Source → Lexer → Parser → S-Expressions → Codegen → JavaScript
+         3,145    340 LOC   (arrays!)       4,738      ES2022
+         LOC      generated                 LOC
+```
+
+**Components:**
+1. **Lexer** (`src/lexer.js`) - CoffeeScript 2.7 lexer (battle-tested, 15 years)
+2. **Parser** (`src/parser.js`) - SLR(1) parser generated by solar.rip
+3. **Codegen** (`src/codegen.js`) - Pattern-matches s-expressions → JavaScript
+
+---
+
+## Essential Commands
+
+### Running Code
+
+```bash
+# Run .rip files directly (Bun loader auto-compiles)
+bun script.rip
+
+# Interactive REPL
+./bin/rip
+
+# Compile to JavaScript
+./bin/rip input.rip
+./bin/rip -o output.js input.rip
+```
+
+### Testing
+
+```bash
+# All tests (864 total)
+bun run test
+
+# Specific test file
+bun test/runner.js test/rip/functions.rip
+
+# During development (bypass Bun cache)
+bun --no-cache test/runner.js test/rip
+```
+
+### Debugging
+
+```bash
+# See tokens (lexer output)
+echo 'x = 42' | ./bin/rip -t
+
+# See s-expressions (parser output)
+echo 'x = 42' | ./bin/rip -s
+
+# See JavaScript (codegen output)
+echo 'x = 42' | ./bin/rip -c
+```
+
+### Build Commands
+
+```bash
+bun run parser   # Rebuild parser from grammar (self-hosting)
+bun run browser  # Build 43KB browser bundle
+bun run serve    # Start dev server (REPL at localhost:3000)
+```
+
+---
+
+## Key Architecture Concepts
+
+### S-Expressions (The Core Innovation)
+
+Rip uses **simple arrays** instead of complex AST node classes:
+
+```javascript
+// Traditional AST approach (complex)
+class BinaryOp {
+  constructor(op, left, right) { ... }
+  compile() { /* 50+ lines */ }
+}
+
+// Rip's s-expression approach (simple)
+['+', left, right]
+
+// Code generation
+case '+': {
+  const [left, right] = rest;
+  return `(${this.generate(left)} + ${this.generate(right)})`;
+}
+```
+
+**S-expression examples:**
+- Assignment: `["=", "x", 42]`
+- Function: `["def", "add", ["a", "b"], body]`
+- Binary op: `["+", left, right]`
+- Function call: `[callee, ...args]` (array without string head)
+
+### Context-Aware Generation
+
+The codegen passes a `context` parameter to determine how to compile patterns:
+
+```javascript
+generate(sexpr, context = 'statement')
+// context: 'statement' | 'value'
+```
+
+**Example - Comprehensions:**
+- **Statement context** (result discarded) → Plain loop
+- **Value context** (result used) → IIFE with array building
+
+### Variable Scoping
+
+CoffeeScript-style function scoping with closure access:
+
+```javascript
+// Program level
+let a, b, fn;
+
+// Function level - only NEW variables
+fn = function() {
+  let x, y;     // New variables
+  a = 1;        // Uses outer 'a' (closure)
+};
+```
+
+**Implementation:**
+- `collectProgramVariables()` - Top-level vars
+- `collectFunctionVariables()` - Function-local vars (excludes outer vars)
+
+### Auto-Detection
+
+Functions automatically become `async` or `function*`:
+
+```javascript
+containsAwait(sexpr)  // → async function
+containsYield(sexpr)  // → function*
+```
+
+Stops at function boundaries (nested functions checked separately).
+
+---
+
+## Critical Patterns
+
+### Block Unwrapping
+
+The parser wraps statements in `["block", ...statements]` everywhere. **Always unwrap:**
+
+```javascript
+if (Array.isArray(body) && body[0] === 'block') {
+  const statements = body.slice(1); // Unwrap!
+}
+```
+
+### for-of Variable Ordering
+
+**Critical:** When using `for own k, v of obj when guard`, the order matters:
+
+```javascript
+for (const k in obj) {
+  if (obj.hasOwnProperty(k)) {    // 1. Own check FIRST
+    const v = obj[k];              // 2. Assign value SECOND
+    if (v > 5) {                   // 3. Guard check THIRD (can use v!)
+      // body
+    }
+  }
+}
+```
+
+**Bug to avoid:** Never check guard before value variable is assigned!
+
+### String Object Metadata
+
+The lexer attaches metadata as String objects (not primitives):
+
+```javascript
+// Check BEFORE converting to primitive
+if (sexpr instanceof String) {
+  const metadata = sexpr.quote || sexpr.heregex || sexpr.await;
+}
+```
+
+**Key metadata:**
+- `.quote` - Original quote type (`'` or `"`)
+- `.heregex` - Extended regex flag
+- `.await` - Dammit operator flag
+
+---
+
+## Language Features Quick Reference
+
+### Dual Optional Syntax (10 operators)
+
+**CoffeeScript soak style:**
+- `arr?[0]` → `(arr != null ? arr[0] : undefined)`
+- `fn?(x)` → `(typeof fn === 'function' ? fn(x) : undefined)`
+
+**ES6 optional chaining:**
+- `obj?.prop` → `obj?.prop` (native)
+- `arr?.[0]` → `arr?.[0]` (native)
+- `fn?.(x)` → `fn?.(x)` (native)
+
+Both can be mixed: `obj?.arr?[0]`
+
+### Sigil Operators (!)
+
+**At call-site (dammit):** Calls AND awaits
+```rip
+result = fetchData!      # → await fetchData()
+user = getUser!(id)      # → await getUser(id)
+```
+
+**At definition (void):** Suppresses returns
+```rip
+def process!             # → function process() { ...; return; }
+  doWork()
+```
+
+### Comprehensions (Context-Aware)
+
+- **Value context** (result used) → IIFE builds array
+- **Statement context** (result discarded) → Plain loop
+
+See `docs/COMPREHENSIONS.md` for complete rules.
+
+### CoffeeScript Compatibility
+
+The lexer automatically converts:
+- Postfix spread: `args...` → `...args`
+- Legacy existential: `x ? y` → `x ?? y` (unless ternary)
+
+---
+
+## File Organization
+
+### Main Source Files
+
+| File | Purpose | Can Modify? |
+|------|---------|-------------|
+| `src/lexer.js` | Tokenization + rewriter | ⚠️ Rewriter only |
+| `src/parser.js` | S-expression parser | ❌ Generated (don't edit) |
+| `src/codegen.js` | JavaScript generator | ✅ Main work happens here |
+| `src/compiler.js` | Pipeline orchestration | ✅ Yes |
+| `src/repl.js` | Terminal REPL | ✅ Yes |
+| `src/browser.js` | Browser integration | ✅ Yes |
+
+### Grammar and Generator
+
+| File | Purpose | Can Modify? |
+|------|---------|-------------|
+| `src/grammar/grammar.rip` | Grammar specification | ⚠️ Expert only |
+| `src/grammar/solar.rip` | Parser generator | ❌ No (given) |
+
+**To regenerate parser:** `bun run parser`
+
+### Test Files
+
+| Directory | Contents |
+|-----------|----------|
+| `test/rip/` | 20 test files, 864 tests total |
+| `test/runner.js` | Test framework |
+
+**Test types:**
+```rip
+test "name", "code", expectedResult  # Execute and compare
+code "name", "input", "output"       # Compare generated code
+fail "name", "code"                  # Expect compilation failure
+```
+
+### Documentation
+
+| File | Content |
+|------|---------|
+| `docs/CODEGEN.md` | 110+ node types reference |
+| `docs/COMPREHENSIONS.md` | Context-aware generation |
+| `docs/SOLAR.md` | Parser generator guide |
+| `docs/STRING.md` | String metadata reference |
+| `docs/REGEX-PLUS.md` | Ruby-style regex features |
+
+---
+
+## Development Workflow
+
+### Modifying Existing Features
+
+1. Check what parser emits: `echo 'code' | ./bin/rip -s`
+2. Find the case in `src/codegen.js`
+3. Modify the generation logic
+4. Run tests: `bun run test`
+5. Commit
+
+### Adding New Features
+
+1. Check if lexer/parser already support it
+2. If grammar change needed: edit `src/grammar/grammar.rip` and run `bun run parser`
+3. Add case to `src/codegen.js`
+4. Write tests in `test/rip/`
+5. Document in `docs/CODEGEN.md`
+6. Run tests and commit
+
+### Debugging Issues
+
+1. Use `-s` to see parser output
+2. Use `-c` to see codegen output
+3. Use `-t` to see lexer tokens
+4. Check for String object metadata
+5. Verify context parameter usage
+
+---
+
+## Common Issues & Solutions
+
+### "Unknown s-expression type: X"
+
+**Problem:** Codegen missing a case for this pattern
+
+**Solution:**
+```bash
+echo 'your code' | ./bin/rip -s  # See what parser emits
+grep "case 'X':" src/codegen.js  # Check if implemented
+# Add the missing case if needed
+```
+
+### "Unexpected token" in Output
+
+**Problem:** Generated JavaScript has syntax error
+
+**Solution:**
+```bash
+echo 'your code' | ./bin/rip -c  # See generated code
+echo 'your code' | ./bin/rip -s  # Check the s-expression
+# Fix the codegen logic for that pattern
+```
+
+### Tests Not Reflecting Changes
+
+**Problem:** Bun aggressively caches compiled modules
+
+**Solution:**
+```bash
+bun --no-cache test/runner.js test/rip
+```
+
+### Block Not Unwrapping
+
+**Problem:** Treating `["block", ...]` as a single node
+
+**Solution:** Always check and unwrap:
+```javascript
+if (Array.isArray(body) && body[0] === 'block') {
+  const statements = body.slice(1);
+}
+```
+
+---
+
+## Key Implementation Details
+
+### Helper Functions in Codegen
+
+**Variable collection:**
+- `collectProgramVariables(sexpr)` - Top-level
+- `collectFunctionVariables(body)` - Function-local
+
+**Body generation:**
+- `generateFunctionBody(body, params, sideEffectOnly)` - Functions with implicit returns
+- `generateLoopBody(body)` - Loops without returns
+- `generateBlockWithReturns(block)` - IIFE blocks
+
+**String processing:**
+- `extractStringContent(strObj)` - Heredoc handling
+- `processHeregex(content)` - Strip whitespace/comments
+
+**Detection:**
+- `containsAwait(sexpr)` - Auto-async
+- `containsYield(sexpr)` - Auto-generator
+
+### Critical Edge Cases
+
+**1. for-of with guards and value variables:**
+```javascript
+// Correct order: own check → value assign → guard check
+for (const k in obj) {
+  if (obj.hasOwnProperty(k)) {
+    const v = obj[k];  // BEFORE guard
+    if (guard) { }     // After v is defined
+  }
+}
+```
+
+**2. Postfix conditionals in assignments:**
+```rip
+x = 5 unless done
+# Generate: if (!done) x = 5;
+# NOT: x = (!done ? 5 : undefined)  ← Would always assign!
+```
+
+**3. Switch in value context:**
+```rip
+result = switch x
+  when 1 then 'one'
+  when 2 then 'two'
+# Needs IIFE wrapper, not statement form
+```
+
+---
+
+## Quick Pattern Reference
+
+### Common S-Expression Patterns
+
+```javascript
+// Assignment
+["=", target, value]
+
+// Function definition
+["def", "name", params, body]
+
+// Arrow functions
+["->", params, body]   // Thin arrow (unbound this)
+["=>", params, body]   // Fat arrow (bound this)
+
+// Conditionals
+["if", condition, thenBlock, elseBlock?]
+["unless", condition, body]
+["?:", condition, trueExpr, falseExpr]  // Ternary
+
+// Loops
+["for-in", vars, iterable, step?, guard?, body]
+["for-of", vars, object, guard?, body]
+["while", condition, body]
+
+// Data structures
+["array", ...elements]
+["object", ...pairs]  // pairs: [key, value]
+
+// Operators
+["+", left, right]
+["==", left, right]  // Maps to ===
+["&&", left, right]
+
+// Property access
+[".", obj, "prop"]
+["?..", obj, "prop"]  // Optional
+["[]", arr, index]
+["?[]", arr, index]   // Soak
+
+// Special
+["await", expr]
+["yield", expr]
+["return", expr?]
+```
+
+See `docs/CODEGEN.md` for complete catalog (110+ node types).
+
+---
+
+## Test Framework
+
+### Three Test Types
+
+```rip
+# Execute code and compare result
+test "addition", "1 + 2", 3
+
+# Compare generated JavaScript
+code "assignment", "x = 42", "x = 42"
+
+# Expect compilation failure
+fail "invalid syntax", "let x ="
+```
+
+### Test Best Practices
+
+- Use `\n` for simple tests (1-2 lines)
+- Use `"""` for complex tests (3+ lines)
+- Never use `;` to separate statements (parser treats as single statement)
+- Test files are in `test/rip/*.rip` (20 files, organized by feature)
+
+---
+
+## Self-Hosting
+
+Rip compiles itself, including its parser generator:
+
+```bash
+# Rebuild the parser in one command
+bun run parser
+
+# What happens:
+# 1. Bun runs solar.rip (parser generator, written in Rip)
+# 2. Solar reads grammar.rip (grammar spec, written in Rip)
+# 3. Outputs parser.js (complete parser)
+```
+
+**Zero external tools required.** Everything needed to modify and rebuild Rip is included.
+
+---
+
+## When Working with Rip
+
+### Most Common Task: Fix or Extend Codegen
+
+1. Determine the s-expression pattern: `echo 'code' | ./bin/rip -s`
+2. Find or add the case in `src/codegen.js`
+3. Implement the generation logic
+4. Test: `bun run test`
+5. Commit
+
+### Less Common: Modify Grammar
+
+1. Edit `src/grammar/grammar.rip`
+2. Regenerate parser: `bun run parser`
+3. Update codegen if new node types added
+4. Test thoroughly
+5. Document in `docs/CODEGEN.md`
+
+### Understanding Existing Code
+
+1. Use `-s` flag to see s-expressions
+2. Search `src/codegen.js` for the pattern
+3. Read tests in `test/rip/` for examples
+4. Check `docs/CODEGEN.md` for documentation
+
+---
+
+## Important Notes
+
+### Zero Dependencies
+
+Rip has **zero runtime or build dependencies**. This is intentional and must be maintained. Everything needed is in the source:
+
+- Full compiler
+- Parser generator (solar.rip)
+- Test framework
+- Browser bundler
+- REPL (terminal, browser, console)
+
+### Bun Loader
+
+The `bunfig.toml` + `rip-loader.ts` enable automatic `.rip` file execution:
+
+```bash
+bun script.rip  # Compiles on-the-fly, no build step
+```
+
+This is the primary way to use Rip. Deno/Node require pre-compilation.
+
+### ES2022 Target
+
+Rip outputs modern JavaScript that runs in:
+- Bun (primary target)
+- Browsers (modern)
+- Deno
+- Node.js 12+
+
+**Features used:** ES6 classes, let/const, arrow functions, template literals, destructuring, async iteration, optional chaining, nullish coalescing, static class fields, top-level await
+
+---
+
+## Quick Tips
+
+1. **Start with tests** - Look at passing tests to understand patterns
+2. **Use `-s` liberally** - See exactly what parser emits
+3. **Context matters** - Pass correct context ('statement' vs 'value') to children
+4. **Check String objects** - Metadata flows through String objects
+5. **Unwrap blocks** - Parser wraps statements in blocks everywhere
+6. **Test immediately** - Don't modify without running tests
+
+---
+
+## Documentation Map
+
+- **AGENT.md** (this file) - AI assistant guide
+- **README.md** - User documentation
+- **docs/CODEGEN.md** - Complete node type reference (110+ types)
+- **docs/COMPREHENSIONS.md** - Context-aware generation rules
+- **docs/SOLAR.md** - Parser generator + grammar guide
+- **docs/STRING.md** - String metadata reference
+- **docs/REGEX-PLUS.md** - Ruby-style regex (=~ and heregex)
+- **docs/BROWSER.md** - Browser bundle + triple REPL
+- **docs/COFFEESCRIPT-COMPARISON.md** - Feature comparison
+
+---
+
+## Philosophy
+
+**Simplicity scales.**
+
+- Keep the IR simple (s-expressions)
+- Keep the pipeline clear (lex → parse → generate)
+- Keep the code minimal (pattern matching)
+- Test everything (864/864 tests passing)
+
+Rip achieves CoffeeScript's elegance with 50% less code by using s-expressions instead of complex AST nodes. The result is easier to understand, extend, and maintain.
+
+---
+
+**For AI Assistants:** You have everything needed to work with Rip. The code is well-tested, the architecture is clear, and the documentation is comprehensive. Trust the tests, use the debug tools, and follow the patterns already established.