rip-lang 1.0.0

package/.cursor/rules/rip-agent-onboarding.md

@@ -0,0 +1,681 @@
+# Rip Language - AI Agent Onboarding
+
+**Welcome!** You're working on Rip, an elegant scripting language compiler. This guide gets you productive quickly.
+
+---
+
+## 🚀 Quick Start (5 Minutes)
+
+### Step 1: Read These First (in order)
+
+1. **AGENT.md** (10 min) - Complete architecture, commands, patterns
+2. **README.md** - Skim for features and current status
+3. **CONTRIBUTING.md** - GitHub workflow with real examples
+4. **docs/WORKFLOW.md** - Quick command reference
+
+### Step 2: Understand the Pipeline
+
+```
+Rip Source → Lexer → Parser → S-Expressions → Codegen → JavaScript
+            (3,146)  (340)    (arrays!)      (4,738)    (ES2022)
+```
+
+**Key insight:** S-expressions (simple arrays like `["=", "x", 42]`) are the IR, not complex AST nodes. This makes the compiler 50% smaller than CoffeeScript.
+
+### Step 3: Essential Commands
+
+```bash
+# Debug any code
+echo 'your code' | ./bin/rip -t  # Tokens (lexer)
+echo 'your code' | ./bin/rip -s  # S-expressions (parser)
+echo 'your code' | ./bin/rip -c  # JavaScript (codegen)
+
+# Run tests
+bun run test                              # All 854 tests
+bun test/runner.js test/rip/FILE.rip     # Specific file
+bun --no-cache test/runner.js test/rip   # Clear Bun cache
+
+# Rebuild parser (after grammar changes)
+bun run parser  # Regenerates src/parser.js from grammar.rip
+```
+
+---
+
+## 🎯 What You're Working On
+
+**Check for active issues/branches:**
+```bash
+gh issue list
+git branch -a
+cat ISSUE-*.md  # Handoff docs for complex issues
+```
+
+**If resuming Issue #11 (nested comprehension IIFEs):**
+- Read `ISSUE-11.md` for complete analysis
+- Branch: `fix/nested-comprehension-context`
+- See "Recommended Approach" section
+
+---
+
+## 🏗️ Architecture Essentials
+
+### Files You'll Modify
+
+| File | Purpose | Can Edit? | Rebuild? |
+|------|---------|-----------|----------|
+| `src/codegen.js` | Code generator | ✅ YES | No |
+| `src/grammar/grammar.rip` | Grammar rules | ⚠️ Expert | `bun run parser` |
+| `src/grammar/solar.rip` | Parser generator | ⚠️ Runtime only | `bun run parser` |
+| `src/compiler.js` | Pipeline | ✅ YES | No |
+| `src/lexer.js` | Tokenizer | ⚠️ Rewriter only | No |
+| `src/parser.js` | Parser | ❌ NEVER | Generated |
+| `test/rip/*.rip` | Tests | ✅ YES | No |
+
+### The Golden Rule
+
+**If it's generated, don't edit it directly!**
+- `src/parser.js` is generated by solar.rip from grammar.rip
+- After editing grammar.rip or solar.rip: `bun run parser`
+
+### S-Expression Patterns
+
+```javascript
+["=", "x", 42]                          // Assignment
+["+", left, right]                      // Binary operator
+["def", "fn", params, body]             // Function definition
+["comprehension", expr, iterators, guards]  // Comprehension
+["for-in", vars, iterable, step, guard, body]  // For loop
+```
+
+See `docs/CODEGEN.md` for complete catalog (110+ node types).
+
+---
+
+## 🧪 Testing Philosophy
+
+### Always Write Tests First
+
+```coffeescript
+# test/rip/RELEVANT_FILE.rip
+
+# Execute and compare result
+test "name", "x = 42; x", 42
+
+# Compare generated JavaScript
+code "name", "x + y", "(x + y)"
+
+# Expect compilation failure
+fail "name", "invalid syntax"
+```
+
+### Test Organization
+
+```
+test/rip/
+├── assignment.rip       (43 tests)
+├── async.rip            (29 tests)
+├── comprehensions.rip   (24 tests) ← Nested comprehension tests here
+├── functions.rip        (71 tests)
+├── loops.rip            (25 tests)
+... 15 more files
+Total: 854 tests (100% passing)
+```
+
+### Test-Driven Development
+
+```bash
+# 1. Add failing test
+# 2. Run: bun test/runner.js test/rip/FILE.rip
+# 3. Verify it fails
+# 4. Implement fix
+# 5. Run tests until they pass
+# 6. Run ALL tests: bun run test
+```
+
+---
+
+## 🔧 Common Tasks
+
+### Fix a Bug in Codegen
+
+```bash
+# 1. Debug the pattern
+echo 'failing code' | ./bin/rip -s  # See s-expression
+
+# 2. Find the case
+grep -A 20 "case 'pattern':" src/codegen.js
+
+# 3. Fix the generation logic
+vim src/codegen.js
+
+# 4. Test immediately
+bun test/runner.js test/rip/RELEVANT.rip
+
+# 5. Run all tests
+bun run test
+```
+
+### Add a Grammar Rule
+
+```bash
+# 1. Edit grammar
+vim src/grammar/grammar.rip
+
+# 2. Regenerate parser
+bun run parser  # ~200ms, instant feedback!
+
+# 3. Add codegen case if needed
+vim src/codegen.js
+
+# 4. Test
+bun run test
+```
+
+### Modify Parser Runtime Behavior
+
+```bash
+# Example: Improve error messages (Issue #7)
+
+# 1. Edit solar.rip (NOT parser.js!)
+vim src/grammar/solar.rip
+# Edit the parseError function, parse function, etc.
+
+# 2. Regenerate parser
+bun run parser  # Compiles solar.rip and regenerates parser.js
+
+# 3. Test
+bun run test
+```
+
+---
+
+## 🎓 Key Concepts to Understand
+
+### 1. Context-Aware Generation
+
+**Most important concept in Rip!**
+
+```javascript
+generate(sexpr, context = 'statement')
+// context: 'statement' | 'value'
+```
+
+**Example - Comprehensions:**
+```coffeescript
+# Value context (result used) → IIFE
+result = (x * 2 for x in arr)
+
+# Statement context (result discarded) → Plain loop
+for x in arr
+  x * 2
+doMore()  # ← comprehension not last, result unused
+```
+
+**Read:** `docs/COMPREHENSIONS.md` for complete rules
+
+### 2. Block Unwrapping
+
+Parser wraps statements in `["block", ...]` everywhere:
+
+```javascript
+if (Array.isArray(body) && body[0] === 'block') {
+  const statements = body.slice(1);  // ALWAYS unwrap!
+}
+```
+
+### 3. String Object Metadata
+
+Lexer attaches metadata to String objects (not primitives):
+
+```javascript
+// Check BEFORE converting to primitive
+if (sexpr instanceof String) {
+  const metadata = sexpr.quote || sexpr.heregex || sexpr.await;
+}
+```
+
+### 4. Variable Scoping
+
+CoffeeScript-style function scoping:
+
+```javascript
+// Program level
+let a, b, fn;
+
+// Function level - only NEW variables
+fn = function() {
+  let x, y;  // New vars
+  a = 1;     // Uses outer 'a' (closure)
+};
+```
+
+**Implementation:**
+- `collectProgramVariables()` - Top-level
+- `collectFunctionVariables()` - Function-local (excludes outer)
+
+---
+
+## 🐛 Debugging Tips
+
+### When Tests Fail
+
+```bash
+# 1. Run the specific test
+bun test/runner.js test/rip/FILE.rip
+
+# 2. Check generated code
+echo 'test code' | ./bin/rip -c
+
+# 3. Check s-expression
+echo 'test code' | ./bin/rip -s
+
+# 4. If Bun is caching old code
+bun --no-cache test/runner.js test/rip
+```
+
+### When Code Won't Compile
+
+```bash
+# New improved error messages show line/column!
+./bin/rip -c file.rip
+# → Parse error at line 157, column 27 (token: ...)
+
+# Check tokens
+./bin/rip -t file.rip | grep -A 5 -B 5 "problem"
+```
+
+### When Generated Code Looks Wrong
+
+```bash
+# Compare with CoffeeScript
+coffee -c file.coffee  # CoffeeScript output
+./bin/rip -c file.coffee  # Rip output
+# Compare the two
+```
+
+---
+
+## 📋 GitHub Workflow
+
+**Follow the 10-step workflow in `docs/WORKFLOW.md`**
+
+### Quick Version
+
+```bash
+# 1-2. Find bug, create issue
+gh issue create --title "..." --label "bug"
+
+# 3. Create branch
+git checkout -b fix/issue-name
+
+# 4-6. Write tests, implement, verify
+bun run test  # Must pass!
+
+# 7. Build browser (if code changed)
+bun run browser
+
+# 8-9. Commit with issue reference
+git commit -m "Fix: ...
+
+Fixes #N  ← Auto-closes issue!
+
+All tests passing: X/Y (100%)"
+
+# 10. PR and merge
+git push origin fix/issue-name
+gh pr create --title "..." --body "Fixes #N"
+gh pr merge <number> --squash --delete-branch
+```
+
+**The magic:** `Fixes #N` in commit/PR auto-closes the issue when merged!
+
+---
+
+## ⚠️ Critical Don'ts
+
+### Never Edit These Files Directly
+
+- ❌ `src/parser.js` - Generated file
+- ❌ `src/grammar/solar.rip` - Given (parser generator)
+
+### Always Do These
+
+- ✅ Run `bun run test` before committing
+- ✅ Run `bun run browser` after codegen changes
+- ✅ Include `Fixes #N` in commits
+- ✅ Update test counts in README.md
+- ✅ Follow existing code patterns
+
+---
+
+## 🎯 Current Project Status
+
+**Version:** 1.0.0
+**Tests:** 854/854 passing (100%)
+**Status:** Production-ready, actively maintained
+
+**Recent Achievements:**
+- ✅ Postfix comprehensions with `by` step (Issue #1)
+- ✅ Parser error locations (Issue #7)
+- ✅ Range loops without variable (Issue #9)
+- ✅ bar.coffee compiles successfully
+- ✅ 48% smaller output than CoffeeScript
+
+**Open Issues:**
+- Issue #11: Nested comprehension IIFEs (in progress, see ISSUE-11.md)
+
+**Active Branches:**
+- `fix/nested-comprehension-context` - Issue #11 work
+
+---
+
+## 📚 Documentation Map
+
+**For Developers:**
+- `AGENT.md` - Complete reference (read this first!)
+- `CONTRIBUTING.md` - Workflow with examples
+- `docs/WORKFLOW.md` - Quick command reference
+- `ISSUE-*.md` - Complex issue handoffs
+
+**Technical Reference:**
+- `docs/CODEGEN.md` - All 110+ node types
+- `docs/COMPREHENSIONS.md` - Context rules (CRITICAL for Issue #11!)
+- `docs/SOLAR.md` - Parser generator guide
+- `docs/STRING.md` - String metadata
+- `docs/REGEX-PLUS.md` - Ruby-style regex
+
+**User Docs:**
+- `README.md` - User guide with examples
+
+---
+
+## 🎨 Code Style Principles
+
+From today's session:
+
+1. **Keep it clean** - No ugly hacks, readable code
+2. **Keep it simple** - S-expressions over complex AST
+3. **Keep it tested** - 100% test coverage
+4. **Keep it efficient** - Optimize hot paths
+5. **Keep it documented** - Update docs with changes
+
+**Example of clean code:**
+```javascript
+// ✅ Good - clear, simple
+case '+': {
+  const [left, right] = rest;
+  return `(${this.generate(left, 'value')} + ${this.generate(right, 'value')})`;
+}
+
+// ❌ Bad - overly complex
+case '+': return this.buildBinaryExpression(rest[0], rest[1], '+', {precedence: 5});
+```
+
+---
+
+## 🏆 Session History
+
+**Today (2025-10-31):** Completed 5 full GitHub workflows
+- Issue #1/PR #2: Postfix `by` step support
+- Issue #3/PR #4: Refactor step handling
+- Issue #5/PR #6: Comprehension context fix
+- Issue #7+#9/PR #10: Parser errors + range loops
+- Issue #11: Analysis complete, implementation pending
+
+**All issues auto-closed via `Fixes #N` in PR descriptions!**
+
+---
+
+## 💡 Pro Tips
+
+### Understand Before Changing
+
+```bash
+# Don't guess - inspect!
+echo 'code' | ./bin/rip -s  # See what parser emits
+grep -A 30 "case 'pattern':" src/codegen.js  # See current implementation
+```
+
+### Use Existing Patterns
+
+```bash
+# Find similar cases
+grep -r "similar pattern" src/codegen.js
+# Copy and adapt, don't reinvent
+```
+
+### Test Edge Cases
+
+```coffeescript
+# Don't just test the happy path
+test "normal", "x = 1", 1
+test "with null", "x = null", null
+test "with undefined", "x = undefined", undefined
+test "empty array", "x = []", []
+```
+
+### Commit Often
+
+```bash
+# Small, focused commits are better
+git commit -m "Add grammar rule for X"
+git commit -m "Add codegen for X"
+git commit -m "Add tests for X"
+# vs one huge commit
+```
+
+---
+
+## 🔍 Finding Your Way Around
+
+### Where to Look for Specific Things
+
+**Syntax/Grammar Issues:**
+- Check: `src/grammar/grammar.rip`
+- Regenerate: `bun run parser`
+
+**Code Generation Issues:**
+- Check: `src/codegen.js`
+- Search: `grep "case 'node-type':" src/codegen.js`
+
+**Parser Runtime Issues:**
+- Check: `src/grammar/solar.rip`
+- Functions: `parseError`, `parse`
+- Regenerate: `bun run parser`
+
+**Test Failures:**
+- Check: `test/rip/CATEGORY.rip`
+- Categories: assignment, async, comprehensions, functions, loops, etc.
+
+**Context Rules:**
+- Read: `docs/COMPREHENSIONS.md`
+- This is CRITICAL for understanding statement vs value context
+
+---
+
+## 🚨 Common Pitfalls (Avoid These!)
+
+### ❌ Editing Generated Files
+```bash
+vim src/parser.js  # WRONG - will be overwritten!
+```
+
+### ✅ Edit Source, Then Regenerate
+```bash
+vim src/grammar/grammar.rip  # RIGHT
+bun run parser               # Regenerate parser.js
+```
+
+### ❌ Forgetting to Test
+```bash
+git commit -m "fix bug"  # WRONG - didn't run tests!
+```
+
+### ✅ Always Test First
+```bash
+bun run test  # MUST be 854/854 passing
+git commit
+```
+
+### ❌ Missing Issue Reference
+```bash
+git commit -m "Fix comprehension bug"  # WRONG - issue won't auto-close!
+```
+
+### ✅ Always Reference Issue
+```bash
+git commit -m "Fix: Description
+
+Fixes #11  # RIGHT - auto-closes issue!
+"
+```
+
+### ❌ Breaking Existing Tests
+```bash
+# Made change, 1 test fails
+git commit anyway  # WRONG!
+```
+
+### ✅ Keep 100% Pass Rate
+```bash
+bun run test  # Must be 854/854 passing
+# If tests fail, fix them or revert your change
+```
+
+---
+
+## 🎓 Understanding Rip's Design
+
+### Why S-Expressions?
+
+**Traditional AST:**
+```javascript
+class BinaryOp {
+  constructor(op, left, right) { ... }
+  compile() { /* 50+ lines */ }
+}
+```
+
+**Rip's S-Expressions:**
+```javascript
+case '+': {
+  const [left, right] = rest;
+  return `(${this.generate(left)} + ${this.generate(right)})`;
+}
+```
+
+**Result:** 50% less code, easier to maintain!
+
+### Why Context Parameter?
+
+**Same syntax, different output:**
+```coffeescript
+# Source (identical)
+for x in arr then x * 2
+```
+
+**Output depends on usage:**
+```javascript
+// Used: result = (for x in arr then x * 2)
+(() => {
+  const result = [];
+  for (const x of arr) { result.push(x * 2); }
+  return result;
+})()
+
+// Unused: (for x in arr then x * 2); doMore()
+for (const x of arr) { (x * 2); }
+```
+
+Context-aware generation = smarter, more efficient code!
+
+---
+
+## 📖 Extended Reading
+
+**When you need deep knowledge:**
+
+### For Comprehension Work (Issue #11)
+1. `docs/COMPREHENSIONS.md` - Complete context rules
+2. `ISSUE-11.md` - Nested IIFE problem analysis
+3. `test/rip/comprehensions.rip` - All test cases
+4. `src/codegen.js:2025-2230` - Comprehension IIFE generation
+5. `src/codegen.js:4080-4250` - Plain loop generation
+
+### For Grammar Work
+1. `docs/SOLAR.md` - Parser generator guide
+2. `src/grammar/grammar.rip` - Grammar specification
+3. `src/grammar/solar.rip` - Parser generator source
+
+### For General Development
+1. `docs/CODEGEN.md` - All 110+ node types
+2. `docs/STRING.md` - String metadata
+3. `docs/REGEX-PLUS.md` - Ruby-style regex
+
+---
+
+## 🎯 Your First Task
+
+**Start here:**
+
+1. ✅ Read AGENT.md (you're ready after this!)
+2. ✅ Run `bun run test` (verify everything passes)
+3. ✅ Check `gh issue list` (see what's open)
+4. ✅ If Issue #11 exists, read `ISSUE-11.md`
+5. ✅ Create branch: `git checkout fix/nested-comprehension-context`
+6. ✅ Implement Solution B from ISSUE-11.md
+7. ✅ Follow the workflow in `docs/WORKFLOW.md`
+
+---
+
+## 🌟 Success Criteria
+
+**Before committing:**
+- ✅ All tests pass (854/854 or higher)
+- ✅ Code follows existing patterns
+- ✅ Documentation updated
+- ✅ Browser bundle rebuilt (if codegen changed)
+- ✅ Commit message references issue
+
+**Before merging PR:**
+- ✅ PR description includes `Fixes #N`
+- ✅ All checklist items complete
+- ✅ Clean, readable code
+- ✅ No regressions
+
+---
+
+## 💬 Getting Help
+
+**Resources available:**
+1. Read the docs (comprehensive!)
+2. Check closed PRs (#2, #4, #6, #10) for examples
+3. Review test files for patterns
+4. Use debug flags extensively
+
+**The docs are excellent** - trust them! Everything you need is documented.
+
+---
+
+## 🚀 Philosophy
+
+**From AGENT.md:**
+
+> Simplicity scales.
+> - Keep the IR simple (s-expressions)
+> - Keep the pipeline clear (lex → parse → generate)
+> - Keep the code minimal (pattern matching)
+> - Test everything (854/854 tests passing)
+
+**From today's session:**
+
+> Follow the workflow.
+> Write tests first.
+> Keep it clean.
+> Document everything.
+
+---
+
+**You have everything you need. The codebase is well-tested, well-documented, and ready for you. Trust the process, follow the patterns, and keep those tests passing!** ✨
+
+**Good luck!** 🎉