rip-lang 1.0.0

package/docs/STRING.md

@@ -0,0 +1,363 @@
+# CoffeeScript STRING Token Properties: Complete Reference
+
+## Overview
+
+In CoffeeScript's lexer and parser, the STRING token carries several metadata properties that are essential for correctly transforming source strings into JavaScript output. These properties handle the complex edge cases of string literals, particularly heredocs (triple-quoted strings) and interpolated strings.
+
+When the lexer encounters a string in source code, it doesn't just extract the raw text. It also annotates the token with metadata about HOW that string was written, which affects how it should be compiled to JavaScript.
+
+## The STRING Token Structure
+
+A STRING token is not just a simple string value. It's an object with these properties:
+- The string content itself (after slicing off quotes with `$1.slice 1, -1`)
+- `quote`: The quote delimiter used
+- `initialChunk`: Boolean flag for first chunk in interpolated string
+- `finalChunk`: Boolean flag for last chunk in interpolated string
+- `indent`: The common indentation to strip from heredocs
+- `double`: Whether backslashes should be doubled in output
+- `heregex`: Object with regex flags for extended regex literals
+
+## Property Descriptions and Usage
+
+### 1. `quote` - Quote Delimiter Type
+
+**Purpose:** Records which quote characters were used to delimit the string in source code.
+
+**Possible Values:**
+- `"` - double quote
+- `'` - single quote
+- `"""` - triple double quote (heredoc)
+- `'''` - triple single quote (heredoc)
+- `"///"` - heregex (extended regex literal)
+- `\`` - backtick (for template literals, though less common in CoffeeScript)
+
+**Why It Exists:**
+CoffeeScript needs to know the original quote style for several reasons:
+1. To determine if a string is a heredoc (`quote.length is 3`)
+2. To calculate proper location data (source maps) by accounting for quote length
+3. To determine the delimiter character (`quote.charAt(0)`)
+4. To preserve quote style when generating output or AST
+
+**How It's Used:**
+
+In the `StringLiteral` constructor:
+- Normalizes heregex quotes: `@quote = null if @quote is '///'`
+- Sets the `fromSourceString` flag: `@fromSourceString = @quote?`
+- Provides default: `@quote ?= '"'`
+- Determines delimiter: `@delimiter = @quote.charAt 0`
+- Detects heredocs: `heredoc = @quote.length is 3`
+
+In `withoutQuotesInLocationData()`:
+- Adjusts first_column by adding quote length
+- Adjusts last_column by subtracting quote length
+- Adjusts last_column_exclusive by subtracting quote length
+- Adjusts range array by quote length
+
+In `StringWithInterpolations`:
+- Stored and passed through: `{@quote, @startQuote}`
+- Used when converting from StringLiteral: `quote: stringLiteral.quote`
+
+**Example:**
+```coffeescript
+x = "hello"      # quote: "
+y = 'world'      # quote: '
+z = """
+  multiline
+  """            # quote: """
+```
+
+### 2. `initialChunk` - First Chunk of Interpolated String
+
+**Purpose:** Marks whether this STRING token is the very first chunk in an interpolated string.
+
+**Possible Values:**
+- `true` - This is the first string chunk
+- `false` or undefined - Not the first chunk
+
+**Why It Exists:**
+Interpolated strings are broken into multiple tokens (string chunks and interpolation expressions). The first chunk has special whitespace handling rules - leading blank lines should be stripped in certain contexts.
+
+**How It's Used:**
+
+In heredoc processing (when `quote.length is 3`):
+- If `@initialChunk` is true, `LEADING_BLANK_LINE` regex is used to strip the leading blank line
+- Applied: `val = val.replace LEADING_BLANK_LINE, '' if @initialChunk`
+
+In simple string processing:
+- Combined with offset check to detect if a newline is at the very start
+- If `@initialChunk and offset is 0`, convert the newline to empty string instead of a space
+- This prevents unwanted leading whitespace in output
+
+In the lexer's `mergeInterpolationTokens()`:
+- Set on the token: `addTokenData token, initialChunk: yes if i is 0`
+- Only set when the loop index `i` equals `0`
+
+**Example:**
+```coffeescript
+x = """
+    first line
+    second line
+    """
+# The token for "    first line\n    second line\n    " has initialChunk: true
+# Leading blank line stripped if present
+
+y = "prefix #{expr}
+     continuation"
+# "prefix " has initialChunk: true
+# "     continuation" has initialChunk: false
+```
+
+### 3. `finalChunk` - Last Chunk of Interpolated String
+
+**Purpose:** Marks whether this STRING token is the very last chunk in an interpolated string.
+
+**Possible Values:**
+- `true` - This is the final string chunk
+- `false` or undefined - Not the final chunk
+
+**Why It Exists:**
+Similar to `initialChunk`, the final chunk has special whitespace handling - trailing blank lines should be stripped in certain contexts.
+
+**How It's Used:**
+
+In heredoc processing:
+- If `@finalChunk` is true, `TRAILING_BLANK_LINE` regex is used to strip the trailing blank line
+- Applied: `val = val.replace TRAILING_BLANK_LINE, '' if @finalChunk`
+
+In simple string processing:
+- Combined with position check to detect if a newline is at the very end
+- If `@finalChunk and offset + match.length is val.length`, convert the newline to empty string
+- Prevents unwanted trailing whitespace
+
+In the lexer's `mergeInterpolationTokens()`:
+- Set on the token: `addTokenData token, finalChunk: yes if i is $`
+- Where `$` is the last index in the tokens array (`tokens.length - 1`)
+
+**Example:**
+```coffeescript
+x = """
+    some text
+
+    """
+# The string token has finalChunk: true
+# Trailing blank line stripped
+
+y = "start #{x} end"
+# "start " has finalChunk: false
+# " end" has finalChunk: true
+```
+
+### 4. `indent` - Common Indentation to Strip
+
+**Purpose:** Records the common leading whitespace found across all lines of a heredoc that should be stripped during compilation.
+
+**Possible Values:**
+- A string of whitespace characters (spaces/tabs)
+- `null` or undefined if no common indentation
+
+**Why It Exists:**
+Heredocs allow you to write multiline strings with proper indentation in your source code. The compiler needs to strip this "code indentation" to produce the actual intended string content. This enables readable code without affecting the runtime string value.
+
+**How It's Used:**
+
+In heredoc processing:
+- If `@indent` exists, create a regex pattern: `indentRegex = /// \n#{@indent} ///g`
+- Apply the regex to strip the indent: `val = val.replace indentRegex, '\n' if indentRegex`
+- This replaces every occurrence of `\n` followed by the indent string with just `\n`
+
+In the lexer's string processing:
+- The lexer scans all lines in a heredoc
+- Finds the minimum indentation across all non-empty lines
+- Stores this as the `indent` value: `indent = attempt if indent is null or 0 < attempt.length < indent.length`
+- Passed to `mergeInterpolationTokens` for attachment to tokens
+
+**Example:**
+```coffeescript
+x = """
+    Line 1
+    Line 2
+      Indented more
+    """
+# indent would be "    " (4 spaces)
+# After processing:
+# "Line 1\nLine 2\n  Indented more"
+# Note: The extra indent on line 3 is preserved
+
+# With irregular indentation:
+y = """
+  First (2 spaces)
+      Second (6 spaces)
+    Third (4 spaces)
+  """
+# indent would be "  " (2 spaces - the minimum)
+# After processing:
+# "First (2 spaces)\n    Second (6 spaces)\n  Third (4 spaces)"
+```
+
+### 5. `double` - Backslash Doubling Flag
+
+**Purpose:** Indicates whether backslash characters in the string should be doubled when generating JavaScript output.
+
+**Possible Values:**
+- `true` - Double backslashes
+- `false` or undefined - Don't double backslashes
+
+**Why It Exists:**
+JavaScript requires different levels of escaping depending on the context. When CoffeeScript generates string literals, it needs to ensure backslashes are properly escaped. The `double` flag controls whether `\` becomes `\\` in the output.
+
+**How It's Used:**
+
+Passed to the `makeDelimitedLiteral()` helper function in two contexts:
+
+1. **Main value generation:**
+   ```coffeescript
+   @value = makeDelimitedLiteral val, {
+     @delimiter
+     @double
+   }
+   ```
+
+2. **Template literal conversion:**
+   ```coffeescript
+   @unquotedValueForTemplateLiteral = makeDelimitedLiteral val, {
+     delimiter: '`'
+     @double
+     escapeNewlines: no
+     includeDelimiters: no
+     convertTrailingNullEscapes: yes
+   }
+   ```
+
+Inside `makeDelimitedLiteral()`:
+- When processing a backslash: `when backslash then (if double then backslash + backslash else backslash)`
+- When processing other escapes: `when other then (if double then "\\#{other}" else other)`
+- This ensures proper escape sequences in the generated JavaScript
+
+**Example:**
+```coffeescript
+# If double is true:
+"\n" → "\\n" (in generated JS)
+"\\" → "\\\\" (in generated JS)
+
+# If double is false:
+"\n" → "\n" (preserved)
+"\\" → "\\" (preserved)
+```
+
+### 6. `heregex` - Extended Regex Metadata
+
+**Purpose:** Contains metadata about extended regular expression (heregex) literals, including flags.
+
+**Possible Values:**
+- An object with a `flags` property containing regex flags
+- `undefined` if not a heregex
+
+**Why It Exists:**
+CoffeeScript supports "heregexes" - extended regular expressions that allow whitespace and comments for readability. These are delimited with `///` and need special processing. The flags (like `g`, `i`, `m`) need to be tracked and preserved.
+
+**How It's Used:**
+
+In the `StringLiteral` constructor:
+- Detects heregex mode: `if @heregex`
+- Applies heregex-specific transformations:
+  - Removes whitespace and comments: `val = val.replace HEREGEX_OMIT, '$1$2'`
+  - Processes unicode escapes with flags: `val = replaceUnicodeCodePointEscapes val, flags: @heregex.flags`
+
+In the lexer:
+- When tokenizing a heregex, comment tokens are marked: `heregex: yes`
+- The flags are extracted and stored in the heregex object
+- Passed through in `mergeInterpolationTokens`: `addTokenData token, {heregex} if heregex`
+
+**Example:**
+```coffeescript
+pattern = ///
+  ^ \d+      # starts with digits
+  \s*        # optional whitespace
+  [a-z]+     # followed by letters
+  $          # end of string
+///i         # case-insensitive flag
+
+# heregex: { flags: 'i' }
+# The whitespace and comments are stripped
+# The 'i' flag is preserved for the output regex
+```
+
+## How These Properties Work Together
+
+### Interpolated String Processing
+
+When an interpolated string is lexed:
+
+1. The lexer identifies quote type → sets `quote`
+2. For heredocs, calculates common indent → sets `indent`
+3. Breaks string into chunks around interpolations
+4. Marks first chunk → sets `initialChunk: true`
+5. Marks last chunk → sets `finalChunk: true`
+6. Determines escape context → sets `double`
+
+When the string is compiled to JavaScript:
+
+1. Check `quote.length is 3` to detect heredoc
+2. If heredoc and `indent` exists, strip common indentation
+3. If `initialChunk`, strip leading blank line
+4. If `finalChunk`, strip trailing blank line
+5. Use `double` to control backslash escaping
+6. Use `quote` for location data adjustments
+
+### Heregex Processing
+
+1. Lexer detects `///` → sets `quote: '///'`
+2. Flags are parsed → sets `heregex: { flags: '...' }`
+3. In compilation, apply special regex transformations
+4. Preserve flags in output
+
+## Why This Matters
+
+Without these properties, it would be impossible to:
+
+1. **Correctly handle heredoc indentation:** You wouldn't know what common indent to strip
+2. **Properly trim whitespace:** You wouldn't know which chunks are first/last
+3. **Generate accurate source maps:** Quote length affects column positions
+4. **Preserve semantic meaning:** Different quote styles have different escaping rules
+5. **Support readable regex:** Heregex flags and content need special processing
+6. **Match JavaScript semantics:** Backslash doubling must match the output context
+
+## Common Pitfalls Without This Information
+
+When implementing a string processor without understanding these properties:
+
+- **Indentation errors:** Heredocs include unwanted leading whitespace
+- **Newline handling:** Extra blank lines appear at start/end of strings
+- **Escape sequence bugs:** Backslashes aren't properly doubled or preserved
+- **Source map misalignment:** Location data doesn't account for quote characters
+- **Regex compilation failures:** Heregex content isn't properly stripped
+
+## Summary
+
+The STRING token is not just a value - it's a rich data structure that captures the INTENT and CONTEXT of how a string was written in source code. Each property serves a specific purpose in the transformation pipeline from CoffeeScript source to JavaScript output. Understanding these properties is essential for correctly implementing string handling in any CoffeeScript-like compiler.
+
+## REGEX Tokens (Rip Extension)
+
+While STRING tokens have properties directly on the String object, **REGEX tokens store metadata in `token.data`**:
+
+```javascript
+// REGEX token structure:
+token = ['REGEX', String("/pattern/flags"), location]
+token.data = {
+  delimiter: '///',      // '/' for normal regex, '///' for heregex
+  heregex: {flags: 'gi'} // Only for heregex
+}
+```
+
+**The Rewriter's `exposeTokenDataToGrammar()` copies these to the String object:**
+```javascript
+token[1].delimiter = token.data.delimiter;
+token[1].heregex = token.data.heregex;
+```
+
+This allows Rip's codegen to detect and process heregex (extended regex with whitespace/comments) by checking:
+```javascript
+if (strObj.delimiter === '///' && strObj.heregex) {
+  // Process heregex: strip whitespace and comments
+}
+```

package/docs/WHY-NOT-COFFEESCRIPT.md

@@ -0,0 +1,184 @@
+# Why Not CoffeeScript: The Case Against Revival
+
+## Executive Summary
+
+CoffeeScript 2 already achieved the goal of modern JavaScript output in 2017. Any effort toward further CoffeeScript development in 2025 represents a misallocation of resources toward a language that has been definitively superseded by the JavaScript ecosystem's evolution. This document presents the compelling case for why CoffeeScript should remain a respected historical artifact rather than receive continued development.
+
+**TL;DR:** The ecosystem has moved on. The tooling has rotted. The community has dispersed. TypeScript won the type safety war. Modern JavaScript absorbed the best ideas. What's left isn't worth the cost.
+
+## The Fundamental Misconception
+
+Many assume CoffeeScript needs updating to output ES6+. **This is false.** CoffeeScript 2 (released 2017) already generates:
+- ES6 classes with `extends` and `super`
+- Arrow functions with lexical `this`
+- Destructuring assignments
+- Template literals
+- `for...of` loops
+- ES6 modules (`import`/`export`)
+- Spread operators
+- Computed property names
+
+The output is already modern. The problem isn't technical—it's existential.
+
+## 1. Ecosystem Abandonment: The Network Effect in Reverse
+
+### The Developer Desert
+- **Hiring Crisis**: Try posting a "CoffeeScript Developer" position in 2025. You'll get tumbleweeds or developers who haven't touched it since 2015.
+- **Knowledge Decay**: Senior developers who knew CoffeeScript have moved on. Junior developers have never heard of it.
+- **AI Blindness**: GitHub Copilot, ChatGPT, and other AI assistants are trained on modern codebases. Their CoffeeScript suggestions are often outdated or incorrect.
+- **Dead Community**: The CoffeeScript forum has ~5 posts per month. The TypeScript forum has ~500. The network effect that once benefited CoffeeScript now strangles it.
+
+### Library Lockout
+- **TypeScript Hegemony**: Every major library ships with `.d.ts` files. Using them from CoffeeScript means abandoning type safety entirely.
+- **No Native Integrations**: Modern frameworks (Next.js, Nuxt, SvelteKit) assume TypeScript or JavaScript. CoffeeScript requires custom configuration that often breaks with updates.
+- **Security Vulnerabilities**: When was the last time CoffeeScript's dependencies were audited? The maintenance burden falls on a tiny, overwhelmed community.
+
+## 2. The Type Safety Revolution Left CoffeeScript Behind
+
+### TypeScript Won by Solving Real Problems
+- **Refactoring at Scale**: Rename a method in TypeScript, and your IDE updates every call site. In CoffeeScript, you grep and pray.
+- **API Contracts**: TypeScript interfaces document and enforce API contracts. CoffeeScript relies on... documentation? Comments? Hope?
+- **Compile-Time Catches**: TypeScript catches errors before runtime. CoffeeScript catches them... in production.
+- **IntelliSense**: Modern developers expect autocomplete that actually works. CoffeeScript's IDE support is "here's a list of every word in your file."
+
+### The Cost of Being Typeless in 2025
+- **Integration Nightmare**: Every third-party library requires manual type annotations or living without IDE support
+- **Team Scaling Breaks**: CoffeeScript works for 1-2 developers who hold the entire codebase in their heads. At 10+ developers, the lack of types becomes a coordination disaster.
+- **No Migration Path**: You can gradually adopt TypeScript in a JavaScript codebase. CoffeeScript requires a full rewrite.
+
+## 3. Tooling Degradation: Death by a Thousand Cuts
+
+### IDE Abandonment
+- **VS Code**: CoffeeScript extension last meaningful update: 2019. TypeScript: Updated weekly. The gap widens daily.
+- **WebStorm**: CoffeeScript support officially deprecated as of 2024. JetBrains gave up.
+- **Vim/Neovim**: CoffeeScript LSP? Might as well ask for a steam-powered smartphone.
+- **GitHub Copilot**: Trained on modern codebases. CoffeeScript suggestions are often wrong or outdated.
+- **Cursor/AI IDEs**: The future of development doesn't include CoffeeScript in their training data.
+
+### Build Tool Afterthought
+- **Vite**: Requires a third-party plugin that breaks every major update
+- **Webpack**: CoffeeScript loader is in maintenance mode
+- **Bun**: Can bundle TypeScript natively. CoffeeScript? "What's that?"
+- **Deno**: Built for TypeScript-first development. CoffeeScript is alien technology.
+
+### Developer Experience Regression
+- **No Prettier Support**: Your team will have formatting wars like it's 2010
+- **ESLint**: Limited rules, most plugins incompatible
+- **Source Maps**: They exist, but debugging through transpilation layers is always worse than native
+- **Hot Module Replacement**: Works... sometimes... if you hold your mouth right
+
+## 4. Modern JavaScript Didn't Just Catch Up—It Surpassed
+
+### JavaScript Absorbed CoffeeScript's Best Ideas
+- **Optional Chaining**: JS `?.` is everywhere now
+- **Nullish Coalescing**: JS `??` handles null/undefined elegantly
+- **Arrow Functions**: JS `=>` is more explicit about `this` binding
+- **Destructuring**: JS destructuring is more powerful and flexible
+- **Classes**: JS classes are native, optimized, and well-understood
+- **Template Literals**: JS backticks are standard
+
+### What's Left Is Not Enough
+- **Significant Whitespace**: Python developers like it. JavaScript developers see it as a footgun.
+- **Implicit Returns**: Causes more bugs than it prevents. Explicit is better than implicit.
+- **Everything is an Expression**: Clever, but creates debugging nightmares.
+- **Existential Operator**: JavaScript's `?.` and `??` cover 95% of use cases more clearly.
+
+## 5. The Performance and Security Argument
+
+### Runtime Overhead
+- **Extra Transpilation**: Every file goes through an additional compilation step
+- **Larger Bundles**: CoffeeScript's runtime helpers add weight
+- **Slower Development**: Live reload, HMR, all slower due to the compilation step
+- **Memory overhead**: Build tools keep both CS and JS versions in memory
+
+### Security Concerns
+- **Supply Chain Risk**: CoffeeScript itself is a dependency with minimal maintenance
+- **Outdated Dependencies**: The CS ecosystem doesn't get security updates quickly
+- **Audit Complexity**: Security tools are built for JS/TS, not CoffeeScript
+
+## 6. The Business Case Against CoffeeScript
+
+### Technical Debt Multiplication
+- **Training Costs**: Every new hire needs CoffeeScript training
+- **Maintenance Burden**: You maintain both the code AND the build pipeline
+- **Migration Inevitable**: Every CS codebase eventually migrates to JS/TS
+- **Opportunity Cost**: Time spent on CS tooling is time not spent on features
+
+### Project Risk Factors
+- **Bus Factor of One**: Often only one person really knows the CS codebase
+- **Vendor Lock-in**: Stuck with outdated tools that support CoffeeScript
+- **Recruitment Nightmare**: Developers actively avoid CoffeeScript positions
+- **Technical Isolation**: Can't easily adopt new tools, libraries, or practices
+
+## 7. The False Economy of Syntax Sugar
+
+### The "Conciseness" Trap
+CoffeeScript is more concise, but:
+- **Readability > Brevity**: Explicit code is easier to maintain
+- **Debugging Nightmare**: Less syntax means more implicit behavior to debug
+- **Onboarding Barrier**: New developers need to learn a whole new syntax
+- **Copy-Paste Broken**: Can't use code examples from Stack Overflow, documentation, or AI assistants
+
+### The Real Cost of "Beautiful" Code
+- **Cognitive Load**: Developers must mentally transpile CS to JS to understand behavior
+- **Error Messages**: Stack traces reference generated JS, not source CS
+- **Code Review**: Reviewers need CS expertise, limiting your reviewer pool
+
+## 8. The Innovation Dead End
+
+### Language Evolution Frozen
+- **No New Features**: When was the last significant CoffeeScript language feature?
+- **No Standards Process**: JavaScript has TC39. CoffeeScript has... GitHub issues?
+- **No Corporate Backing**: Microsoft backs TypeScript. Who backs CoffeeScript?
+
+### Missing Modern Features
+- **No Decorators**: The rest of the ecosystem is adopting them
+- **No Private Fields**: Modern JS has `#private`, CS has... conventions?
+- **No Pattern Matching**: TC39 is working on it for JS, CS will never get it
+- **No Type Parameters**: Generic programming is impossible
+
+## 9. The Maintainer Burden
+
+### The Human Cost
+- **Burnout**: The few CS maintainers are overwhelmed
+- **No Succession Plan**: When current maintainers leave, who takes over?
+- **Feature Requests**: The backlog grows while maintainer energy shrinks
+- **Breaking Changes**: Fear of breaking existing code prevents evolution
+
+### The Quality Spiral
+- **Bug Accumulation**: Bugs pile up faster than they're fixed
+- **Documentation Rot**: Outdated docs are worse than no docs
+- **Example Decay**: Sample code uses patterns from 2015
+
+## 10. The Strategic Dead End
+
+### No Path Forward
+- **No Migration Strategy**: You can't gradually move from CS to modern alternatives
+- **No Ecosystem Integration**: New tools don't consider CoffeeScript at all
+- **No Corporate Use Cases**: No major company is choosing CoffeeScript for new projects
+- **No Educational Value**: Bootcamps and universities don't teach it
+
+### The Opportunity Cost
+Every hour spent on CoffeeScript 3 could be spent on:
+- **Learning Rust/Go/Zig**: Languages with actual momentum
+- **Contributing to TypeScript**: Where improvements benefit millions
+- **Building Tools**: That solve real problems for modern developers
+- **Creating Libraries**: That the ecosystem actually needs
+
+## The Verdict: Let It Rest
+
+CoffeeScript served its purpose. It showed JavaScript could be better and influenced ES6+. That's a tremendous legacy. But continuing to develop CoffeeScript in 2025 is like maintaining a telegraph network in the age of 5G.
+
+The language works—CoffeeScript 2 outputs modern JavaScript that runs everywhere. But "works" isn't enough when the entire ecosystem has moved on. Development is a team sport, and CoffeeScript has no team left.
+
+### The Harsh Truth
+
+CoffeeScript 3 would be a technical achievement that nobody asked for, solving problems nobody has, for a community that no longer exists. The language's elegant syntax cannot overcome the brutal reality of ecosystem abandonment.
+
+The kindest thing for CoffeeScript is to recognize it as a successful historical artifact that accomplished its mission: pushing JavaScript to become better. That mission is complete. CoffeeScript won by losing—its ideas live on in modern JavaScript while the language itself can finally rest.
+
+**Building CoffeeScript 3 isn't just unnecessary—it's actively harmful to the developers who would be trapped maintaining codebases in a dead language, cut off from the vibrant, innovative JavaScript ecosystem of 2025 and beyond.**
+
+---
+
+*The real question isn't "Why not CoffeeScript 3?" but "Why would anyone choose suffering when better alternatives exist?"*