ranger-compiler 3.1.0 → 3.2.0

package/CHANGELOG.md

@@ -7,6 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [3.2.0] - 2026-08-01
+
+### Fixed
+
+- **`@serialize(true)` array / hash / property types without the annotation** — the generator assumed every referenced class had `toDictionary` / `fromDictionary` and emitted the call regardless, so the compile failed inside `extension <Class>` with `Could not match argument types for push` / `for case` and never named the real cause. Hash-valued fields were worse: the field was silently dropped from the serialized output. A `@serialize(true)` class that references a class which is neither `@serialize(true)` nor implements the pair by hand now fails at the declaration:
+
+  ```text
+  [FAIL] Parent.kids: [Child] can not be serialized - class Child is not @serialize(true).
+         Add @serialize(true) to Child, or implement toDictionary / fromDictionary in it.
+      9 │     def kids:[Child]
+                  ^── here
+  ```
+
+- **`@serialize(true)` with mutually referencing classes** — serializer extensions were generated one class at a time and `is_serialized` was set as each was reached, so whichever class came first in the dependency sort saw its peer as non-serializable and emitted primitive element code. All `@serialize(true)` classes are now marked before any extension is generated, which mutual references could not satisfy by sorting.
+
+- **`@serialize(true)` element types with hand-written `toDictionary` / `fromDictionary`** — accepted as serializable; the generator now routes them through the object path instead of treating them as primitives.
+
+- **Duplicate serializer extension when one file is reached by two import spellings** — a file imported as both `Child.rgr` and `domain/Child.rgr` was expanded twice, and with `@serialize(true)` the second expansion produced `method with the same name and parameter signature declared earlier` pointing at generated code. Fixed since 3.1.1 was published, released here; covered by a regression test.
+
+- **`SourceCode` constructed with three arguments in `ng_CodeNode.rgr`** — `SourceCode` takes a single `code_str`, but `vref1`, `vref2`, `newStr`, `newBool` and nine sibling factories passed `(name 0 (strlen name))` or `("" 0 0)`, copying the shape of the adjacent `new CodeNode(code 0 ...)`. JavaScript discards the extra arguments, so this was invisible until the TypeScript API bundle was type checked, where it produced all 13 of the `TS2554: Expected 1 arguments, but got 3` errors that `build:dist:module` reported. Dropping the dead arguments makes `tsc` clean, so the step is `&&`-chained again and a real type error blocks the build instead of scrolling past
+
+- **`file_mtime` C++ backend** — added missing `cpp` template in `Lang.rgr` (`stat()` + ms, matching es6 `mtimeMs`); fixes `game_runtime.rgr` hot-reload compile for `game_sdl` native binary
+
+- **`TSLexer` UTF-8 / native C++ tokenization** — code-unit length vs byte `charAt` mismatch after multi-byte characters (e.g. em dash in comments) no longer desyncs the lexer; fixes native SDL parse failures on [`invaders.game.tsx`](./gallery/game_engine/scripting/invaders.game.tsx) and similar scripts
+
+### Added
+
+- **LPC spritesheet compositor (standalone MVP)** — Ranger-native `gallery/game_engine/lpc/` (`png_decoder.rgr`, layer blit, male walk demo); embedded `pack/demo-male-walk` (~28 KB); `npm run engine:lpc:build` / `engine:lpc:run`; design doc [`LPC_HEADLESS_SPRITESHEET.md`](./gallery/game_engine/LPC_HEADLESS_SPRITESHEET.md)
+
+- **JPEG background loading for scripted games** — `GameImageLoader` + `GameRunner` asset queue (`resources()`, `backgroundImage()`, `onLoading()`); `splash_demo.game.tsx` smoke test; `game.d.ts` `ResourceDef` / `LoadingProps` types
+
+- **In-process TS hot reload (Path A)** — `TSAstPatcher` (`ts_ast_patch.rgr`), `ComponentEngine.patchScript()`, `GameRunner` runtime options (`setHotReload`, `trackScriptFile`, `maybeHotReload`); `game_sdl --hot-reload` / `--no-hot-reload`; default on for interactive SDL, off for `maxFrames` smoke runs
+
+- **SDL + native compiled games (Path B)** — `game_sdl_native_host.rgr`, `invaders_native_sdl_runner.rgr`, `build-game-sdl-native.sh`; `npm run engine:game-sdl-native:run:invaders` runs emitter output in an SDL window without `ComponentEngine`
+
+- **`@serialize(true)` regression suite** — `tests/compiler-serialize.test.ts` and `tests/fixtures/serialize/`: round trip over primitives, nested objects, object arrays and object hashes; mutual references; hand-written `toDictionary` / `fromDictionary`; duplicate import spellings; and the three diagnostics above
+
+### Changed
+
+- **`prepublishOnly` no longer runs the whole repository suite** — publishing the compiler ran all 56 test files, including the gallery, game-engine and native-toolchain suites. Those need SDL2, `g++`, Cannon and game fixtures that ship with neither the repo nor the package, so `npm publish` failed on the publisher's machine for reasons unrelated to the compiler (missing `SDL2/SDL.h`, `gallery/game_engine/games/ylos/index.tsx` and `physics_race/index.tsx` are absent from the repository entirely). `prepublishOnly` and `.github/workflows/publish.yml` now run `npm run test:publish` — 44 files, 355 tests, ~90s — covering parsing, type checking and code generation for every target backend. `npm test` still runs everything, and `ci.yml` is unchanged.
+
+- **`build:dist` now includes `build:dist:module`** — `dist/api.js` was not rebuilt by the release build, which is how it drifted behind `bin/output.js`
+
+- **`dist/api.js` rebuilt from current sources** — the committed programmatic-API bundle predated several compiler fixes, so `require("ranger-compiler")` shipped older behaviour than the `rgrc` CLI. `scripts/patch-chain-desugar.js` now patches `dist/api.js` as well as `bin/output.js`, and `build:dist:module` runs it after `tsc`
+
+## [3.1.1] - 2026-06-23
+
+### Changed
+
+- Version bump for npm publish — recommended for cloud CI and projects that install `ranger-compiler` from npm (e.g. koodisampo) instead of a sibling `../agent/Ranger` checkout
+- Default `npm test` / `prepublishOnly` skips `compiler-llvm.test.ts` (experimental LLVM/WAT backend); run `npm run test:llvm` when working on native/WASM codegen
+
+### Added
+
+- **IsoDate stdlib** — `lib/IsoDate/` (`DateMath`, `IsoDateParse`, `IsoCalendar`) and `lib/IsoDateLib.rgr` for portable ISO calendar dates without host `Date`; see [ai/ISO_DATE.md](ai/ISO_DATE.md)
+- **IsoDate compiler intrinsics** — `iso_add_days`, `iso_compare`, `iso_between` in `Lang.rgr` (Kotlin/Java `java.time.LocalDate`, ES6 UTC-safe helper)
+- **IsoDate regression** — `tests/fixtures/iso_date_ops.rgr` in Kotlin compiler tests
+- **Regex stdlib** — `lib/Regex/` (`RegexMatch`) and `lib/RegexLib.rgr` for string-pattern matching without `/literal/` syntax; see [ai/REGEX.md](ai/REGEX.md)
+- **Regex compiler intrinsic** — `regex_test(pattern, haystack)` in `Lang.rgr` (Kotlin/Java `Regex`/`Pattern`, ES6 `RegExp`, Swift `range(of:options:)`)
+- **Regex regression** — `tests/fixtures/regex_test_ops.rgr` in Kotlin compiler tests
+
+- **JavaScript/TypeScript source maps (`-sourcemap`)** — `SourceMapBuilder` with VLQ encoding (`compiler/ng_SourceMap.rgr`); `CodeWriter` line/column tracking, `walkNodeStack`, and `outMapped()`; embedded `sourcesContent` for `.rgr` sources; `.js.map` + `//# sourceMappingURL=` on save; statement/expression mappings via `LiveCompiler.WalkNode` walk context; expression `names` from `node.vref` / parameter names; regression `tests/compiler-sourcemap.test.ts`; README section *JavaScript / TypeScript source maps*
+
+### Fixed
+
+- **Source map VLQ line breaks** — `buildMappingsString` no longer resets source/original relative state on `;` (fixes DevTools breakpoints on `.rgr` sources)
+- **Source map original line** — `addMappingFromNode` uses `node.getLine()` from `sp` instead of stale `node.row`
+- **Kotlin `floor` / `int2double`** — `int2double` emits `.toDouble()`; reserved parameter names escaped (`val`, `object`, …)
+- **`proc_send` dispatch wrapping** — turn boundaries around handler calls; `ProcessRuntime.beginDispatchTurn` / `endDispatchTurn`
+
 ## [3.1.0] - 2026-06-02
 
 ### Added

package/README.md

@@ -1,6 +1,6 @@
 # Ranger cross language compiler
 
-**Version 3.1.0** | Status: `experimental`
+**Version 3.1.1** | Status: `experimental`
 
 Ranger is a self-hosting cross-language compiler for writing portable algorithms, parsers, generators, and small tools once and compiling them to multiple target languages.
 
@@ -56,7 +56,7 @@ const host = ProcessUiHost.__singleton();
 host.notifyPath = (path) => { /* sync view model + re-render */ };
 ```
 
-**Docs:** [PROCESS_MVP.md](PROCESS_MVP.md) (scope), [PROCESS_STATUS.md](PROCESS_STATUS.md) (compiler checklist), [PROCESS_UI_NOTIFY.md](PROCESS_UI_NOTIFY.md) (notify batching), [PROCESS_UI_VIEW_MODELS.md](PROCESS_UI_VIEW_MODELS.md) (view DTO assignment). **Gallery:** [process_counter_board](gallery/process_counter_board/README.md). **Pilot:** [realtrainer `app-ranger` Active Workout demo](https://github.com/terotests/realtrainer/tree/copilot/create-watch-ui-components/app-ranger/demo/active-workout-process).
+**Docs:** [PROCESS_MVP.md](PROCESS_MVP.md) (scope), [PROCESS_STATUS.md](PROCESS_STATUS.md) (compiler checklist), [PROCESS_RUNTIME_INVARIANTS.md](PROCESS_RUNTIME_INVARIANTS.md) (dispatch turn / one notify), [PROCESS_UI_NOTIFY.md](PROCESS_UI_NOTIFY.md) (notify batching), [PROCESS_UI_VIEW_MODELS.md](PROCESS_UI_VIEW_MODELS.md) (view DTO assignment). **Gallery:** [process_counter_board](gallery/process_counter_board/README.md) (Vite + React host for `@process`).
 
 ## Where To Start
 
@@ -69,6 +69,8 @@ host.notifyPath = (path) => { /* sync view model + re-render */ };
 - `gallery/js_parser` - substantial parser example with benchmarks and README
 - `gallery/pdf_writer` - EVG / TSX document tooling and preview server
 - `gallery/invaders` - cross-target demo game
+- `gallery/game_engine` - retained-mode game runner, SDL launcher, and TSX games (Pong, Breakout, Invaders, Pac-Man); see `gallery/game_engine/scripting/GAME_SCRIPTING.md`
+- `gallery/invaders/llvm/invaders.ll` - checked-in LLVM IR sample from the experimental `-l=llvm` backend
 
 ## Compatibility Snapshot
 
@@ -80,8 +82,26 @@ The project can target `JavaScript`, `Java`, `Go`, `Swift`, `PHP`, `C++`, `C#`,
 | Self-hosting | Actively used, but full compiler generation quality is strongest in JavaScript |
 | JavaScript / ES6 | Best baseline target and most reliable place to start |
 | Go / Swift / Rust / Kotlin / C++ | Useful and increasingly capable, but expect edge cases and target-specific gaps |
+| **LLVM / WASM** (`-l=llvm`) | **Experimental.** Lowers to LLVM IR; optional WAT export for freestanding WASM. Native libc builds work for demos like Space Invaders; browser WASM is still rough. See `npm run test:llvm`, `npm run game:build:llvm`. |
 | Gallery examples | Good for understanding direction and capability, but some require manual setup or platform-specific tooling |
 
+### Conformance suite (Track 1)
+
+Cross-target semantic fixtures live in `tests/conformance/`. Run `npx vitest run tests/compiler-conformance.test.ts`.
+Regenerate the fixture list with `node scripts/generate-conformance-table.mjs`.
+
+<!-- BEGIN CONFORMANCE_TABLE -->
+| Fixture | Topic | Targets |
+| --- | --- | --- |
+| `array_param_mutate` | array parameters use reference semantics (Issue #58; Go known gap) | ES6, Go, Kotlin (when toolchain present) |
+| `clear_then_push` | clear resets slice without nil, push refills (Issue #59) | ES6, Go, Kotlin (when toolchain present) |
+| `int_division_to_double` | Conformance: integer division promoted to double (Issue #4) | ES6, Go, Kotlin (when toolchain present) |
+| `lf_line_endings` | LF-only source (Issue #12 class must not break operator spacing) | ES6, Go, Kotlin (when toolchain present) |
+| `math_ops` | Conformance: arithmetic and comparisons | ES6, Go, Kotlin (when toolchain present) |
+| `string_codepoint_index` | Conformance: Unicode code-point string indexing (Issue #57) | ES6, Go, Kotlin (when toolchain present) |
+| `while_loop` | Conformance: while loop control flow | ES6, Go, Kotlin (when toolchain present) |
+<!-- END CONFORMANCE_TABLE -->
+
 ## What's New in Version 3.0
 
 - **New File Extension** - Transitioning from `.clj` to `.rgr` for Ranger identity
@@ -115,9 +135,9 @@ See [CHANGELOG.md](CHANGELOG.md) for full version history and [PLAN_3.md](PLAN_3
 The compiler is _self hosting_ which means that it has been written using the compiler itself and thus it can be hosted
 on several platforms. At the moment the official platform is node.js, because external plugins are only available as npm packages.
 
-The target languages supported are `JavaScript`, `Java`, `Go`, `Swift`, `PHP`, `C++`, `C#`, `Scala`, `Python`, and `Rust`. The quality
-of the target translation still varies and at the moment of this writing the compiler can only be compiled fully to JavaScript
-target. However, most targets already can compile reasonably good code.
+The target languages supported are `JavaScript`, `Java`, `Go`, `Swift`, `PHP`, `C++`, `C#`, `Scala`, `Python`, and `Rust`. An **experimental LLVM backend** (`-l=llvm`) emits LLVM IR and optional freestanding WAT for WASM toolchains; it is not a separate surface language but another codegen path from the same Ranger sources.
+
+The quality of the target translation still varies and at the moment of this writing the compiler can only be compiled fully to JavaScript target. However, most targets already can compile reasonably good code.
 
 ## Recent Updates (December 2025)
 
@@ -394,11 +414,12 @@ See `gallery/pdf_writer/examples/test_for_loop.tsx` for a complete demonstration
 
 ### Space Invaders Demo Game
 
-A complete terminal-based Space Invaders game demonstrating Ranger's cross-language capabilities. The same source code compiles to **4 different targets**:
+A complete terminal-based Space Invaders game demonstrating Ranger's cross-language capabilities. The same source code compiles to **several targets**:
 
-| Target         | Executable          | Build Command               |
+| Target         | Output              | Build Command               |
 | -------------- | ------------------- | --------------------------- |
 | ES6/JavaScript | `invaders.js`       | `npm run game:compile`      |
+| **LLVM native**| `tmp/invaders-native/invaders` | `npm run game:build:llvm` |
 | Rust           | `invaders_rust.exe` | `npm run game:build:rust`   |
 | Go             | `invaders_go.exe`   | `npm run game:build:go`     |
 | Kotlin         | `invaders.jar`      | `npm run game:build:kotlin` |
@@ -415,8 +436,24 @@ npm run game:build:all
 npm run game:run        # JavaScript
 npm run game:run:rust   # Rust
 npm run game:run:go     # Go
+./tmp/invaders-native/invaders   # LLVM native (after game:build:llvm)
+```
+
+#### Experimental LLVM backend (Space Invaders)
+
+Ranger can lower the same `invaders.rgr` through a **Low IR → LLVM IR** pipeline (`-l=llvm`), then link with `clang` and a small C runtime (`runtime/ranger_term.c`) for terminal I/O.
+
+```bash
+npm run compile              # refresh bin/output.js after compiler changes
+npm run game:build:llvm      # invaders.rgr → tmp/invaders-native/invaders.ll → native binary
+npm run test:llvm            # LLVM/WASM fixture tests (vitest)
+npm run demo:wasm            # smaller freestanding WASM demo (tests/fixtures/llvm_wasm_demo.rgr)
 ```
 
+**Checked-in sample:** [`gallery/invaders/llvm/invaders.ll`](gallery/invaders/llvm/invaders.ll) is LLVM IR kept in the repo so you can inspect codegen without building. Regenerate with `cp tmp/invaders-native/invaders.ll gallery/invaders/llvm/invaders.ll` after `game:build:llvm` (see [`gallery/invaders/llvm/README.md`](gallery/invaders/llvm/README.md)).
+
+Status: experimental — libc-linked native builds are the most reliable path. The freestanding WAT/WASM path (`-wasmrc`) now has a reference-counting runtime (free-list heap with `memory.grow`, typedesc-driven recursive object destruction), classes with reference-counted fields, singletons, strings, `[T]`/`[K:V]` collections, and lambdas/closures (function table + `call_indirect`, with value/object capture and mutation); the `gallery/game_engine/games/ranger_autopeli` guest is authored in Ranger and compiles to WASM this way. Terminal-import lowering for the full Space Invaders game is still incomplete. Smaller freestanding demos run under `npm run demo:wasm`.
+
 #### Cross-Compiling the Game
 
 The Space Invaders game demonstrates cross-platform compilation from a single source file.
@@ -470,6 +507,15 @@ npm run game:compile:swift  # Generates invaders.swift
 swiftc invaders.swift -o invaders_swift
 ```
 
+**LLVM native (macOS / Linux, experimental)**
+
+```bash
+npm run game:build:llvm
+./tmp/invaders-native/invaders
+```
+
+Requires `clang` on `PATH`. On macOS the script picks `arm64-apple-macos` or `x86_64-apple-macos` automatically.
+
 #### Platform-Specific Keyboard Input
 
 The game uses `on_keypress` and `poll_keypress` operators with platform-specific implementations:
@@ -492,6 +538,7 @@ The Space Invaders game provides an interesting comparison of how the same Range
 | Target     | Generated File   | Size (bytes) | Lines | Notes                           |
 | ---------- | ---------------- | ------------ | ----- | ------------------------------- |
 | **Ranger** | `invaders.rgr`   | 11,289       | ~400  | Original source                 |
+| **LLVM IR**| `llvm/invaders.ll` | ~60,000    | ~1,700 | Low-level IR (sample in repo)   |
 | Python     | `invaders.py`    | 9,271        | ~330  | Most compact generated code     |
 | JavaScript | `invaders.js`    | 10,301       | ~350  | Clean, readable output          |
 | Swift      | `invaders.swift` | 12,554       | ~470  | Verbose type annotations        |
@@ -499,11 +546,12 @@ The Space Invaders game provides an interesting comparison of how the same Range
 | C++        | `invaders.cpp`   | 14,148       | ~500  | Headers and type declarations   |
 | Rust       | `invaders.rs`    | 17,918       | ~600  | Most verbose (ownership, types) |
 
-**Executable Sizes (Windows):**
+**Executable Sizes (native binaries):**
 
 | Target | Executable           | Size   | Notes                             |
 | ------ | -------------------- | ------ | --------------------------------- |
-| Swift  | `invaders_swift.exe` | 76 KB  | Smallest native binary            |
+| **LLVM** | `tmp/invaders-native/invaders` | ~34 KB (arm64 macOS) | Smallest in recent local builds; libc + minimal runtime |
+| Swift  | `invaders_swift.exe` | 76 KB  | Dynamic link to system libraries  |
 | Rust   | `invaders_rust.exe`  | 291 KB | Optimized, statically linked      |
 | Go     | `invaders_go.exe`    | 2.3 MB | Includes Go runtime               |
 | C++    | `invaders_cpp.exe`   | 3.0 MB | Static linking with MinGW/pthread |
@@ -766,6 +814,7 @@ Flags: -<flag>
   -nodecli       Insert node.js command line header #!/usr/bin/env node to the beginning of the JavaScript file
   -nodemodule    Export classes as CommonJS modules using module.exports (disables static main function)
   -esm           Export classes as ES6/ESM modules using export keyword (disables static main function)
+  -sourcemap     Emit .js.map / .ts.map with embedded .rgr sourcesContent (ES6/TypeScript only)
   -client        the code is ment to be run in the client environment
   -scalafiddle   scalafiddle.io compatible output
   -compiler      recompile the compiler
@@ -800,6 +849,47 @@ node bin/output.js -es6 -esm myfile.rgr -o=myfile.mjs
 **File Extensions:**
 The compiler automatically detects JavaScript-related extensions (`.js`, `.ts`, `.mjs`, `.cjs`) and won't double-add them. You can safely specify the full filename with extension.
 
+### JavaScript / TypeScript source maps (`-sourcemap`)
+
+Use `-sourcemap` with `-es6` or `-typescript` to emit a sibling `.js.map` / `.ts.map` file and append `//# sourceMappingURL=…` to the generated output. Kotlin, Swift, and other non-JS targets ignore this flag.
+
+**What you get**
+
+| Output | Purpose |
+| --- | --- |
+| `output.js.map` | Source map v3 (VLQ `mappings`, `names`, `sources`) |
+| `sourcesContent` | Full `.rgr` source embedded in the map — Chrome DevTools can open `.rgr` files without a separate file server |
+| Statement + expression mappings | `LiveCompiler.WalkNode` walk context plus `outMapped()` on identifiers, calls, literals |
+
+**Compile example**
+
+```bash
+# Standalone ES module + map
+node bin/output.js -es6 -esm -nodemodule -sourcemap ./myapp/App.rgr -o=app.js
+
+# Result: bin/app.js and bin/app.js.map
+```
+
+**Debug in Chrome / Edge**
+
+1. Serve the generated `.js` (and `.map` beside it). Vite/webpack are optional when `sourcesContent` is embedded.
+2. Open DevTools → **Sources**. Original `.rgr` files appear under the map tree (from `sourcesContent`).
+3. Set breakpoints on **executable** lines (e.g. `def`, `if`, `return`) — not only blank lines or signatures.
+4. Breakpoints must be **solid red**. A hollow/grey breakpoint means no mapping for that line; rebuild with `-sourcemap` and hard-refresh (disable cache).
+
+**Tests**
+
+```bash
+npm run compile
+npx vitest run tests/compiler-sourcemap.test.ts
+```
+
+**Implementation notes** (for compiler hackers)
+
+- `compiler/ng_SourceMap.rgr` — `SourceMapBuilder`, VLQ encoder, `addMappingFromNode()` uses `node.getLine()` + `node.code.getColumn(sp)` (not stale `node.row`).
+- `compiler/ng_writer.rgr` — `lineNumber` / `columnNumber` on emit, `walkNodeStack`, `outMapped()`, `.map` write in `CodeFileSystem.saveTo`.
+- Flag: `compiler/ng_Compiler.rgr` → `flag sourcemap`; enabled in `VirtualCompiler.rgr` via `fileSystem.enableSourceMaps()`.
+
 ## Getting started with Hello World
 
 Create file `hello.rgr`
@@ -1257,6 +1347,8 @@ this.foo({
 })
 ```
 
+Lambdas compile on every backend, including the freestanding WASM/WAT path (`-wasmrc`): each body is hoisted to a function-table entry and calls go through `call_indirect`, with the value a reference-counted closure record. Captured variables are copied (values/strings) or retained (objects); a captured object can be mutated through the closure, and a captured value can be shared by boxing it in a heap cell.
+
 # Automatically infixed math support
 
 It is easy to define new mathematical operations in the Lang.clj file or in modules. However, some mathematical operations are automatically infixed