bundler-spinel 0.0.1.pre

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: fedb059cdc7900cd98ccfff408a28e1fb4e4eaad68d9e7afe48c9d967cb97b71
+  data.tar.gz: a1d6fbfb31256d58cacab1cc120290455bd38de869295504a1c6dcdee96ea8da
+SHA512:
+  metadata.gz: acd75d23500e6af2d3e55c116bc0f5d82e8a4c8b1912049a349967e9cc920134ffe4820862abeeb6a753445f0a668cf26a9248c6aed4b983ea0dff7c262c13ee
+  data.tar.gz: 1bd2fa39872c215534c919f6a8c05cf01cf6dc30e5cce162d470d8d0ec3d750a09e06bc3a404d05f1ac4195a0c86232ca6e93f9d761e5d99771603361fd58756

data/ARCHITECTURE.md

@@ -0,0 +1,164 @@
+# bundler-spinel — architecture
+
+Make Spinel gem-incompatibility fail **at `bundle lock` time**, not at compile
+time (where Spinel may silently emit a no-op instead of an error) — and make
+that verdict **forward-compatible**, so a gem rejected today flips to accepted
+the moment a future Spinel learns the feature it needed.
+
+## The problem it solves
+
+Spinel is a whole-program Ruby→C AOT compiler with no gems, no eval, no
+metaprogramming. Two facts make naive dependency management dangerous:
+
+1. **Bundler can't gate compatibility.** There is no `required_ruby_engine`
+   gemspec field; `required_ruby_version` is engine-blind; you can't fabricate a
+   platform variant for a gem you don't publish. The Gemfile `engine: "spinel"`
+   directive is a *post-resolution* check — `bundle lock` ignores it (exit 0);
+   only `bundle install` fires the guard (exit 18). So resolution can't reject
+   an incompatible gem.
+
+2. **Spinel doesn't fail loudly.** On unsupported Ruby it prints
+   `warning: ... cannot resolve call to 'eval' ... (emitting 0)` and degrades
+   the call to a no-op, exiting 0. Worse, some constructs (`define_method`,
+   silent miscompiles like local-var-name collapse / Int-0-as-nil) produce no
+   warning at all. So "it compiled" ≠ "it works."
+
+This tool moves the decision to resolution time and grounds it in actually
+running Spinel over the gem source.
+
+## The ledger — single source of truth
+
+`ledger/compat.jsonl`, append-only, one line per **`(gem, version, engine_rev)`**:
+
+```json
+{"gem":"rake","version":"13.4.2","rev":"git:0adca86+dirty",
+ "verdict":"rejected","reasons":["analyze-failed"],
+ "risks":["eval","method_missing","needs:rbconfig"],"probe":"compile+scan","at":"…"}
+```
+
+**`engine_rev` is the forward-compat key.** Spinel ships no version string
+(`spinel --version` prints usage; `git describe` is a bare SHA), so we key on the
+git revision of the Spinel checkout, falling back to a binary content-hash.
+Upgrade Spinel → new rev → cache miss → automatic re-probe. No hand-maintained
+blocklist; a `rejected` verdict is always "rejected *as of this rev, because of
+these named features*," never "rejected forever." `spinel-compat reprobe` sweeps
+known gems under the current rev to surface what newly passes.
+
+## The verdict ladder
+
+Spinel's failure modes aren't exit codes, so one signal isn't enough:
+
+| Verdict | Earned by | Trusted by |
+|---|---|---|
+| `rejected` | unsupported **call** (`cannot resolve call to 'X'` → `unresolved:X`), or `analyze failed` / non-zero exit | gate fails the lock |
+| `risky` | compiles clean, but static scan found constructs Spinel degrades silently (`define_method`, `eval`, `send`, `method_missing`, C-ext…) | gate allows; `--strict` fails |
+| `clean` | compiles clean, no risky constructs | gate allows |
+| `verified` | `clean` **and** the gem's own tests pass through a Spinel-compiled harness | curated whitelist + platform badge |
+
+Two probe signals feed this:
+
+- **Compile signal** (`spinel -c` over the gem's lib entrypoints) — parses stderr.
+  `cannot resolve call to 'X'` is the gold signal: precise and forward-compatible.
+- **Static risk scan** — catches constructs that are silently degraded (no
+  warning) or hidden behind dead-code elimination.
+
+### Honest limitations (why `verified` must exist)
+
+- **No load path.** Spinel resolves plain `require "x"` only against
+  `<spinel>/lib`. A gem's internal `require "gem/part"` and its stdlib requires
+  don't resolve, so the compile probe doesn't fully follow multi-file gems
+  (recorded as `needs:X` notes, not rejections). The compile signal is a **lower
+  bound** on problems. `require_relative`-based gems probe well; plain-`require`
+  gems under-probe.
+- **Silent miscompiles** (var-name collapse, Int-0-as-nil) emit no warning and
+  exit 0. The static scan can't see them either.
+
+So `clean` means "no problem *found cheaply*," not "correct." Only `verified` —
+compiling the gem's test suite and running it — is trustworthy, which is exactly
+why the curated whitelist and the platform badge require it.
+
+## Placement — "make it work" (the primary job)
+
+Independent of the ledger: `spinel-compat vendor` reads a `Gemfile.lock`, copies
+each gem's `lib/` into `vendor/spinel/<name>/`, and writes `vendor/spinel/deps.rb`
+(a `require_relative` manifest in lock order). A Spinel program does
+`require_relative "vendor/spinel/deps"`. This is the reusable form of the
+hand-vendoring projects do today, and the reason the convention is usable at all
+given Spinel has no load path. Gating is layered on top but advisory — placement
+and compatibility are different jobs.
+
+## Gating consumers, all views over the ledger
+
+### 1. Lock-time gate — `bundle spinel-lock` (BUILT)
+`bundle lock` (resolves, ignoring the engine directive), then `check` resolves a
+verdict for every locked gem (ledger hit, or probe-on-miss) and exits non-zero on
+any `rejected`. The headline: resolution-time failure with feature-named reasons.
+Also the **backstop** for consumer #2's leak (below): it re-checks every gem in
+the *resulting* lock against the ledger regardless of where it resolved from.
+
+### 1b. The `verified` rung — `spinel-compat verify` (BUILT)
+Differential testing: run a smoke program once under CRuby and once Spinel-
+compiled, diff stdout. The only signal that catches Spinel's **silent
+miscompiles** — they emit no warning and exit 0, so the cheap probe calls them
+`clean`, but a differential run diverges. Proven: a method doing `h[k].nil? ? -1
+: v` over a stored `0` is `clean` to the probe but caught as `rejected:miscompile`
+by verify (`L2 cruby="-1" spinel="0"` — the Int-0-as-nil footgun). The smoke is
+the unit of trust, so `verified` is opt-in and smoke-supplied.
+
+### 2. Curated source / proxy — `spinel-compat serve` (BUILT, MVP)
+A Compact Index source serving only vetted gems from a local store of .gem
+artifacts (`--min verified|clean|risky`). Proven end-to-end: `bundle lock` against
+it resolves a vetted gem (exit 0) and fails on an absent one (exit 7, "could not
+find gem … in repository … or installed locally"). The "whitelist" is not a file:
+it's the acceptable-verdict subset of the ledger at the pinned rev. `path:`/`git:`
+siblings (`gem "tep", path: …`) are the degenerate one-gem curated source and
+probe in place via `probe --dir`.
+
+**Leak caveat (important):** Bundler *also* considers locally-installed gems
+("…or installed locally"). In a polluted environment an unvetted-but-installed
+gem can resolve and even be mis-attributed to the source. So the curated source
+gates a **clean** environment (CI); pair it with consumer #1 as a backstop for
+dev machines. Read-through filtering of upstream rubygems is a documented
+extension; empirically the third-party ecosystem is ~all-rejected today, so the
+local-store mode is what carries weight.
+
+### 3. Platform-variant opt-in (designed — `platform.rb` stub)
+The bridge from "our ledger says verified" to "stock Bundler selects it,"
+reusing the same machinery JRuby uses with the `java` platform. A `verified` gem
+is republished (to the curated source) under a `spinel` platform token — likely
+`<cpu>-spinel-<engine_rev>` so the badge is rev-scoped — and Bundler's normal
+platform resolution prefers it. Opt-in because earning the badge means someone
+ran the gem's tests through a Spinel-compiled harness.
+
+## Dogfooding — the curated source served by Tep
+
+The MVP proxy is CRuby/WEBrick (fast to prove Bundler resolves against it). The
+target is to serve the same Compact Index endpoints from **Tep** — the Sinatra-
+flavoured framework that *itself compiles via Spinel*. Then the Spinel
+dependency-manager's source is a Spinel program: the system vets its own
+substrate. Tep already has the pieces (`sphttp` server, routing, an HTTP client
+for upstream fetch); the open questions are Spinel support for the bits the proxy
+needs — MD5/SHA256 (digest), and reading the JSONL ledger — which are themselves
+good probe targets. Serving `/names`, `/versions`, `/info/<gem>`, `/gems/<file>`
+is plain text + file bytes, well within Tep's range.
+
+## Layout
+
+```
+lib/bundler/spinel/
+  engine.rb        # locate compiler, derive forward-compat engine rev
+  ledger.rb        # append-only JSONL verdict store, keyed on (gem,version,rev)
+  gem_fetcher.rb   # gem fetch + unpack (source, not install), cached
+  probe.rb         # compile signal + static risk scan -> verdict
+  verifier.rb      # differential CRuby-vs-Spinel smoke -> verified
+  vendorer.rb      # place lockfile deps where Spinel finds them + deps.rb
+  survey.rb        # parallel wholesale review -> reason histogram
+  checker.rb       # Gemfile.lock -> per-gem verdict -> pass/fail gate
+  cli.rb           # `spinel-compat` dispatcher
+  command.rb       # Bundler plugin command (bundle spinel-lock / -check)
+  proxy.rb         # STUB: curated source
+  platform.rb      # STUB: platform-variant opt-in
+exe/spinel-compat  # standalone CLI
+plugins.rb         # Bundler plugin entry
+ledger/compat.jsonl
+```

data/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to `bundler-spinel` are documented here. The format
+follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
+project aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.0.1.pre] — 2026-05-26
+
+First pre-release. **Experimental** — the CLI surface, the verdict vocabulary,
+and the ledger format may all change before `0.0.1`. Install with `--pre`.
+
+### Added
+- Initial public release: the Gemfile convention for Spinel projects, the
+  `spinel-compat` CLI (`vendor`, `check`, `probe`, `verify`, `survey`,
+  `serve`, `ledger`, `reprobe`), and the `spinel-lock` / `spinel-check`
+  Bundler plugin commands.
+- Forward-compatible, engine-rev-keyed compatibility ledger.
+- Wholesale `survey` with a thread-safe ledger, a per-compile wall-clock
+  timeout (`analyze-timeout` reject reason), and a ledger-based report.
+
+[0.0.1.pre]: https://github.com/OriPekelman/spinelgems/releases/tag/v0.0.1.pre

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ori Pekelman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,84 @@
+# bundler-spinel
+
+> ⚠️ **Pre-release / experimental** (`0.0.1.pre`). The CLI surface, the verdict
+> vocabulary, and the ledger format may all change before `0.0.1`. Install with
+> `--pre`. (`spinelgems.org` is not live yet.)
+
+**Use a standard `Gemfile` for your [Spinel](https://github.com/matz/spinel)
+project — for now.** Spinel-compiled projects have no shared way to declare or
+exchange dependencies, so each vendors by hand. Rather than design a package
+manager, borrow a format everyone already knows and revisit later. See
+[RFC.md](RFC.md) for the proposal.
+
+`bundler-spinel` is a small Bundler plugin that makes that practical in two ways:
+
+1. **Makes it work** — places resolved dependencies where Spinel can actually
+   find them. Spinel has no load path and inlines `require_relative`, so a dep
+   has to be *placed* and wired. `spinel-compat vendor` does that from a lockfile.
+2. **Gates** — Spinel silently emits a no-op for unsupported Ruby (exit 0), so
+   "it compiled" ≠ "it works". The plugin probes gems and flags incompatible ones
+   at `bundle lock` time, with reasons that name the missing feature — nicer than
+   a silent miscompile. Verdicts are forward-compatible (keyed on the Spinel rev).
+
+## The Gemfile convention
+
+```ruby
+source "https://rubygems.org"
+ruby "3.3.0", engine: "spinel", engine_version: "0.0.0"
+
+gem "tep", git: "https://…/tep.git"   # siblings via path:/git: (replaces rsync)
+gem "some_pure_ruby_lib"
+```
+
+`bundle lock` resolves normally (it ignores the engine marker); the marker guards
+`bundle install` (exit 18 under CRuby). Nothing here is novel — that's the point.
+
+## Quick start
+
+```sh
+# 1. make it work — place deps where Spinel finds them
+bundle lock
+exe/spinel-compat vendor                 # -> vendor/spinel/<gem>/lib + vendor/spinel/deps.rb
+#    then `require_relative "vendor/spinel/deps"` from your Spinel entrypoint
+
+# 2. gate — flag what Spinel can't compile, early
+exe/spinel-compat check Gemfile.lock     # exit 1 if any gem is rejected
+```
+
+As a Bundler plugin:
+
+```sh
+bundle plugin install bundler-spinel --git https://github.com/OriPekelman/spinelgems.git   # or --path .
+bundle spinel-lock      # bundle lock, then report incompatible gems
+bundle spinel-check     # gate an existing Gemfile.lock
+```
+
+## The rest of the toolbelt
+
+```sh
+exe/spinel-compat engine                 # detected compiler + engine rev
+exe/spinel-compat probe rake [--dir P]   # probe one gem (or a local/sibling dir)
+exe/spinel-compat verify NAME --smoke F  # differential CRuby-vs-Spinel run -> verified
+exe/spinel-compat survey --list F        # wholesale review -> reason histogram
+exe/spinel-compat serve --store DIR      # curated source (only vetted gems)
+exe/spinel-compat ledger / reprobe       # inspect / re-probe under current rev
+```
+
+Verdicts: `✓ clean` · `★ verified` · `~ risky` · `✗ rejected`.
+
+## Environment
+
+- `SPINEL_DIR` — path to the Spinel checkout (default `~/spinel`; falls back to a `spinel` on `PATH`).
+- `SPINEL_COMPAT_LEDGER` — ledger path (default `ledger/compat.jsonl`).
+
+## Status
+
+Working: the Gemfile convention, `vendor` (placement), the lock-time gate +
+Bundler plugin, the probe + forward-compat ledger, the `verified` differential
+harness, the curated source (`serve`), and the wholesale `survey`.
+
+The probe is a **lower bound** — Spinel's lack of a load path means multi-file
+plain-`require` gems under-probe, and silent miscompiles are invisible to it.
+Trust `verified` (smoke runs identically under CRuby and Spinel), not `clean`,
+where it matters. Empirically most third-party gems reject today, so the weight
+is on your own vetted gems and `path:`/`git:` siblings — not a rubygems mirror.

data/RFC.md

@@ -0,0 +1,120 @@
+# RFC: Use a Gemfile for Spinel projects (for now)
+
+**Status:** draft (for discussion)
+**Author:** Ori Pekelman
+**Prototype:** [`bundler-spinel`](./README.md) — working, see [ARCHITECTURE.md](./ARCHITECTURE.md)
+
+## Summary
+
+Spinel-compiled projects have no shared way to declare or exchange dependencies,
+so each one vendors by hand. This RFC proposes a deliberately small, temporary
+answer that requires **no new design**:
+
+1. **Declare dependencies in a standard `Gemfile`.** Mark the project with
+   `ruby "3.x", engine: "spinel", engine_version: "0.0.0"`. That's the whole
+   convention. It buys a familiar manifest, `path:`/`git:` sources for sibling
+   projects, and Bundler's resolver + lockfile — for free.
+
+2. **A small Bundler plugin** ([`bundler-spinel`](./README.md)) that does two
+   things: **(a) makes it work** — places resolved dependencies where Spinel can
+   actually find them; and **(b) gates** — flags gems Spinel can't compile early,
+   so the experience is nicer than a silent miscompile.
+
+The point is to **postpone the big discussion**. Spinel isn't near a release, and
+there's no expectation it adopts any particular dependency-management design. But
+projects need *some* interop today — so let's borrow a format everyone knows and
+revisit later, rather than design a package manager now.
+
+## Motivation
+
+Interop is already happening, ad-hoc. Roundhouse (a separate author's project)
+vendors part of Tep; our own projects carry a mix of rsync'd copies and hand-
+maintained concatenation. Every project reinvents "get this other code into a
+shape Spinel will compile." A shared, boring convention removes that duplication
+without anyone committing to a long-term model.
+
+Two facts make a thin tool worthwhile (both verified — Bundler 2.7.2 / CRuby):
+
+- **Bundler already does the right thing with the engine marker.** `bundle lock`
+  ignores `engine: "spinel"` and resolves normally (exit 0); only `bundle
+  install` fires the guard ("Your Ruby engine is ruby, but your Gemfile specified
+  spinel", exit 18). So the convention costs nothing and fails loudly in the
+  right place.
+
+- **Spinel doesn't fail loudly on unsupported Ruby.** It prints
+  `warning: … cannot resolve call to 'eval' … (emitting 0)` and degrades the call
+  to a no-op, exiting **0**; some constructs (and silent miscompiles) warn not at
+  all. So "it compiled" ≠ "it works" — which is why a little gating help is worth
+  having.
+
+## Proposal 1 — the Gemfile convention
+
+A Spinel project's `Gemfile`:
+
+```ruby
+source "https://rubygems.org"
+ruby "3.3.0", engine: "spinel", engine_version: "0.0.0"
+
+gem "tep", git: "https://…/tep.git"   # sibling projects via path:/git:
+gem "some_pure_ruby_lib"              # third-party, if it compiles
+```
+
+Nothing here is novel — that's the point. `bundle lock` produces a normal
+lockfile; siblings resolve through `path:`/`git:` sources (replacing rsync /
+manual vendoring); third-party gems resolve from rubygems.org. The engine marker
+documents the target and guards `bundle install`.
+
+## Proposal 2 — the Bundler plugin
+
+### (a) Make it work — placement
+
+Spinel has no load path (plain `require "x"` resolves only against
+`<spinel>/lib`) and inlines `require_relative`. So a resolved dependency has to
+be *placed* where Spinel will follow it. The plugin vendors each locked gem's
+`lib/` into a project dir and generates a `require_relative` manifest:
+
+```
+spinel-compat vendor            # reads Gemfile.lock
+# -> vendor/spinel/<gem>/lib/…  and  vendor/spinel/deps.rb
+```
+
+A Spinel program then just `require_relative "vendor/spinel/deps"`. This is the
+reusable form of what projects already do by hand (concatenation scripts, partial
+vendoring).
+
+### (b) Gating — a nicer experience
+
+Because Spinel silently no-ops unsupported Ruby, the plugin probes gems (compile
++ static scan, optionally a differential CRuby-vs-Spinel run) and flags ones that
+won't work — at `bundle lock` time, with reasons that name the missing feature:
+
+```
+bundle spinel-lock     # bundle lock, then report incompatible gems
+```
+
+Verdicts are **forward-compatible**: each is keyed on the Spinel revision (Spinel
+ships no version, so we key on the git rev / binary hash, scoped by platform).
+Upgrade Spinel → re-probe → a gem rejected today clears the moment the feature it
+needed lands. No hand-maintained blocklist.
+
+Gating is advisory by design — placement and compatibility are different jobs.
+The plugin makes the failure visible and early; it doesn't try to be a wall.
+
+## Open questions (for discussion, not asks)
+
+These are things we ran into, offered as discussion points — not feature
+requests, and not assuming Spinel wants any of them:
+
+- **Signalling unsupported constructs.** Today they're silent no-ops at exit 0.
+  Would a non-zero exit or a structured (e.g. JSON) diagnostic — perhaps behind a
+  `--check`/`--strict` flag — be in scope or interest? It's the difference
+  between inferring compatibility from stderr and reading it from a contract.
+- **A stable engine identity.** We key verdicts on a git rev / binary hash for
+  now. If Spinel ever wanted to expose a version/rev, the ledger and the
+  `engine_version:` marker could line up — but there's no rush.
+- **Resolving `require` beyond `<spinel>/lib`.** A load-path notion would let
+  multi-file gems and stdlib shims resolve (and would make probing more
+  accurate). Possibly useful well beyond this tool; possibly out of scope.
+
+None of these are needed for the convention or the plugin to be useful today.
+They're just where the seams are, if there's ever appetite to smooth them.

data/exe/spinel-compat

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+require_relative "../lib/bundler/spinel/cli"
+
+exit Bundler::Spinel::CLI.new.run(ARGV)

data/lib/bundler/spinel/checker.rb

@@ -0,0 +1,58 @@
+require "bundler"
+require "bundler/lockfile_parser"
+
+module Bundler
+  module Spinel
+    # The resolution-time gate. Reads a Gemfile.lock, resolves a verdict for
+    # every locked gem (from the ledger, or by probing on a cache miss), and
+    # decides pass/fail. This is what turns Spinel's compile-time-or-never
+    # failure into a `bundle lock`-time failure.
+    class Checker
+      Result = Struct.new(:verdict, :rejected, :risky, :probed, keyword_init: true)
+
+      def initialize(engine: Engine.new, ledger: Ledger.new)
+        @engine = engine
+        @ledger = ledger
+        @fetcher = GemFetcher.new
+        @probe = Probe.new(@engine, @ledger)
+      end
+
+      # strict: treat `risky` as a failure too.
+      def check(lockfile = "Gemfile.lock", strict: false)
+        @engine.ensure!
+        parsed = Bundler::LockfileParser.new(File.read(lockfile))
+        rejected = []
+        risky = []
+        verdicts = []
+
+        parsed.specs.each do |spec|
+          # path:/git: sources (e.g. a sibling like tep) probe in place; the
+          # GEM source fetches. We only have name+version here, so prototype
+          # handles the rubygems case; path/git probing is a documented TODO.
+          v = verdict_for(spec.name, spec.version.to_s)
+          next unless v # skipped (unfetchable / TODO source)
+
+          verdicts << v
+          rejected << v if v.rejected?
+          risky << v if v.risky?
+        end
+
+        ok = rejected.empty? && (!strict || risky.empty?)
+        Result.new(verdict: ok, rejected: rejected, risky: risky, probed: verdicts)
+      end
+
+      private
+
+      def verdict_for(name, version)
+        cached = @ledger.lookup(name, version, @engine.rev)
+        return cached if cached
+
+        dir = @fetcher.fetch(name, version)
+        @probe.probe(name, version, dir)
+      rescue Error => e
+        warn "[spinel-compat] skipped #{name} #{version}: #{e.message}"
+        nil
+      end
+    end
+  end
+end