spinel_native 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 341a37c7333aabb9a5ca73e8bfc1dfeea3949c9629fdc809c81c34ee83369c43
-  data.tar.gz: d5a28eafd41ffe17e21c5fee4af88807f1e7103209a163a8e78a88e23d25a6f7
+  metadata.gz: 861579157d9270857eede60bc891c64f7b6b78c152a62512ef0ad098f112fad4
+  data.tar.gz: 132e7349789d035475fd5386ffa5bf6c60b39a3c08880e9adcd01244563d2e4e
 SHA512:
-  metadata.gz: 79161e43d39d1d61384f14675c2b732c21c2b4d34f7ca0e38760ce929c339b04e51b0818d306c83bb74698e03b7bd447c42f1e4f257381a6e549090c5ac2b95c
-  data.tar.gz: cf96d9b21b54cb6a1adc910caceeced6aa8f14c766b2a70624ec9b13b1775918ba20e1551f154480ae066ba5ebc6ead092bba4d0bdd3d2ebfcffc7492cad9306
+  metadata.gz: b195d6cf158a5bc71c802c7f181f9956a868c1989fd3901c6cd2ea93bfd04ddcb40eaf08fcd92e34e09f10253a4403ce2ef4275aa2d084332d6c75a7318ea1f2
+  data.tar.gz: 2ed20f171858b8f635bfda4dc66cbad803235430bd162a3cdbc32b3b8dc741acb4638f618c0b361b5eb4d3789b8a3dd094354df457ee6546fc14efe8810edc90

data/CHANGELOG.md

@@ -1,6 +1,27 @@
 # Changelog
 
-## 0.1.0
+## 0.3.0 (2026-09-18)
+
+- `native_entries :a, :b`: in a stateful kernel, name the methods called from
+  Ruby (exported across the extension boundary). Every other native method
+  stays internal to the kernel, so its parameters and return value need not be
+  boundary types and it needs no signature -- letting a real stateful renderer
+  keep poly-typed helpers behind a small typed surface. A non-entry method
+  called from Ruby runs its Ruby definition on the module's Ruby-side ivars,
+  not the kernel state; treat it as private to the kernel.
+
+## 0.2.0 (2026-09-16)
+
+- `native_state { ... }`: module state kept inside the compiled kernel
+  across calls. A stateful module is compiled as a whole when its body ends
+  and needs a signature on every native method; the block also initialises
+  the Ruby definitions' ivars.
+- `native_prelude`: constants, Structs and helper defs emitted into the
+  kernel ahead of the native methods.
+
+## 0.1.0 (2026-09-15)
+
+First release on RubyGems.
 
 - `native def` marks a method; its first call samples the argument types,
   compiles the kernel with `spinel --ext cruby`, and rebinds the method.

data/README.md

@@ -1,5 +1,8 @@
 # spinel_native
 
+[![Gem Version](https://badge.fury.io/rb/spinel_native.svg)](https://rubygems.org/gems/spinel_native)
+[![CI](https://github.com/khasinski/spinel_native/actions/workflows/ci.yml/badge.svg)](https://github.com/khasinski/spinel_native/actions/workflows/ci.yml)
+
 Compile a single Ruby method to native code with the
 [Spinel](https://github.com/matz/spinel) AOT compiler, from inside a running
 CRuby program. Mark the hot method, keep everything else on CRuby.
@@ -33,34 +36,259 @@ stays in place and a warning says why.
 
 ## Interface
 
-- `native def name(args)` marks a method. In a module it is also callable as
-  `Mod.name` (it can never use `self`, so the distinction is moot). `native
-  def self.name` works too, and so do instance methods of a class.
-- `native "(Array[Float], Integer) -> Float"` on the line before a `def`
-  declares the parameter types instead of sampling them. The return type is
-  inferred by Spinel and only checked for shape.
-- `Spinel::Native.compile!(Mod)` compiles every declared entry now, at boot,
-  so compile errors surface before the first request.
-- `SPINEL_NATIVE=off` runs the Ruby definitions only. `verify` runs both paths
-  on every call and raises `Spinel::Native::Mismatch` when they disagree (the
-  Ruby definition is the oracle). `strict` turns the silent fallback into a
-  raised `CompileError` / `TypeError`. Same knob: `Spinel::Native.mode = :verify`.
-- `SPINEL=/path/to/spinel` names the compiler (otherwise `spinel` on PATH),
-  `SPINEL_NATIVE_CACHE` the build cache (default `~/.cache/spinel-native`),
-  `SPINEL_NATIVE_VERBOSE=1` prints the commands and timings.
+### Mark a method
+
+`native def` is the whole DSL. Types are sampled from the first call.
+
+```ruby
+module Physics
+  extend Spinel::Native
+
+  native def dot(a, b)
+    s = 0.0
+    i = 0
+    while i < a.length
+      s += a[i] * b[i]
+      i += 1
+    end
+    s
+  end
+end
+
+Physics.dot([1.0, 2.0], [3.0, 4.0])   # => 11.0, compiled on this call
+```
+
+In a module the method is also callable as `Physics.dot`, since a native
+method never uses `self`. Singleton and instance forms work the same way:
+
+```ruby
+module Text
+  extend Spinel::Native
+
+  native def self.count_vowels(s)
+    n = 0
+    s.each_char { |c| n += 1 if "aeiou".include?(c) }
+    n
+  end
+end
+
+class Sieve
+  extend Spinel::Native
+
+  native def count_primes(n)
+    flags = Array.new(n + 1, true)
+    count = 0
+    i = 2
+    while i <= n
+      if flags[i]
+        count += 1
+        j = i * i
+        while j <= n
+          flags[j] = false
+          j += i
+        end
+      end
+      i += 1
+    end
+    count
+  end
+end
+
+Text.count_vowels("spinel")      # => 2
+Sieve.new.count_primes(100)      # => 25
+```
+
+Native methods may call each other; everything reachable must be `native`
+too, because the kernel is exactly the set of marked methods:
+
+```ruby
+module Fractal
+  extend Spinel::Native
+
+  native def mandel(cr, ci, limit)
+    zr = zi = 0.0
+    n = 0
+    while n < limit && zr * zr + zi * zi < 4.0
+      zr, zi = zr * zr - zi * zi + cr, 2.0 * zr * zi + ci
+      n += 1
+    end
+    n
+  end
+
+  native def row(ci, width, limit)
+    (0...width).map { |x| mandel(-2.0 + 3.0 * x / width, ci, limit) }
+  end
+end
+```
+
+### Declare the types
+
+A signature on the line before the `def` replaces sampling. The return
+type is inferred by Spinel and only checked for shape.
+
+```ruby
+module Stats
+  extend Spinel::Native
+
+  native "(Array[Float]) -> Float"
+  def mean(xs)
+    xs.sum / xs.length
+  end
+end
+```
+
+Declared entries can be compiled at boot, so a compile error surfaces
+before the first request rather than in the middle of one:
+
+```ruby
+Spinel::Native.compile!(Stats)
+```
+
+### Keep state between calls
+
+`native_state` gives the module state that lives inside the compiled kernel,
+so a large input can be copied in once and queried many times:
+
+```ruby
+module Index
+  extend Spinel::Native
+
+  native_state do
+    @docs = []
+  end
+
+  native "(Array[String]) -> Integer"
+  def load(docs)
+    @docs = docs
+    docs.length
+  end
+
+  native "(String) -> Array[Integer]"
+  def find(word)
+    hits = []
+    i = 0
+    while i < @docs.length
+      hits << i if @docs[i].include?(word)
+      i += 1
+    end
+    hits
+  end
+end
+
+Index.load(corpus)      # copied across once
+Index.find("spinel")    # no copy, just the answer
+```
+
+A stateful module is one kernel with one state, so it is compiled as a
+whole the moment its body ends; there is nothing to call first. Every native
+method in it therefore needs a declared signature, and a missing one is
+reported when the module closes. The block also runs on the module itself,
+so the Ruby definitions (used by `off`, `verify`, and the fallback) start
+from the same state. State is per process, not per object: a stateful
+module is a singleton, and a forked worker gets its own copy.
+
+### Extra kernel source
+
+Only `native` methods are pulled out of the source file. Constants, `Struct`
+definitions, and helpers the kernel needs but that are not entries go in a
+`native_prelude`:
+
+```ruby
+module Raster
+  extend Spinel::Native
+
+  native_prelude <<~RUBY
+    WIDTH = 320
+    Point = Struct.new(:x, :y)
+
+    def self.clamp(v, lo, hi)
+      v < lo ? lo : (v > hi ? hi : v)
+    end
+  RUBY
+
+  native def pixel(x, y)
+    clamp(y, 0, 239) * WIDTH + clamp(x, 0, WIDTH - 1)
+  end
+end
+```
+
+### Choose the entry methods
+
+A stateful module is compiled as one kernel, and by default every native method
+is an entry that Ruby can call -- so every one needs a boundary-crossable
+signature. `native_entries` names the few methods actually called from Ruby; the
+rest stay internal to the kernel, reachable only from other native methods.
+Internal methods need no signature and their parameters and return value need
+not be boundary types (they may be poly and get boxed) -- which is what lets a
+real, mutually-recursive kernel expose a small typed surface:
+
+```ruby
+module Renderer
+  extend Spinel::Native
+
+  native_state { @fb = Array.new(76800, 0) }
+  native_entries :render          # the only method CRuby calls
+
+  native "(Integer, Integer, Float) -> Array[Integer]"
+  def render(px, py, angle)
+    draw_walls(px, py, angle)     # internal; no signature required
+    @fb
+  end
+
+  native def draw_walls(px, py, angle) # stays inside the kernel
+    # ...
+    0
+  end
+end
+```
+
+Internal methods are still defined on the Ruby module, but calling one from
+Ruby runs its **Ruby** definition, against the module's Ruby-side ivars, not
+the kernel's state. `Renderer.draw_walls(...)` would draw into the Ruby
+`@fb`, which the compiled `render` never sees. Treat internal methods as
+private to the kernel; make a method an entry if Ruby needs its result.
+
+### Modes
+
+The Ruby definition is always kept. Which path runs is a process-wide
+switch, settable from code or from the environment:
+
+```ruby
+Spinel::Native.mode = :on       # default: compile on first call, fall back on failure
+Spinel::Native.mode = :off      # never compile, run the Ruby definitions
+Spinel::Native.mode = :verify   # run both, raise Spinel::Native::Mismatch if they differ
+Spinel::Native.mode = :strict   # a compile or type failure raises instead of falling back
+```
+
+```sh
+SPINEL_NATIVE=verify ruby app.rb     # the Ruby definition is the oracle
+SPINEL_NATIVE=off    ruby app.rb     # e.g. in a debugger
+```
+
+### Environment
+
+```sh
+SPINEL=/path/to/spinel/bin/spinel   # the compiler; otherwise `spinel` on PATH
+SPINEL_HDR_DIR=/path/to/lib         # runtime headers, if not next to the binary
+SPINEL_NATIVE_CACHE=~/.cache/spinel-native   # where compiled kernels live
+SPINEL_NATIVE_VERBOSE=1             # print the commands and timings
+```
 
 ## Rules for a native method
 
 - Parameters and the return value cross the boundary **by copy**. Supported
   types: `Integer` (64-bit), `Float`, `String`, `bool`, and `Array` of those.
   Mutating a parameter is refused at compile time; return the result instead.
-- The body must not touch `self`, instance variables, or anything outside the
-  module. Native methods may call each other; everything reachable must be
-  `native` too, because the kernel is exactly the set of marked methods.
+- The body must not touch `self` or anything outside the module's `native`
+  methods, its `native_prelude`, and its `native_state` ivars.
 - A `raise` inside the kernel arrives in Ruby as the same exception class and
   message. Integer overflow raises `RangeError` where CRuby would promote to a
   Bignum, and a Bignum argument is a `RangeError` at the boundary.
-- The kernel runs without the GVL, one call at a time per module.
+- The kernel runs without the GVL, one call at a time per module, so
+  `native_state` needs no locking of its own.
+- In a stateful module with `native_entries`, only the entries reach the
+  kernel's state. A non-entry method called from Ruby runs as Ruby, on the
+  module's own ivars.
 
 ## How it works
 
@@ -84,14 +312,31 @@ stays in place and a warning says why.
 
 ## Install
 
-Spinel is not on RubyGems; build it from source once and point the gem at it:
+The gem is on RubyGems:
+
+```sh
+gem install spinel_native
+```
+
+or in a Gemfile:
+
+```ruby
+gem "spinel_native"
+```
+
+The gem needs the Spinel compiler at run time, and Spinel is not on RubyGems.
+Build it from source once and point the gem at the binary, either through
+`SPINEL` or by putting `spinel` on `PATH`:
 
 ```sh
 git clone https://github.com/matz/spinel && cd spinel && make deps && make
 export SPINEL=$PWD/bin/spinel
-gem install spinel_native   # or: gem "spinel_native" in the Gemfile
 ```
 
+The runtime headers and sources are found next to the binary. If Spinel
+was installed elsewhere, `SPINEL_HDR_DIR` names the directory with
+`spinel_rt.h`.
+
 ## Running the example
 
 ```sh

data/lib/spinel/native/builder.rb

@@ -107,10 +107,10 @@ module Spinel
 
       # +body+ is the module body (the `def self.` methods), +entries+ maps the
       # exported names to their parameter types.
-      def initialize(body, entries)
-        @body = body
+      def initialize(body, entries, state: nil)
+        @body = state && !state.empty? ? "#{state}\n\n#{body}" : body
         @entries = entries
-        digest = Digest::SHA256.hexdigest([self.class.fingerprint, body, entries.inspect].join("\0"))[0, 12]
+        digest = Digest::SHA256.hexdigest([self.class.fingerprint, @body, entries.inspect].join("\0"))[0, 12]
         @module_name = "SpinelKernel#{digest}"
         @feature = "spinel_kernel_#{digest}"
         @dir = File.join(self.class.cache_dir, @feature)

data/lib/spinel/native/registry.rb

@@ -9,13 +9,57 @@ module Spinel
       Entry = Struct.new(:name, :kind, :pure, :source, :types, :compiled, :module_function, keyword_init: true)
 
       attr_accessor :pending_signature
-      attr_reader :entries
+      attr_reader :entries, :prelude
 
       def initialize(owner)
         @owner = owner
         @entries = {}
+        @prelude = []
         @last_def = nil
         @installing = false
+        @state = nil
+        @disabled = false
+        @entry_names = nil
+        @stateful_compiled = false
+      end
+
+      def stateful?
+        !@state.nil?
+      end
+
+      # Restrict which methods of a stateful kernel are exported across the
+      # extension boundary (called from Ruby). The rest stay internal to the
+      # kernel -- reachable only from other native methods -- so their parameter
+      # and return types need not be boundary-crossable (they may be poly and
+      # get boxed). Without this every native method is an entry.
+      def set_entries(names)
+        @entry_names = names.map(&:to_sym)
+      end
+
+      # `native_state { ... }`: run the block on the module so the Ruby
+      # definitions start from the same state, keep its source for the
+      # kernel, and arm a compile of the whole module for when its body ends.
+      def state(block)
+        raise Error, "#{@owner}: native_state declared twice" if @state
+        @state = Source.of_state_block(block)
+        @owner.instance_exec(&block)
+        owner = @owner
+        tp = TracePoint.new(:end) do |ev|
+          next unless ev.self.equal?(owner)
+          tp.disable
+          compile_stateful
+        end
+        @end_hook = tp
+        tp.enable
+      end
+
+      # Extra top-level source (constants, Structs, plain helper defs) emitted
+      # into the kernel module ahead of the marked methods. The kernel is
+      # otherwise exactly the set of `native` methods, so anything they
+      # reference by name -- a SCREEN_WIDTH constant, a state Struct, a helper
+      # that is not itself an exported entry -- has to be declared here.
+      def add_prelude(source)
+        @prelude << source.to_s
       end
 
       # Called from the method_added hooks.
@@ -45,6 +89,7 @@ module Spinel
       end
 
       def compile_known!
+        return compile_stateful ? @entries.values : [] if stateful?
         known = @entries.values.select { |e| e.types && !e.compiled }
         compile(known) unless known.empty?
         known
@@ -60,7 +105,13 @@ module Spinel
       # First call of a not-yet-compiled entry: settle its types, compile the
       # kernel, then either forward or (verify mode) keep comparing.
       def dispatch(entry, args, this)
-        return pure_call(entry, args, this) if Native.mode == :off
+        return pure_call(entry, args, this) if Native.mode == :off || @disabled
+        if stateful?
+          # A module without a closing `end` event (Module.new) compiles here instead.
+          compile_stateful unless entry.compiled
+          return pure_call(entry, args, this) unless entry.compiled
+          return native_call(entry, args, this)
+        end
         unless entry.compiled
           entry.types ||= args.map { |a| Types.of_value(a) }
           compile(@entries.values.select { |e| e.types && !e.compiled })
@@ -75,6 +126,36 @@ module Spinel
         pure_call(entry, args, this)
       end
 
+      # A stateful module is one kernel with one state, so it is compiled in
+      # full, once: every native method must carry a declared signature. Runs
+      # when the module body ends, or at the first call if that never fires.
+      # Returns true when the module is now native.
+      def compile_stateful
+        @end_hook&.disable
+        return false if @disabled || Native.mode == :off
+        return true if @stateful_compiled
+        exports = @entry_names ? @entries.values.select { |e| @entry_names.include?(e.name) } : @entries.values
+        raise Native::TypeError, "#{@owner}: native_entries names no known method" if exports.empty?
+        # Only exported (boundary-crossing) methods need signatures; internal
+        # helpers are inferred from the call graph, like a non-stateful kernel.
+        untyped = exports.reject(&:types).map(&:name)
+        unless untyped.empty?
+          raise Native::TypeError, "#{@owner} keeps state, so every native entry needs a signature; missing: #{untyped.join(', ')}"
+        end
+        compile(exports)
+        # Internal helpers are compiled into the kernel but not callable from
+        # Ruby; if one is invoked directly, fall back to its Ruby definition.
+        (@entries.values - exports).each { |e| install(e) { |a, t| pure_call(e, a, t) } }
+        @stateful_compiled = true
+        true
+      rescue Native::TypeError, CompileError => e
+        raise if Native.mode == :strict
+        warn "[spinel-native] #{@owner}: staying on Ruby (#{e.message.lines.first.strip})"
+        @disabled = true
+        @entries.each_value { |entry| install(entry) { |a, t| pure_call(entry, a, t) } }
+        false
+      end
+
       def native_call(entry, args, this)
         got = entry.compiled.public_send(entry.name, *args)
         if Native.mode == :verify
@@ -89,8 +170,8 @@ module Spinel
       # Every marked method goes into the kernel (they may call each other);
       # the ones with known types are exported.
       def compile(exports)
-        body = @entries.values.map(&:source).join("\n\n")
-        builder = Builder.new(body, exports.to_h { |e| [e.name, e.types] })
+        body = (@prelude + @entries.values.map(&:source)).join("\n\n")
+        builder = Builder.new(body, exports.to_h { |e| [e.name, e.types] }, state: @state)
         result = builder.build
         mod = Object.const_get(result.module_name)
         exports.each do |e|

data/lib/spinel/native/source.rb

@@ -15,6 +15,28 @@ module Spinel
         as_module_function(node)
       end
 
+      # The body of the `native_state { ... }` block at the block's source location.
+      def of_state_block(block)
+        file, line = block.source_location
+        raise Error, "native_state: no source location" unless file && File.exist?(file)
+        call = find_state_call(Prism.parse_file(file).value, line)
+        raise Error, "native_state: no `native_state { ... }` found at #{file}:#{line}" unless call
+        body = call.block.body
+        body ? body.slice : ""
+      end
+
+      def find_state_call(node, line)
+        if node.is_a?(Prism::CallNode) && node.name == :native_state &&
+           node.block.is_a?(Prism::BlockNode) && node.location.start_line == line
+          return node
+        end
+        node.compact_child_nodes.each do |child|
+          found = find_state_call(child, line)
+          return found if found
+        end
+        nil
+      end
+
       def find_def(node, name, line)
         return node if node.is_a?(Prism::DefNode) && node.name == name && node.location.start_line == line
         node.compact_child_nodes.each do |child|

data/lib/spinel/native/version.rb

@@ -2,6 +2,6 @@
 
 module Spinel
   module Native
-    VERSION = "0.1.0"
+    VERSION = "0.3.0"
   end
 end

data/lib/spinel/native.rb

@@ -23,6 +23,21 @@
 # re-bound to the compiled entry. If anything fails the pure-Ruby definition
 # stays in place, so the program is never worse off than without the gem.
 #
+# A module can also keep state inside the kernel between calls:
+#
+#   module Index
+#     extend Spinel::Native
+#     native_state { @docs = [] }
+#     native "(Array[String]) -> Integer"
+#     def load(docs) = (@docs = docs).length
+#     native "(Integer) -> String"
+#     def doc(i) = @docs[i]
+#   end
+#
+# Such a module is compiled as a whole when its body ends (all its native
+# methods must declare signatures), and the Ruby definitions run against the
+# module's own ivars, initialised by the same block.
+#
 # SPINEL_NATIVE=off     never compile, run the Ruby definitions
 # SPINEL_NATIVE=verify  run both and raise Spinel::Native::Mismatch on divergence
 # SPINEL_NATIVE=strict  a compile failure raises instead of falling back
@@ -92,6 +107,17 @@ module Spinel
       end
     end
 
+    # native_state { @items = []; @sum = 0 }
+    #
+    # Module-level state that lives inside the compiled kernel across calls
+    # (and, for the Ruby fallback, in the module's own ivars). A stateful
+    # module is compiled as a whole when its body ends, so every native
+    # method in it needs a declared signature.
+    def native_state(&block)
+      raise ArgumentError, "native_state needs a block" unless block
+      @__spinel_native.state(block)
+    end
+
     # native def foo(a, b) ... end          types sampled from the first call
     # native "(Array[Float], Integer) -> Float"; def foo(a, b) ... end
     #                                         types declared, compiled on first call
@@ -105,5 +131,31 @@ module Spinel
       end
       arg
     end
+
+    # Emit extra top-level source into the compiled kernel, ahead of the marked
+    # methods: constants, `Struct.new` state aggregates, and plain helper defs
+    # the native methods call. Only `native` methods are pulled from their
+    # source files; everything else a kernel needs must be declared here.
+    #
+    #   native_prelude <<~RUBY
+    #     WIDTH = 320
+    #     St = Struct.new(:buf, :n)
+    #   RUBY
+    def native_prelude(source)
+      @__spinel_native.add_prelude(source)
+      source
+    end
+
+    # native_entries :render, :load_map
+    #
+    # In a stateful kernel, name the methods that are called from Ruby (exported
+    # across the extension boundary). Every other native method stays internal
+    # to the kernel -- reachable only from native code -- so its parameter and
+    # return types need not be boundary types. Without this, every native method
+    # is an entry and must have boundary-crossable signatures.
+    def native_entries(*names)
+      @__spinel_native.set_entries(names)
+      names
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: spinel_native
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Chris Hasiński