farm.rb 0.0.0 → 0.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 42bdfdd82e47daa951ea8607257d95bb223f070cc2decbb9aafe931c7921b8f2
-  data.tar.gz: 46a18d3d16da99e59273b2c15c2cf9109e8cc4c009a1fbecb32916e859c2e55c
+  metadata.gz: '094ced22f83044093d73911b2e7732b2fc459135baab73f07250746a6a9b348b'
+  data.tar.gz: da8a822c3a21ad9c8068fa52ea9fb005252ef3e3678872d181b43fa0d43a90cc
 SHA512:
-  metadata.gz: adaef059970b22e34b553e5d981433b1b13d4290e1270501c4cff5f6c451d60614afbf10a288ff345cc42b489fc97b426751a19d42c56bd8d591d98a40e38e06
-  data.tar.gz: e8a26416d0d8002838fcb4998f006adc26eaee2cefe0fc695d1e73371fcb5c8e0907bc146a0c49b106ac9e84696ef2ab0525d33c6d15a2f29d045945c716eaf9
+  metadata.gz: 13ff9638c92f96168afed6c43d5533f3479f45a2239557a7895cd65e9ca89894d220caa1f8b87995ed9c11f6ddcda9a5d618668b652303434d8551d14572a256
+  data.tar.gz: 7e6ab7b1dccd567f72b936e871ddba3c1545bf3a39004c654eb3b64d9a71f84cfd1adf15493be8df7632f14de09b57535b57e5df06d82c94e341557a4f7c58ab

data/CHANGELOG

@@ -1,5 +1,11 @@
 # CHANGELOG
 
+## 20261005
+
+0.0.1: send-time TypeError folded into Farm::NotParallelisable
+
+1. Farm::RactorExecutor: a job which cannot be copied to a worker at all — a Proc in its graph — failed at send time as TypeError, escaping the rescue Farm falls back upon though the failure was the machinery's rather than the work's. Workers are now collected as they spawn, and a TypeError at spawn or send winds the survivors down and raises NotParallelisable, the work running serially as promised and the caller none the wiser.
+
 ## 20261004
 
 0.0.0: + Farm, + Farm.each, map, Farm::Probe, Farm::RactorExecutor, Farm::SerialExecutor

data/ROADMAP.md

@@ -1,35 +1,27 @@
-# Farm Roadmap
+# farm/ROADMAP.md
 
-Date: 20261004
+Date: 20261005
 
-The figures quoted here were measured for the library and are recorded in MEASUREMENTS.md — with the machine, the method, and the run-to-run spread. They travel as priors to be corrected upon the machine in hand, not as constants.
-
-
-## Design decisions
-
-Recorded as 0.0.0 was cut. The version entries beneath may change; these hold.
-
-
-### The work travels as a name, or it does not travel
+## The work travels as a name, or it does not travel
 
 A Proc cannot cross a Ractor boundary, and nothing launders one: `define_method` from a Proc keeps the unshareable body it was defined from — isolation is decided by the Proc, not by the method built around it — and a Method object cannot cross whatever it was built from. A Proc built at the top level cannot even be isolated in place, `self` there being `main`. What runs is a module reached by name.
 
 Farm therefore asks for a callable and a method upon it, runs a block serially and says so, and leaves hosting to the consumer: render the body as source into a module of one's own and it crosses as well as anything typed by hand. The one route which should not be taken is regenerating a Proc's source into a shared carrier — a captured local silently rebinds to a same-named method already collected there, the answer changes, and nothing complains. Work should be born hosted: source first, closure never. (Measured, surveys and tables in MEASUREMENTS.md.)
 
 
-### Embeddable, in both directions
+## Embeddable, in both directions
 
 Farm is meant to sit underneath other libraries, and to mean nothing to their users. The whole contract is `map(enumerable, callable, method)`: work travels as a name. A consumer renders whatever it has — Namo's formulae, or anything else's named computation — into a module and a method upon it, and Farm cannot tell that module from one typed by hand. Hosting source-carried computation is the consumer's business: eval and its trust questions never enter Farm.
 
 Invisibility runs both ways. The host's users never meet Farm — the consumer soft-requires the gem, and without it everything runs serially and answers identically. Farm never meets the host's domain — no rows, no formulae, nothing above `map`'s three arguments. Farm.rb stays swappable for anything answering the same three-argument shape, and nothing upstream of the consumer needs to change.
 
 
-### Observational equivalence
+## Observational equivalence
 
 Every executor and every fall-back answers the same values in the same order, and raises the same exceptions serial would raise. Speed is the only admissible tell. This is the acceptance test for everything below and everything after: a parallel run which differs from the serial run in anything but time is a bug, wherever the difference surfaced.
 
 
-### Measured, never tabulated
+## Measured, never tabulated
 
 Width, substrate, and whether a job is worth spreading are read off the machine and the work in front of it, never configured by the caller and never assumed from a table. Four rules of practice, each bought with a measurement which first came out wrong:
 
@@ -39,7 +31,7 @@ Width, substrate, and whether a job is worth spreading are read off the machine
 - Every measured figure is a prior with a noise floor. Run-to-run spread for `fork` is around forty-five per cent, so a difference under about two-fold is no finding, and one run of anything will always name a winner. More than one conclusion here was first arrived at wrongly by an instrument returning a plausible figure for the wrong question; where a number underlies a decision it is re-measured rather than trusted.
 
 
-## 0.1.0: Embeddable
+## Embeddable
 
 The version which makes sitting underneath possible — the fall-back closed over transport, the substrate set widened beyond ractors, and the first measurement feeding the deciding.
 
@@ -47,7 +39,7 @@ The version which makes sitting underneath possible — the fall-back closed ove
 
 2. + Farm::ForkExecutor: a pool of forked processes, the one substrate which imposes no isolation rules at all — the child holds the whole process, so blocks run as written and code touching globals or class state just works. Its costs are measured and they decide when it is used rather than whether it exists: about a millisecond to start against a Ractor's 0.033; a cost which belongs to the calling process rather than the machine, rising with the resident set's high-water mark — twenty-fold at a gigabyte held — and never coming back down; results home through Marshal and a pipe at about 380MB/s, so a large enough answer argues against fork on its own. Its startup figure is estimated from `Probe.resident_memory` times a calibrated rate and corrected from the first real dispatch — never probed at startup, the probe costing more than it settles. Fork is the fall-back for computation, not the default substrate.
 
-3. + ThreadExecutor: the fall-back for waiting work. Threads do not scale upon computation — the GVL is held throughout, 1.00x at every width — and scale nearly perfectly upon waiting, where a worker occupies no core. So the measured policy stands: try a Ractor first whatever the work, and fall back by class — threads for waiting, fork for computation. Nothing recommends `spawn` for Ruby work; exec'ing the interpreter afresh costs forty milliseconds a unit.
+3. + Farm::ThreadExecutor: the fall-back for waiting work. Threads do not scale upon computation — the GVL is held throughout, 1.00x at every width — and scale nearly perfectly upon waiting, where a worker occupies no core. So the measured policy stands: try a Ractor first whatever the work, and fall back by class — threads for waiting, fork for computation. Nothing recommends `spawn` for Ruby work; exec'ing the interpreter afresh costs forty milliseconds a unit.
 
 4. The measurement loop goes in. Three inputs decide: the job's total serial work, the per-unit duration, and the size of what comes back. Whether to parallelise is decided upon the job, not the unit — a pool pays setup per worker rather than per unit, so `N·D > W·S/(1−1/k)` per candidate substrate and width: a Ractor pool of eight wants about a third of a millisecond of work, of forty-eight about 1.6ms, fork about 12ms in a small process. Which class the work is: a two-width sweep watching whether wall time falls as workers rise — a few milliseconds, and no understanding of the work required. Whether the work will cross: speculate one unit inside a Ractor, one unit plus 0.03ms, whereupon isolation refusal routes to the class's fall-back and anything else is the work's own error and re-raises. Below the threshold for every substrate the job runs serially; units too small to clear it are batched until the batch does.
 

data/lib/Farm/RactorExecutor.rb

@@ -7,9 +7,11 @@ module Farm
   # globals, shared mutable state — is discovered inside the worker, tagged,
   # and answered with NotParallelisable so the caller can run it serially;
   # the Ractor API does its own discovering at spawn and send time, which
-  # arrives here as Ractor::Error and is met the same way by Farm.  Workers
-  # rescue and tag rather than being allowed to die, a worker's death
-  # delivering nothing to the port and leaving its replacement to hang.
+  # arrives here as Ractor::Error and is met the same way by Farm — as is
+  # TypeError, what a job which cannot be copied at all arrives as, the
+  # survivors wound down first.  Workers rescue and tag rather than being
+  # allowed to die, a worker's death delivering nothing to the port and
+  # leaving its replacement to hang.
   class RactorExecutor
     class << self
       def map(enumerable, callable, method = :call)
@@ -45,21 +47,28 @@ module Farm
 
     def run
       port = Ractor::Port.new
-      workers = width.times.map do
-        Ractor.new(port, @callable, @method) do |results, callable, method|
-          loop do
-            job = Ractor.receive
-            break if job.nil?
-            index, item = job
-            begin
-              results << [:ok, index, callable.public_send(method, item)]
-            rescue => error
-              results << [:error, index, error.class.name, error.message]
+      workers = []
+      begin
+        width.times do
+          workers << Ractor.new(port, @callable, @method) do |results, callable, method|
+            loop do
+              job = Ractor.receive
+              break if job.nil?
+              index, item = job
+              begin
+                results << [:ok, index, callable.public_send(method, item)]
+              rescue => error
+                results << [:error, index, error.class.name, error.message]
+              end
             end
           end
         end
+        @jobs.each_with_index{|job, index| workers[index % width].send(job)}
+      rescue TypeError => error
+        workers.each{|worker| worker.send(nil)}
+        workers.each(&:join)
+        raise NotParallelisable, "TypeError feeding a Ractor worker: #{error.message}"
       end
-      @jobs.each_with_index{|job, index| workers[index % width].send(job)}
       workers.each{|worker| worker.send(nil)}
       collected = Array.new(@jobs.length){port.receive}
       workers.each(&:join)

data/lib/Farm/VERSION.rb

@@ -2,5 +2,5 @@
 # Farm::VERSION
 
 module Farm
-  VERSION = '0.0.0'
+  VERSION = '0.0.1'
 end

data/test/Farm/RactorExecutor_test.rb

@@ -26,6 +26,10 @@ module FarmRactorExecutorTestBoom
   end
 end
 
+# A job carrying a Proc cannot be copied to a worker at all: the copy fails
+# at send time as TypeError, before any work is done.
+FarmRactorExecutorTestProcPayload = Struct.new(:value, :fn)
+
 describe Farm::RactorExecutor do
   describe ".map" do
     it "maps with a callable, #call being the default method" do
@@ -51,6 +55,14 @@ describe Farm::RactorExecutor do
       }).must_raise(Farm::NotParallelisable)
       _(error.message).must_match(/Ractor::IsolationError/)
     end
+
+    it "answers NotParallelisable when a job cannot cross" do
+      payloads = [1, 2, 3].map{|value| FarmRactorExecutorTestProcPayload.new(value, proc{|item| item})}
+      error = _(proc{
+        Farm::RactorExecutor.map(payloads, FarmRactorExecutorTestDoubler)
+      }).must_raise(Farm::NotParallelisable)
+      _(error.message).must_match(/TypeError/)
+    end
   end
 
   describe ".each" do

data/test/farm_test.rb

@@ -22,6 +22,17 @@ module FarmAPITestGlobalToucher
   end
 end
 
+# A job carrying a Proc cannot be copied to a worker at all: the copy fails
+# at send time as TypeError, before any work is done, and Farm's answer to
+# the caller ought to be the serial answers, the machinery kept out of it.
+FarmAPITestProcPayload = Struct.new(:value, :fn)
+
+module FarmAPITestProcPayloadDoubler
+  def self.call(item)
+    item.fn.call(item.value)
+  end
+end
+
 describe Farm do
   describe ".map" do
     it "maps with a callable across the Ractor executor" do
@@ -44,6 +55,11 @@ describe Farm do
       _(Farm.map([1, 2, 3], FarmAPITestGlobalToucher)).must_equal([2, 4, 6])
       _($farm_api_test_counter).must_equal(3)
     end
+
+    it "falls back to the serial executor when a job cannot cross" do
+      payloads = [1, 2, 3].map{|value| FarmAPITestProcPayload.new(value, proc{|item| item * 2})}
+      _(Farm.map(payloads, FarmAPITestProcPayloadDoubler)).must_equal([2, 4, 6])
+    end
   end
 
   describe ".each" do
@@ -63,5 +79,10 @@ describe Farm do
       Farm.each([1, 2, 3], FarmAPITestGlobalToucher)
       _($farm_api_test_counter).must_equal(3)
     end
+
+    it "falls back to the serial executor when a job cannot cross" do
+      payloads = [1, 2, 3].map{|value| FarmAPITestProcPayload.new(value, proc{|item| item * 2})}
+      _(Farm.each(payloads, FarmAPITestProcPayloadDoubler)).must_be_same_as(payloads)
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: farm.rb
 version: !ruby/object:Gem::Version
-  version: 0.0.0
+  version: 0.0.1
 platform: ruby
 authors:
 - thoran