vicary 0.2.4 → 0.2.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 585c77607d6c331c3cebe14a5da44463a0231f7c3a15e5c1ba68f79d8b9f588b
-  data.tar.gz: 883c07a66721ce9c67fb0cfe1f1ea16ce486decd6be62164b1ebbbf53fec9bf0
+  metadata.gz: 86411ff6192dfe36ba02d8159d4000bc55814d5eb8dd782713f373199d0bef98
+  data.tar.gz: 85ad752a1663010614adec4820efb015e0ee9b63ad74bf048f8869760f438246
 SHA512:
-  metadata.gz: 060ff017a7a0de4ede9652d3e87ddd14083022725ce225739758ce0c7882fe953d6bfaf27ccf48f4fa8b73b480fce98f623dfaf02406944e19e841763b42c022
-  data.tar.gz: 0b80b3df83f7399b5118d67abbf2e8c1da589b0ec902d7f2cc04a60be8498e41df9ebdbc569af9c57b487025994bfd5715a58b0d0c2a9a6994e06271fa2e2dcd
+  metadata.gz: c2197ab3c0b5759a94bd8027e8c4096c886b3186e37be9cd06e562710db94677450cce228c3a9c106156a43857591a1c0c18fd3dd8eddba617c644fa6f3bee6c
+  data.tar.gz: c1a2612a43896411f0462732d37641fb864fb84dc48eef2c81d96e182295bb69c0b4ad8be6c1fa04bc53368ba8a906669bd7e8e58735abc5fd034b7e7821f627

data/README.md

@@ -47,15 +47,23 @@ zip reader and a binary read parsed as CSV yields zero rows — a *lower* exposu
 than the truth, and the wrong direction to fail in silently. The shipped table is
 gzip, which `zlib` reads, so that hazard does not arise on the default path.
 
-The last three need an essay corpus no package here ships, and this gem measures
-them once `VICARY_EVAL_CORPUS_TSV` points at one: 100% carrier recall (29/29
-held-out REDACT spans) and 0.60 over-fired spans per essay (15 across 25),
-identical to Python and TypeScript, plus its own latency p95 of 8.8–9.9 ms. **That
-last one is worth knowing about**: the bar is ≤ 10 ms, and this port runs nearest
-it of the three — roughly 4× Python — so the latency gate is a live constraint
-here rather than a formality. The carrier essays are built from offsets recorded
-in `conformance/carrier.json` rather than from a reimplementation of Python's
-RNG, and the suite asserts their sha256.
+Two of the last three read the corpus the repository now ships in
+`conformance/corpora/`, so they measure on a bare checkout with no environment
+set: 100% carrier recall and 8.150 over-fired spans per essay against a ≤ 8.15
+bar, identical to Python and TypeScript. `VICARY_EVAL_CORPUS_TSV` is an override
+for a different corpus, not a requirement.
+
+The ninth is latency, and **this port's absolute figure no longer constrains it**.
+The gate was a ≤ 10 ms bar, which this port ran nearest of the three; it is now a
+ratio against the last release timed on the same machine, held to ≤ +8%. That
+change matters most here: across three CPU models this port's absolute median
+spreads **31.8%**, the same axis that made the old bar a coin flip, while its
+ratio spreads 0.36 pp. Measured, the ratio holds σ 0.46% — the *widest* margin of
+the three ports, where the absolute figure gave it the narrowest.
+
+The carrier essays are built from offsets recorded in `conformance/carrier.json`
+rather than from a reimplementation of Python's RNG, and the suite asserts their
+sha256.
 
 The last two need the reference interpreter — run `just py-setup` from the
 repository root first.

data/lib/vicary/corpus.rb

@@ -445,13 +445,23 @@ module Vicary
         over_fire = 0
         rewrites = 0
 
-        # Load the gazetteer before the clock starts. It is a one-time ~207 ms
-        # cost in this port, and whichever essay happens to be first pays all of
-        # it: at n=25 that single sample lands at or above p95 and sets the
-        # gate's answer by itself — 14.3 ms cold against 7.6 ms warm, on a 10 ms
-        # bar. The number the gate claims is essay-length redaction latency, not
-        # process startup. Excluded in all three ports alike.
-        yield(cases.first.base[0, 200], identity) unless cases.empty?
+        # Warm up before the clock starts, over the WHOLE corpus rather than one
+        # 200-char call. Two costs are being excluded, and the second one is why
+        # this grew.
+        #
+        # The gazetteer load is a one-time ~207 ms cost in this port, and
+        # whichever essay happens to be first pays all of it — 14.3 ms cold
+        # against 7.6 ms warm.
+        #
+        # The second belongs to TypeScript, where V8 tiers the redaction path up
+        # over roughly the first four essays and runs them at about twice their
+        # steady-state cost. This port barely moves under a full warmup and does
+        # it anyway: the three ports measure identically or the gate is three
+        # different gates.
+        cases.each do |kase|
+          yield(kase.text, identity)
+          yield(kase.base, identity)
+        end
 
         cases.each do |kase|
           # The median of LATENCY_REPEATS, not one sample — see that constant.

data/lib/vicary/latency_baseline.rb

@@ -6,37 +6,77 @@ require "pathname"
 module Vicary
   # Is this build slower than the last release, and is that a fair question here?
   #
-  # The latency gate used to hold an absolute number — 10 ms — which is a claim
-  # about the machine as much as about the code. It passed on a laptop and failed
-  # on the CI runner enforcing it, so v0.2.3 published to PyPI and npm and was
-  # refused by RubyGems on the same commit. This gem is the one that caught it.
+  # The gate has asked this three ways. The first two are worth keeping in view,
+  # because each looked correct until it decided a release.
   #
-  # What replaced it asks a relative question: is this port slower than it was at
-  # the last release, by more than the tolerance. That only means something
-  # between measurements taken on comparable hardware, so this module's real work
-  # is REFUSING to compare when they are not — a machine difference reported as a
-  # code regression is worse than no gate, because it trains the reader to ignore
-  # it.
+  # **An absolute bar — 10 ms.** A claim about the machine as much as about the
+  # code. It passed on a laptop and failed on the CI runner enforcing it, so
+  # v0.2.3 published to PyPI and npm and was refused by RubyGems on the same
+  # commit. This gem is the one that caught it.
   #
-  # This port reaches its own verdict from the shared file. It does not read
+  # **A stored baseline** — record each release's number and compare the next run
+  # against it, refusing unless the run claims the profile the baseline was
+  # recorded on. Better, and still wrong, for a reason no estimator fixes: the
+  # profile `github-ubuntu-latest` is not a machine. Thirty-six processes across
+  # six runners per port, on identical code, spread 67% in THIS port — 6.53 ms on
+  # an Intel Xeon 6973P-C against 10.63 ms on an EPYC 7763 — 26% in Python and
+  # 21% in TypeScript, against an 8% bar. One probe run drew five CPU models from
+  # that one label, and two runners of the same model still differed by 26%.
+  #
+  # **A pair, measured here.** The previous release's code and this checkout,
+  # measured on the SAME machine, interleaved and counterbalanced, by
+  # `tools/latency_pair.py`. Every property of the machine is common to both
+  # sides and cancels; what is left is within-process noise, 1.7% in this port.
+  #
+  # Which leaves this module the job it has always had: REFUSING to compare when
+  # the two sides would not be like for like. What changed is that the refusals
+  # are about the pair record — is there one, is it this port's, was it measured
+  # on these essays, was it measured for this commit — rather than about the
+  # profile of a machine somewhere else.
+  #
+  # This port reaches its own verdict from the shared record. It does not read
   # Python's answer.
   module LatencyBaseline
-    BASELINE_FILENAME = "latency_baseline.json"
-
-    # Set by CI on the one matrix entry whose language version matches the
-    # recorded profile. Absent everywhere else on purpose: a developer's laptop
-    # measures the same commit two to three times faster than the runner, and
-    # comparing that against a runner baseline reports a phantom improvement.
-    PROFILE_ENV_VAR = "VICARY_LATENCY_PROFILE"
+    # The tolerance and the protocol, in the repository. Not a measurement:
+    # nothing is recorded at release time any more, because the comparison point
+    # is the previous release's *code*, which the repository already has.
+    SPEC_FILENAME = "latency_baseline.json"
+
+    # Where `tools/latency_pair.py` left the paired measurement. Set by CI in the
+    # same job, seconds before the gate runs. Absent on a laptop unless the
+    # harness was run there by hand, and that absence is a refusal to compare
+    # rather than a pass — measuring one side of a comparison is not a gate.
+    PAIR_ENV_VAR = "VICARY_LATENCY_PAIR"
+
+    # What this reader understands. A record from a future shape is refused
+    # rather than half-read: a partly-understood record still yields a number,
+    # and a number is exactly what must not be invented here.
+    PAIR_DOCUMENT_VERSION = 1
 
     IMPLEMENTATION = "ruby"
 
+    # The bar, chosen rather than derived — 8% is what a reviewer is willing to
+    # call a regression. What the noise decides is whether the bar is USABLE,
+    # and it is: the gate statistic holds **sigma 0.46%** in this port (95% CI
+    # 0.34-0.72%, sixteen runs across eight CI runners, fixed head and tag),
+    # putting 8% at 17.2 sigma — the widest margin of the three. It was about a
+    # third of a sigma under the stored baseline, which is how that one red-lit
+    # `main` on unchanged code, and how it refused this port's 0.2.3 while the
+    # other two took the same commit.
+    #
+    # This is also the port where the pairing earns the most. Across three CPU
+    # models its absolute figure spreads **31.8%** — the same axis, and nearly
+    # the same size, as the 67% that killed the stored baseline — while its ratio
+    # spreads 0.36 pp. Same runs, same data. See `tools/latency_pair.py`.
+    #
+    # It does not catch drift: +5% a release passes every time and compounds.
+    # That is deliberate — this gate is for the step change, not the trend.
     DEFAULT_TOLERANCE_PCT = 8.0
 
     # The gate's answer, and — when it declines — why.
     Comparison = Struct.new(
-      :measured_ms, :baseline_ms, :regression_pct, :tolerance_pct,
-      :comparable, :reason,
+      :measured_ms, :previous_ms, :current_ms, :regression_pct, :tolerance_pct,
+      :against, :comparable, :reason,
       keyword_init: true
     ) do
       def holds?
@@ -47,97 +87,114 @@ module Vicary
     end
 
     class << self
-      def baseline_path(dir = nil)
+      def spec_path(dir = nil)
         root = dir || Conformance.directory
         return nil if root.nil?
 
-        path = Pathname.new(root).join(BASELINE_FILENAME)
+        path = Pathname.new(root).join(SPEC_FILENAME)
         path.exist? ? path : nil
       end
 
       def load(dir = nil)
-        path = baseline_path(dir)
+        path = spec_path(dir)
         return nil if path.nil?
 
         JSON.parse(path.read)
       end
 
-      # `major.minor` of the running Ruby, matching how the profile records it.
-      def language_version
-        RUBY_VERSION.split(".").first(2).join(".")
+      # The paired measurement, or why there is none to read.
+      #
+      # An unreadable file and an absent one stay distinguishable: the first is a
+      # broken harness and the second is an ordinary laptop, and they should not
+      # report the same thing.
+      def load_pair(path = nil)
+        given = (path || ENV[PAIR_ENV_VAR] || "").strip
+        if given.empty?
+          return [nil,
+                  "#{PAIR_ENV_VAR} is unset, so no paired measurement was taken on " \
+                  "this machine; the gate compares this build against the last " \
+                  "release measured HERE, and one side of a comparison is not a gate"]
+        end
+        return [nil, "#{PAIR_ENV_VAR}=#{given.inspect} does not exist"] unless File.exist?(given)
+
+        begin
+          [JSON.parse(File.read(given)), nil]
+        rescue StandardError => e
+          [nil, "the pair record at #{given} could not be read: #{e.message}"]
+        end
       end
 
-      # Compare +measured_ms+ against the recorded baseline for this port.
+      # Compare the pair measured on this machine, for this port.
       #
-      # Every reason below is a refusal to compare, not a failure to measure: the
-      # number was measured either way and is reported either way. What is
-      # withheld is the verdict, because the two sides would not be like for like.
+      # +measured_ms+ is this process's own figure. It is reported either way and
+      # it is never the verdict: the verdict comes from the two numbers in the
+      # pair record, taken back to back on one machine. Mixing this process's
+      # measurement with the pair's other side would reintroduce exactly the
+      # machine difference the pair exists to cancel.
       def compare(measured_ms, corpus_id, dir: nil, implementation: IMPLEMENTATION,
-                  observed_language_version: nil, profile_env: nil)
-        doc = load(dir)
-        tolerance = (doc && doc["tolerance_pct"] || DEFAULT_TOLERANCE_PCT).to_f
-        lang = observed_language_version || language_version
-
-        declined = lambda do |reason, baseline_ms = nil|
-          Comparison.new(measured_ms: measured_ms, baseline_ms: baseline_ms,
-                         regression_pct: nil, tolerance_pct: tolerance,
+                  pair_path: nil, building_sha: nil)
+        doc = load(dir) || {}
+        tolerance = (doc["tolerance_pct"] || DEFAULT_TOLERANCE_PCT).to_f
+
+        declined = lambda do |reason|
+          Comparison.new(measured_ms: measured_ms, previous_ms: nil, current_ms: nil,
+                         regression_pct: nil, tolerance_pct: tolerance, against: nil,
                          comparable: false, reason: reason)
         end
 
-        return declined.call("no #{BASELINE_FILENAME} in this checkout") if doc.nil?
+        record, why = load_pair(pair_path)
+        return declined.call(why || "no paired measurement") if record.nil?
 
-        profile = doc["profile"] || {}
-        want_profile = profile["id"]
-        have_profile = (profile_env || ENV[PROFILE_ENV_VAR] || "").strip
-        if have_profile.empty?
+        unless record["document_version"] == PAIR_DOCUMENT_VERSION
           return declined.call(
-            "#{PROFILE_ENV_VAR} is unset, so this machine does not claim to be " \
-            "#{want_profile.inspect}; the baseline was recorded there"
+            "the pair record is document_version #{record['document_version']} " \
+            "and this reader knows #{PAIR_DOCUMENT_VERSION}"
           )
         end
-        unless have_profile == want_profile
+        unless record["implementation"] == implementation
           return declined.call(
-            "#{PROFILE_ENV_VAR}=#{have_profile.inspect} but the baseline was " \
-            "recorded on #{want_profile.inspect}"
+            "the pair record measures #{record['implementation'].inspect}, " \
+            "not #{implementation.inspect}"
           )
         end
-
-        want_lang = (profile["language_versions"] || {})[implementation]
-        if !want_lang.nil? && want_lang.to_s != lang
+        unless record["corpus"] == corpus_id
           return declined.call(
-            "#{implementation} #{lang} is not the #{want_lang} the baseline was " \
-            "recorded on; interpreter versions differ by more than the bar"
+            "the pair was measured on corpus #{record['corpus'].inspect} and this " \
+            "run is #{corpus_id.inspect}; latency scales with essay length"
           )
         end
 
-        want_corpus = doc["corpus"]
-        if !want_corpus.nil? && want_corpus != corpus_id
+        # Only where there is something to check against. `GITHUB_SHA` names the
+        # commit the job is building, so a record left over from an earlier
+        # commit is caught here rather than being read as this build's verdict.
+        # Locally there is no such witness and no such risk: the harness is run
+        # by hand, minutes before, on the tree in front of you.
+        building = (building_sha || ENV["GITHUB_SHA"] || "").strip
+        head = record["head_sha"].to_s
+        if !building.empty? && !head.empty? && building != head
           return declined.call(
-            "corpus #{corpus_id.inspect} is not the #{want_corpus.inspect} the " \
-            "baseline was recorded on; latency scales with essay length"
+            "the pair was measured for commit #{head[0, 12]} and this job is " \
+            "building #{building[0, 12]}; the record is stale"
           )
         end
 
-        entry = (doc["implementations"] || {})[implementation] || {}
-        recorded = entry["pooled_median_ms"]
-        if recorded.nil?
-          return declined.call(
-            "no baseline recorded for #{implementation} yet — the next release " \
-            "records one"
-          )
+        previous = record["previous_ms"]
+        current = record["current_ms"]
+        unless previous.is_a?(Numeric) && current.is_a?(Numeric)
+          return declined.call("the pair record carries no pair of measurements")
         end
-
-        recorded = recorded.to_f
-        if recorded <= 0
+        if previous <= 0
           return declined.call(
-            "recorded baseline for #{implementation} is not positive", recorded
+            "the previous release measured #{previous} ms, which is not positive"
           )
         end
 
         Comparison.new(
-          measured_ms: measured_ms, baseline_ms: recorded,
-          regression_pct: (measured_ms / recorded - 1.0) * 100.0,
-          tolerance_pct: tolerance, comparable: true, reason: nil
+          measured_ms: measured_ms, previous_ms: previous.to_f,
+          current_ms: current.to_f,
+          regression_pct: (current.to_f / previous.to_f - 1.0) * 100.0,
+          tolerance_pct: tolerance, against: (record["against"] || {})["ref"],
+          comparable: true, reason: nil
         )
       end
 
@@ -149,9 +206,10 @@ module Vicary
         end
 
         sign = c.regression_pct >= 0 ? "+" : ""
-        format("latency %.3f ms vs %.3f ms at the last release — %s%.2f%% " \
-               "against a %d%% bar",
-               c.measured_ms, c.baseline_ms, sign, c.regression_pct, c.tolerance_pct)
+        format("latency %.3f ms here; paired on this machine, %.3f ms against " \
+               "%s's %.3f ms — %s%.2f%% against a %d%% bar",
+               c.measured_ms, c.current_ms, c.against || "the last release",
+               c.previous_ms, sign, c.regression_pct, c.tolerance_pct)
       end
 
       # The keyword arguments Gates.measure wants. Returns the *detail* rather

data/lib/vicary/version.rb

@@ -6,5 +6,5 @@ module Vicary
   # Shared across all three front doors on purpose: one detector, one number. A
   # gem 0.3.0 that corresponds to nothing on PyPI cannot be reasoned about, and
   # the parity claim is between *versions*, not between package names.
-  VERSION = "0.2.4"
+  VERSION = "0.2.5"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: vicary
 version: !ruby/object:Gem::Version
-  version: 0.2.4
+  version: 0.2.5
 platform: ruby
 authors:
 - Blake Thomas