vicary 0.2.1 → 0.2.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 94923f9a64b909c5a6ab25e3d7dc1d8caa41843068e1ede77fe938905a1a5745
-  data.tar.gz: 43e9513288654b871a78115c0f90d86c4f51701944dc01e20df469a62991852d
+  metadata.gz: 585c77607d6c331c3cebe14a5da44463a0231f7c3a15e5c1ba68f79d8b9f588b
+  data.tar.gz: 883c07a66721ce9c67fb0cfe1f1ea16ce486decd6be62164b1ebbbf53fec9bf0
 SHA512:
-  metadata.gz: 888df67ec5e7842da4d43a9398ea28edcb6552f47c84fada77bc9912e3dcbc4a56f1ea4f2b19c67dc0c5701eefc9fef6b6e671e7b29f2c034c92fc7d5dc0e129
-  data.tar.gz: 650b3ca105ef6dc6801c0232606077bef713f3c84d55668183707c296233c9d6c6c3438e21dbb5d218457735782fb671c5bb97a7fd15d06242059649c46ca549
+  metadata.gz: 060ff017a7a0de4ede9652d3e87ddd14083022725ce225739758ce0c7882fe953d6bfaf27ccf48f4fa8b73b480fce98f623dfaf02406944e19e841763b42c022
+  data.tar.gz: 0b80b3df83f7399b5118d67abbf2e8c1da589b0ec902d7f2cc04a60be8498e41df9ebdbc569af9c57b487025994bfd5715a58b0d0c2a9a6994e06271fa2e2dcd

data/README.md

@@ -32,10 +32,31 @@ Three layers, because each catches what the one above it cannot.
 | command | what it says |
 |---|---|
 | `rake conformance` | the scoreboard against the 54 frames — the final bar, and a coarse first one |
+| `rake gates` | the nine gates, all nine measured from what the repository ships |
 | `rake test` | the unit suites, including `primitives_test.rb`: forty-odd primitives over the shared corpus, which says *which brick* is crooked |
 | `rake parity` | gazetteer verdicts, name by name, against the Python reference |
 | `rake redaction_parity` | masked bytes against the Python reference, on prose no fixture contains |
 
+A sixth gate — bare-surname exposure — this gem measures from the surname table
+shipped in `conformance/census/`, reporting 1.20% of US surname bearers, the same
+figure Python and TypeScript report from the same table. `VICARY_EVAL_CENSUS_CSV`
+overrides it with your own Census copy, and must be the **extracted**
+`Names_2010Census.csv` from the census.gov 2010 surnames release: a `.zip` is
+refused by name rather than read as text, because Ruby's standard library has no
+zip reader and a binary read parsed as CSV yields zero rows — a *lower* exposure
+than the truth, and the wrong direction to fail in silently. The shipped table is
+gzip, which `zlib` reads, so that hazard does not arise on the default path.
+
+The last three need an essay corpus no package here ships, and this gem measures
+them once `VICARY_EVAL_CORPUS_TSV` points at one: 100% carrier recall (29/29
+held-out REDACT spans) and 0.60 over-fired spans per essay (15 across 25),
+identical to Python and TypeScript, plus its own latency p95 of 8.8–9.9 ms. **That
+last one is worth knowing about**: the bar is ≤ 10 ms, and this port runs nearest
+it of the three — roughly 4× Python — so the latency gate is a live constraint
+here rather than a formality. The carrier essays are built from offsets recorded
+in `conformance/carrier.json` rather than from a reimplementation of Python's
+RNG, and the suite asserts their sha256.
+
 The last two need the reference interpreter — run `just py-setup` from the
 repository root first.
 

data/lib/vicary/candidates.rb

@@ -166,6 +166,47 @@ module Vicary
     # corpus.
     SENTENCE_BREAK = /(?:\A|[.!?]["'’”)]*\s+|\n+|(?:(?<=\s)|\A)["'‘“](?=[A-Za-z]))\s*/
 
+    # Words whose trailing period abbreviates rather than ends a sentence, and
+    # which are followed by a name more often than not.
+    #
+    # Load-bearing for {.sentence_starts}, and the reason is a leak. The break
+    # pattern reads `[.!?]\s+` as a sentence boundary, so "Mrs. Okonkwo" put
+    # "Okonkwo" in sentence-initial position — where a capital is orthographically
+    # required and therefore proves nothing — and the document's one piece of
+    # testimony about that surname was discarded. In a persuade-20 carrier essay
+    # that withdrew the corroboration the lowercase route needed and leaked
+    # "terrence okonkwo". An honorific is the exact case where the capital that
+    # follows is *most* likely to be a name, so reading it as a sentence start
+    # inverts the signal.
+    #
+    # Deliberately only titles, not every abbreviation. "etc." or "vs." are also
+    # not sentence ends, but nothing follows them that this set exists to protect,
+    # and a wider list costs precision everywhere for no recall.
+    TITLE_ABBREVIATIONS = Set.new(%w[
+      mr mrs ms dr prof rev fr sr jr st
+      sgt capt lt col gen gov sen rep hon
+    ]).freeze
+
+    # Matches the abbreviation a break candidate sits directly behind.
+    TRAILING_WORD = /([A-Za-z]+)\.\s*\z/
+
+    # {SENTENCE_BREAK} with the `\A` arm removed, so it cannot match empty.
+    #
+    # A dialect difference, and it moved a number. `SENTENCE_BREAK` matches the
+    # empty string at offset 0 via its `\A` arm. Python's `re` then retries a
+    # **non-empty** match at that same offset before advancing; {.each_match}
+    # advances past a zero-width match instead and never looks again. On an essay
+    # that opens with a quotation — `"Pedestrian, bicycle, private cars…` —
+    # Python records sentence starts at both 0 and 1, this port recorded only 0,
+    # and `Pedestrian` lost the sentence-initial discount its capital is owed. It
+    # was then read as a name and masked: one over-fire span on `persuade-20`
+    # that Python does not produce.
+    #
+    # Every arm here consumes at least one character, which is what makes the
+    # retry terminate.
+    SENTENCE_BREAK_NONEMPTY =
+      /(?:[.!?]["'’”)]*\s+|\n+|(?:(?<=\s)|\A)["'‘“](?=[A-Za-z]))\s*/
+
     # One entirely-lowercase word. The leading boundary is what keeps this from
     # matching the tail of a capitalised word — there is no word boundary between
     # the "T" and the "errence" of "Terrence", so the capitalised route keeps
@@ -728,8 +769,35 @@ module Vicary
       # ---------------------------------------------------------------------
 
       # Offsets at which a sentence begins.
+      #
+      # A break directly behind a title abbreviation is not one — see
+      # {TITLE_ABBREVIATIONS} for the leak that rule exists to close.
       def sentence_starts(text)
-        each_match(text, SENTENCE_BREAK).map { |m| m.begin(0) + m[0].length }.to_set
+        out = Set.new
+        keep = lambda do |finish|
+          preceding = TRAILING_WORD.match(text[0...finish])
+          next if preceding && TITLE_ABBREVIATIONS.include?(preceding[1].downcase)
+
+          out << finish
+        end
+
+        pos = 0
+        while pos <= text.length && (m = SENTENCE_BREAK.match(text, pos))
+          keep.call(m.begin(0) + m[0].length)
+          if m[0].empty?
+            # Empty match: Python retries a non-empty one here before moving on.
+            again = SENTENCE_BREAK_NONEMPTY.match(text, m.begin(0))
+            if again && again.begin(0) == m.begin(0)
+              keep.call(again.end(0))
+              pos = again.end(0)
+            else
+              pos = m.begin(0) + 1
+            end
+          else
+            pos = m.end(0)
+          end
+        end
+        out
       end
 
       # Character ranges of all-caps runs SHORTER than {ALLCAPS_RUN}.
@@ -1427,10 +1495,6 @@ module Vicary
             index += 1
             next
           end
-          if !corroborate.nil? && !corroborate.include?(strip(word, "'’"))
-            index += 1
-            next
-          end
           if index.positive? && DETERMINERS.include?(tokens[index - 1][0])
             # Only a directly-adjacent determiner counts. "the day terrence
             # arrived" must stay reachable, and punctuation between the two means
@@ -1460,6 +1524,23 @@ module Vicary
             index += 1
             next
           end
+          # Corroboration is asked of the WHOLE span, not of its first token, and
+          # it is asked here rather than before `reach` is known because the span's
+          # extent is what decides which tokens may vouch for it.
+          #
+          # Checking only the given name is what leaked "terrence okonkwo" out of a
+          # persuade-20 carrier essay. That document is INCONSISTENT, so this route
+          # runs with corroboration required; it capitalises "Okonkwo" mid-sentence
+          # and never writes "Terrence" at all, so the one token consulted was the
+          # one the writer happened not to capitalise — while the surname of the
+          # same person sat in the same document as exactly the evidence being
+          # asked for. {.corroborated?} already settled this question the other way
+          # ("ANY token counts, not just the first"); this channel simply never
+          # adopted it.
+          if !corroborate.nil? && (index..reach).none? { |i| corroborate.include?(strip(tokens[i][0], "'’")) }
+            index += 1
+            next
+          end
           joined = text[start...span_end]
           out << Candidate.new(joined, start, span_end, classify(words(joined), settlement))
           index = reach + 1

data/lib/vicary/census.rb

@@ -0,0 +1,285 @@
+# frozen_string_literal: true
+
+require "digest"
+require "json"
+require "pathname"
+require "set"
+require "zlib"
+
+module Vicary
+  # The false-positive control the fixture cannot provide.
+  #
+  # The fixture reports zero leaks on bare surnames partly because the private
+  # surnames in it are rare — Okonkwo, Bramwell, Pritchard, Ybarra. A clean
+  # control needs an *unlikely* clean, so this scores the single-token tiers
+  # against **every American surname**: the population-weighted rate at which a
+  # bare surname resolves notable, regardless of whose surname it is.
+  #
+  # Read the headline number as: *for a private person named by bare surname
+  # only — no first name, no title, no same-document corroboration — this share
+  # resolves "notable" and leaks.* It is conditional on that surface form, which
+  # is a minority of private-name mentions in real prose, so it is not an
+  # essay-level leak rate.
+  #
+  # The source is the US Census 2010 surname file, and this repository now ships
+  # the two columns of it this measurement uses — see `conformance/census/`,
+  # built by `tools/census_build.py`. So the gate is measured on a bare checkout
+  # and in CI, which it was not: census.gov stopped serving the upstream, and the
+  # gate reported NOT MEASURED everywhere but on a machine holding a
+  # hand-downloaded copy.
+  #
+  # `VICARY_EVAL_CENSUS_CSV` still wins when set — an operator holding a newer
+  # release gets the number their file gives.
+  #
+  # **For that operator file, this port reads the extracted `.csv` only.** Python
+  # additionally accepts the distributed `.zip` because its standard library has
+  # a zip reader and Ruby's does not. A `.zip` here is refused by name rather
+  # than parsed as text, since the alternative is a binary read that yields zero
+  # rows — which is a *lower* exposure rate than the truth, and the wrong
+  # direction to fail in silently. The shipped table sidesteps this entirely: it
+  # is gzip, which `zlib` reads.
+  module Census
+    # Where a locally-held copy of the Census surname file is configured.
+    EVAL_CENSUS_CSV_ENV_VAR = "VICARY_EVAL_CENSUS_CSV"
+
+    # The member name inside the distributed archive, and the file this port
+    # wants handed to it directly.
+    CENSUS_SURNAMES_MEMBER = "Names_2010Census.csv"
+
+    # Where the operator gets the file, quoted when it is missing.
+    CENSUS_SURNAMES_URL = "https://www2.census.gov/topics/genealogy/2010surnames/names.zip"
+
+    # The row-count floor. Not decoration: this list is scored *against* the
+    # single-token tiers, so a short read shrinks the denominator and reports a
+    # more comfortable exposure rate than the truth.
+    MINIMUM_ROWS = 100_000
+
+    # Directory under `conformance/` holding the shipped table and its provenance.
+    SHIPPED_DIRNAME = "census"
+    SHIPPED_TABLE_FILENAME = "surnames.txt.gz"
+    SHIPPED_PROFILE_FILENAME = "profile.json"
+
+    # How much of the US surname population the single-token tiers claim.
+    Exposure = Struct.new(
+      :surnames_scored,   # Distinct surnames in the Census file.
+      :surnames_matched,  # Distinct surnames matching some single-token tier.
+      :bearers_total,     # Total bearers across the file.
+      :bearers_exposed,   # Bearers whose surname matches some single-token tier.
+      :bearers_via_short, # Bearers exposed via the `short` tier specifically.
+      :bearers_via_place, # Bearers exposed via a single-token `place` entry.
+      # Bearers exposed via the `demonym` tier. Counted here for the same reason
+      # the other two are: it is a KEEP granted to a bare single token, which is
+      # exactly the surface form this control measures. Leaving it out would make
+      # adding a tier look free.
+      :bearers_via_demonym,
+      keyword_init: true
+    ) do
+      # Population-weighted exposure, as a percentage. The headline.
+      def rate
+        100.0 * bearers_exposed / bearers_total
+      end
+
+      def short_rate
+        100.0 * bearers_via_short / bearers_total
+      end
+
+      def place_rate
+        100.0 * bearers_via_place / bearers_total
+      end
+
+      def demonym_rate
+        100.0 * bearers_via_demonym / bearers_total
+      end
+
+      def distinct_rate
+        100.0 * surnames_matched / surnames_scored
+      end
+    end
+
+    class << self
+      # Configured path to a local Census surname file, or `""`.
+      def census_source
+        (ENV[EVAL_CENSUS_CSV_ENV_VAR] || "").strip
+      end
+
+      # `{normalised surname => number of US bearers}` from the Census CSV text.
+      #
+      # Field-indexed off the header rather than positional, so a column added
+      # upstream shifts nothing. The file carries no quoted fields — every row is
+      # 11 bare comma-separated values — so this splits rather than requiring
+      # `csv`, and a row that does not yield an integer count is skipped the same
+      # way Python's `DictReader` loop skips it.
+      def parse_census_surnames(text)
+        counts = {}
+        lines = text.split(/\r?\n/, -1)
+        header = (lines.first || "").split(",")
+        name_at = header.index("name")
+        count_at = header.index("count")
+        if name_at.nil? || count_at.nil?
+          raise ArgumentError,
+                "Census surname file has no 'name'/'count' header; got #{header.join(',')}"
+        end
+
+        lines.drop(1).each do |line|
+          next if line.empty?
+
+          fields = line.split(",")
+          name = Gazetteer.normalize(fields[name_at] || "")
+          next if name.empty? || name == "all other names"
+
+          count = Integer(fields[count_at], exception: false)
+          next if count.nil?
+
+          counts[name] = count
+        end
+
+        if counts.size < MINIMUM_ROWS
+          raise RuntimeError,
+                "Census surname file parsed to only #{counts.size} rows; expected ~162k. " \
+                "Refusing to score exposure against a truncated list, because the " \
+                "failure mode is a more comfortable rate than the truth."
+        end
+        counts
+      end
+
+      # `conformance/census/`, or nil outside a checkout.
+      def shipped_dir
+        candidate = Conformance.directory.join(SHIPPED_DIRNAME)
+        candidate.join(SHIPPED_TABLE_FILENAME).file? ? candidate : nil
+      rescue Conformance::SpecError
+        nil
+      end
+
+      # `{normalised surname => bearers}` from the table this repository ships.
+      #
+      # The digest in `profile.json` is checked, not trusted. This table is used
+      # to SUBTRACT exposure from a permissive tier, so a truncated or edited
+      # copy scores the gazetteer against a smaller America and reads as a
+      # *better* number — the one direction this measurement must never fail in
+      # quietly. A bad digest raises rather than degrading.
+      def load_shipped_census(directory = nil)
+        dir = directory ? Pathname.new(directory) : shipped_dir
+        if dir.nil?
+          raise Errno::ENOENT,
+                "no conformance/#{SHIPPED_DIRNAME}/ above this module. The shipped " \
+                "table lives in the repository, not in an installed gem."
+        end
+
+        payload = dir.join(SHIPPED_TABLE_FILENAME).binread
+        profile = JSON.parse(dir.join(SHIPPED_PROFILE_FILENAME).read)
+        expected = profile.dig("table", "sha256").to_s
+        actual = Digest::SHA256.hexdigest(payload)
+        if !expected.empty? && actual != expected
+          raise RuntimeError,
+                "#{SHIPPED_TABLE_FILENAME} has sha256 #{actual}, but " \
+                "#{SHIPPED_PROFILE_FILENAME} pins #{expected}. Refusing to score the " \
+                "gazetteer against a table that is not the one this repository " \
+                "measured, because a short read reads as a better number. Rebuild " \
+                "with `python tools/census_build.py --write`."
+        end
+
+        counts = {}
+        Zlib.gunzip(payload).force_encoding("UTF-8").each_line do |line|
+          name, _, bearers = line.chomp.partition("\t")
+          counts[name] = Integer(bearers) unless name.empty?
+        end
+        if counts.size < MINIMUM_ROWS
+          raise RuntimeError,
+                "#{SHIPPED_TABLE_FILENAME} parsed to only #{counts.size} rows; " \
+                "expected at least #{MINIMUM_ROWS}."
+        end
+        counts
+      end
+
+      # `{normalised surname => bearers}`, resolved in this order:
+      #
+      # 1. An explicit `source`, or `VICARY_EVAL_CENSUS_CSV`. An operator holding
+      #    a newer Census release still wins, and gets the number *their* file
+      #    gives.
+      # 2. The table shipped in `conformance/census/`, which is the same 162,253
+      #    rows the 2010 release carries and therefore the same rate to the last
+      #    bearer. This is why the gate no longer skips on a bare checkout.
+      #
+      # There is no third step. census.gov answers the documented URL with a WAF
+      # rejection page under a 200 status, which is why the shipped table exists.
+      def load_census(source = nil)
+        path = (source || census_source).strip
+        if path.empty?
+          return load_shipped_census unless shipped_dir.nil?
+
+          raise Errno::ENOENT,
+                "no conformance/#{SHIPPED_DIRNAME}/ in this tree and no " \
+                "#{EVAL_CENSUS_CSV_ENV_VAR} set. Point that at a copy of " \
+                "#{CENSUS_SURNAMES_MEMBER}, extracted from #{CENSUS_SURNAMES_URL}, " \
+                "or run from a checkout"
+        end
+        if path.downcase.end_with?(".zip")
+          raise ArgumentError,
+                "#{path} is a .zip and this port reads the extracted .csv only. " \
+                "Extract #{CENSUS_SURNAMES_MEMBER} from it and point " \
+                "#{EVAL_CENSUS_CSV_ENV_VAR} at that."
+        end
+        parse_census_surnames(File.read(path, encoding: "UTF-8"))
+      end
+
+      # Score the loaded gazetteer's single-token tiers against the Census file.
+      def measure(census, gaz = nil)
+        gaz ||= Gazetteer.load
+        single_token_places = gaz.place.reject { |n| n.include?(" ") }.to_set
+        single = single_token_places | gaz.short.to_set | gaz.demonym.to_set
+
+        bearers_total = 0
+        bearers_exposed = 0
+        surnames_matched = 0
+        via_short = 0
+        via_place = 0
+        via_demonym = 0
+
+        census.each do |name, count|
+          bearers_total += count
+          if single.include?(name)
+            bearers_exposed += count
+            surnames_matched += 1
+          end
+          via_short += count if gaz.short.include?(name)
+          via_place += count if single_token_places.include?(name)
+          via_demonym += count if gaz.demonym.include?(name)
+        end
+
+        Exposure.new(
+          surnames_scored: census.size,
+          surnames_matched: surnames_matched,
+          bearers_total: bearers_total,
+          bearers_exposed: bearers_exposed,
+          bearers_via_short: via_short,
+          bearers_via_place: via_place,
+          bearers_via_demonym: via_demonym
+        )
+      end
+
+      # The report block, for a CLI or a gate's failure message.
+      def render(exposure)
+        [
+          "BARE-SURNAME FALSE-POSITIVE RATE (US Census 2010 surname file)",
+          "  distinct surnames scored   #{group(exposure.surnames_scored)}",
+          "  any single-token tier hit  #{group(exposure.surnames_matched)} " \
+          "(#{format('%.2f', exposure.distinct_rate)}% of distinct)",
+          "  population-weighted rate   #{format('%.2f', exposure.rate)}% " \
+          "(#{group(exposure.bearers_exposed)} / #{group(exposure.bearers_total)} bearers)",
+          "    via the short tier       #{format('%.2f', exposure.short_rate)}%",
+          "    via single-token places  #{format('%.2f', exposure.place_rate)}%",
+          "    via the demonym tier     #{format('%.2f', exposure.demonym_rate)}%",
+          "  reads as: for a private person named by BARE SURNAME ONLY — no",
+          "            first name, no title, no corroboration — this share",
+          "            resolves 'notable'. Conditional on that surface form."
+        ].join("\n")
+      end
+
+      private
+
+      def group(value)
+        value.to_s.reverse.scan(/\d{1,3}/).join(",").reverse
+      end
+    end
+  end
+end

data/lib/vicary/conformance.rb

@@ -33,8 +33,20 @@ module Vicary
                         keyword_init: true)
     Spec = Struct.new(:fixture_version, :reference_arm, :identity, :frames,
                       :golden, keyword_init: true)
-    Gate = Struct.new(:id, :label, :unit, :op, :bar, :requires, :why,
-                      keyword_init: true)
+    # `bars_by_corpus` overrides `bar` per corpus id. Only `over_fire_prose`
+    # carries one: it is the sole gate whose bar describes the *prose* rather
+    # than the detector, and ASAP-AES's 0.61 against PERSUADE's 8.15 is what one
+    # corpus naming real entities and another naming `@PERSON` tokens costs.
+    # `bar` remains the fallback, and is the tighter of the two.
+    Gate = Struct.new(:id, :label, :unit, :op, :bar, :bars_by_corpus, :requires,
+                      :why, keyword_init: true) do
+      # The bar this gate holds `corpus_id` to — its override, else its default.
+      def bar_for(corpus_id)
+        return bar if corpus_id.nil?
+
+        (bars_by_corpus || {}).fetch(corpus_id, bar)
+      end
+    end
     GateSpec = Struct.new(:reference_arm, :requirements, :gates,
                           keyword_init: true)
     Outcome = Struct.new(:frame_id, :requires_masking, :matched, :expected,
@@ -110,8 +122,8 @@ module Vicary
           gates: raw.fetch("gates").map do |g|
             Gate.new(id: g.fetch("id"), label: g.fetch("label"),
                      unit: g.fetch("unit"), op: g.fetch("op"),
-                     bar: g.fetch("bar"), requires: g.fetch("requires"),
-                     why: g.fetch("why"))
+                     bar: g.fetch("bar"), bars_by_corpus: g["bars_by_corpus"],
+                     requires: g.fetch("requires"), why: g.fetch("why"))
           end,
         )
       end
@@ -189,9 +201,16 @@ module Vicary
       #
       # Leads with the masking-required ratio, the one a null implementation
       # cannot inflate. Gates print NOT MEASURED per gate rather than being
-      # reduced out of the denominator — five of nine held is a different
+      # reduced out of the denominator — eight of nine held is a different
       # statement from nine of nine, and a badge cannot tell them apart.
-      def report(board, gates)
+      #
+      # +gate_block+ is a rendered gate block from `gates.rb`. Passed in rather
+      # than computed here because measuring a gate needs the detector and the
+      # asset, and this module is the spec loader — requiring them would make the
+      # loader depend on the thing it exists to score. Absent, the unmeasured
+      # block below is printed, which is the honest output for a caller that
+      # measured nothing.
+      def report(board, gates, gate_block = nil)
         lines = []
         lines << "conformance — fixture #{board.fixture_version}, arm #{board.reference_arm}"
         lines << ("-" * 58)
@@ -202,14 +221,19 @@ module Vicary
                         board.matched, board.total,
                         board.total - board.requiring_masking)
         lines << ("-" * 58)
+        if gate_block
+          lines << gate_block
+          return lines.join("\n")
+        end
+
         lines << "  gates:"
         gates.gates.each do |gate|
           needs = gate.requires.empty? ? "" : "  NEEDS #{gate.requires.join('+')}"
           lines << format("    NOT MEASURED  %-28s %s %s %s%s",
                           gate.label, gate.op, gate.bar, gate.unit, needs)
         end
-        lines << "  -> no gate is measured by this port yet. A green run here " \
-                 "means the spec loads,"
+        lines << "  -> the caller measured no gate. A green run here means the " \
+                 "spec loads,"
         lines << "     never that the gate set is clear."
         lines.join("\n")
       end