vicary 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 030a648056336b3ad6905e692be55844a1a3fbffd16b73a5ef37bcfc275402cc
+  data.tar.gz: 3753c57622c1b2ab6dfb82fa12f79eb862681dcc8377a8fe01094e471d235ad7
+SHA512:
+  metadata.gz: 37aa4e0e3ccd3bb7860f97125862e1f7b3b30e63594549f201fec0f42bd92e4883e34a4b0c01cffe0f631e46f6576f4dbfc0e56f89e8d30df9e844575b512eb8
+  data.tar.gz: ae700f44fc829bb7431ffd26a20db4e3dc3c3e15ab0b4e047e5c57ff996b1ac701da0d9023ad2de6167b3047fff929469b948068c77421633ba8935a741a616f

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Blake Thomas
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,57 @@
+# vicary (Ruby)
+
+The RubyGems front door. **Not published yet** — the gem name is claimed by a
+pending trusted publisher and the first release has not run.
+
+The detector, the data asset and the measured numbers are described in the
+[project README](https://github.com/bwthomas/vicary#readme). What lives here is a
+port, and the bar it has to clear before it is published is the shared
+conformance suite in [`conformance/`](../conformance): for every fixture frame it
+must produce **byte-identical output to the Python implementation, placeholder
+numbering included**.
+
+It clears that bar — 36 of 36 masking-required frames, 52 of 52 overall.
+
+```ruby
+require "vicary"
+
+Identity = Struct.new(:first_name, :last_name, :school_name)
+identity = Identity.new("Marguerite", "Delacroix-Whitfield", "Westfield High School")
+
+Vicary.redact("My cousin Terrence Okonkwo came over that summer.", identity)
+# => "My cousin {NAME_1} came over that summer."
+
+masked, n, restore_map = Vicary.redact_with_report(essay, identity)
+Vicary.restore(masked, restore_map) == essay   # => true
+```
+
+## Checking it
+
+Three layers, because each catches what the one above it cannot.
+
+| command | what it says |
+|---|---|
+| `rake conformance` | the scoreboard against the 52 frames — the final bar, and a coarse first one |
+| `rake test` | the unit suites, including `primitives_test.rb`: forty-odd primitives over the shared corpus, which says *which brick* is crooked |
+| `rake parity` | gazetteer verdicts, name by name, against the Python reference |
+| `rake redaction_parity` | masked bytes against the Python reference, on prose no fixture contains |
+
+The last two need the reference interpreter — run `just py-setup` from the
+repository root first.
+
+**Why there are four and not one**, measured on the day the port landed: of
+eleven deliberate mutations to `candidates.rb`, the conformance frames caught
+**one**. The primitives spec caught seven. Three were inert. The last was a real
+divergence that both corpora were blind to, because both are single-line and the
+rule only differs across a newline — which is what `redaction_parity` and
+`test/dialect_test.rb` exist for.
+
+## Porting notes
+
+`lib/vicary/candidates.rb` opens with the regex-dialect differences between Ruby
+and Python that run through the detector. The short version: `^` and `$` mean
+*line* in Ruby and *string* in Python, so every one of them is written `\A`, `\z`
+or `\Z`; `\w`, `\d` and `\s` are ASCII-only in Ruby and Unicode-aware in Python;
+and `\b` — unlike JavaScript's — already agrees with Python, so the explicit
+lookarounds here are belt-and-braces rather than load-bearing.
+`test/dialect_test.rb` pins all of it in both directions.

data/assets/MANIFEST.json

@@ -0,0 +1,33 @@
+{
+  "assets": {
+    "notability.txt.gz": {
+      "bytes": 2234918,
+      "cut_date": "2026-08-07",
+      "format": 5,
+      "min_package_version": "0.1.0",
+      "sha256": "ea882c2ab1a80bd5f3b56e32a41e9c32a314ae44389f9596ea5e7ec013d92d69",
+      "sources": [
+        "https://qlever.dev/api/wikidata",
+        "https://www2.census.gov/topics/genealogy/2010surnames/names.zip"
+      ],
+      "tiers": {
+        "demonym": 1047,
+        "full": 295049,
+        "given": 8138,
+        "place": 25444,
+        "settlement": 23234,
+        "short": 1229,
+        "title": 38024
+      }
+    },
+    "stop_words.txt": {
+      "bytes": 3996,
+      "entries": 421,
+      "format": 1,
+      "min_package_version": "0.2.0",
+      "sha256": "f4dfe89490efe47c7520dac3b9185f39c0eeb397f3ab33f5d1c1211591dfcf30"
+    }
+  },
+  "manifest_version": 1,
+  "written_by": "vicary 0.2.0"
+}

data/assets/stop_words.txt

@@ -0,0 +1,63 @@
+#!lexicon 1
+#!list stop_words 421
+# Ordinary words that must never become name candidates.
+#
+# Language-neutral on purpose. This list is the only thing standing between
+# candidate generation and "mask every capitalised word", so all three front doors
+# have to work from the same 421 words — a stoplist transliterated by hand into a
+# second language is a second detector wearing the first one's name, and the
+# difference shows up as prose corruption in one language and not the others,
+# which no parity check on masked output would catch.
+#
+# Skewed toward over-inclusion deliberately. A missed name is one span and shows
+# up in the recall number; a wrongly-masked common word corrupts every essay that
+# uses it and shows up nowhere unless somebody reads the prose.
+#
+# Format: `#!` lines are directives, `#` lines are comments, and every other line
+# contributes whitespace-separated words. The count on the `#!list` directive is
+# the number of DISTINCT words after case-folding, and each front door asserts it
+# against what it parsed. That is not ceremony: a short read here silently makes
+# the redactor MORE aggressive, which looks privacy-safe and passes any check that
+# only asks whether something was masked.
+#
+# A word appearing twice is not an error — the groupings below are thematic and
+# overlap ("else", "may", "us"), and enforcing uniqueness in the source would make
+# the list harder to read for no benefit.
+  a an the this that these those there here it its it's
+  i me my mine myself we us our ours ourselves you your yours
+  he him his she her hers they them their theirs who whom whose which what
+  and or but so because although though however therefore thus hence yet
+  if then else when while until since before after during once whenever
+  for from to into onto out off over under above below between among across
+  through around about against along beside besides beyond within without
+  at by in on up down near next last first second third finally
+  is am are was were be been being have has had having do does did doing
+  can could will would shall should may might must let lets
+  not no nor none nothing never always sometimes often usually rarely
+  all any both each every few many more most much several some such
+  another other others same different new old good bad better best worst
+  great big small long short high low young happy sad hard easy
+  one two three four five six seven eight nine ten hundred thousand million
+  also even just only really very too still again ever else quite rather
+  call called come came go went get got give gave take took make made
+  see saw look looked think thought know knew say said tell told ask asked
+  want wanted need needed try tried help helped work worked feel felt
+  find found keep kept leave left put set start started stop stopped
+  remember remembered learn learned teach taught write wrote read
+  everyone everybody someone somebody anyone anybody nobody everything
+  something anything people person thing things time times day days
+  year years week weeks month months hour hours minute minutes
+  school schools class classes teacher teachers student students friend
+  friends family families home house mom dad mother father parent parents
+  brother sister sisters brothers grandma grandpa
+  life world way ways place places part parts kind sort lot lots
+  yes yeah ok okay maybe perhaps well now today tomorrow yesterday
+  january february march april may june july august september october
+  november december monday tuesday wednesday thursday friday saturday sunday
+  mr mrs ms dr am pm usa us u.s tv
+  im ive ill id dont cant wont didnt isnt aint thats theres whats
+  as than instead unless whether either neither plus versus etc
+  getting making looking thinking talking playing living walking running
+  sitting standing growing learning moving trying using
+  back away together alone everywhere somewhere anywhere nowhere
+  right wrong true false sure certain important special favorite

data/lib/vicary/asset.rb

@@ -0,0 +1,190 @@
+# frozen_string_literal: true
+
+require "digest"
+require "json"
+require "pathname"
+require "set"
+require "zlib"
+
+module Vicary
+  # Load the gazetteer asset — the same bytes the Python package loads.
+  #
+  # The asset is a gzipped, line-oriented text file, chosen over a binary format
+  # precisely so three languages can read it without a schema compiler:
+  #
+  #     #!gazetteer 5                 format number, checked not sniffed
+  #     #!meta {"cut_date": ...}      provenance, one JSON object
+  #     #!tier demonym 1047           tier name and its DECLARED entry count
+  #     abidjanese                    one normalised entry per line
+  #
+  # Two properties matter more than convenience.
+  #
+  # **The format number is refused, not tolerated.** An unknown format means the
+  # file's meaning changed, and a reader that skips lines it does not recognise
+  # degrades into a smaller gazetteer — which redacts MORE, reads as
+  # privacy-safe, and is invisible to any test that only asks whether something
+  # was masked.
+  #
+  # **The declared tier count is checked against the parsed count.** A truncated
+  # read is the same silent failure in a different costume: fewer notable people
+  # means fewer public figures kept, so an essay about Rosa Parks comes back with
+  # her name removed.
+  module Asset
+    # Asset format this reader understands. Refuse anything else.
+    SUPPORTED_FORMAT = 5
+
+    ASSET_FILENAME = "notability.txt.gz"
+    MANIFEST_FILENAME = "MANIFEST.json"
+
+    # Environment override, spelled the same as the Python package's.
+    ASSET_PATH_ENV_VAR = "VICARY_ASSET_PATH"
+
+    Gazetteer = Struct.new(:format, :meta, :tiers, :sha256, :path,
+                           keyword_init: true)
+
+    class FormatError < StandardError; end
+    class MissingAssetError < StandardError; end
+
+    class << self
+      # Candidate asset locations, most specific first.
+      #
+      # The env override first, so an operator can point at a different cut
+      # without reinstalling. Then this gem's vendored copy, which is what an
+      # installed gem has. Then the monorepo's Python package, which is what a
+      # checkout has before `rake sync_assets` — so `git clone && rake test`
+      # works with no bootstrap step rather than failing in a way that reads as
+      # a broken port.
+      def search_path
+        gem_root = Pathname.new(__dir__).join("..", "..").expand_path
+        repo_root = gem_root.join("..").expand_path
+        candidates = []
+        override = ENV.fetch(ASSET_PATH_ENV_VAR, "").strip
+        candidates << Pathname.new(override) unless override.empty?
+        candidates << gem_root.join("assets")
+        candidates << repo_root.join("python", "src", "vicary", "data")
+        candidates
+      end
+
+      def locate
+        tried = search_path
+        found = tried.find { |dir| dir.join(ASSET_FILENAME).file? }
+        return found if found
+
+        raise MissingAssetError,
+              "no #{ASSET_FILENAME} found. Looked in: " \
+              "#{tried.join(', ')}. In a checkout, run `rake sync_assets`; set " \
+              "#{ASSET_PATH_ENV_VAR} to override."
+      end
+
+      # Parse the decompressed asset text.
+      #
+      # Public so a test can feed it a deliberately malformed document. A parser
+      # reachable only through a 2.1 MB file on disk is a parser whose failure
+      # paths are never exercised.
+      def parse(text)
+        lines = text.split("\n", -1)
+        header = /\A\#!gazetteer (\d+)\z/.match(lines.first.to_s)
+        unless header
+          raise FormatError,
+                "asset does not begin with a \#!gazetteer header (got " \
+                "#{lines.first.to_s[0, 40].inspect})"
+        end
+
+        format = header[1].to_i
+        unless format == SUPPORTED_FORMAT
+          raise FormatError,
+                "asset format #{format} is not #{SUPPORTED_FORMAT}. Refusing to " \
+                "read it rather than skipping the parts that changed: a " \
+                "partially understood gazetteer is a smaller one, and a smaller " \
+                "one redacts more while looking correct."
+        end
+
+        meta = {}
+        tiers = {}
+        declared = {}
+        current = nil
+
+        lines.each_with_index do |line, index|
+          next if index.zero? || line.empty?
+
+          if line.start_with?("#!meta ")
+            meta = JSON.parse(line.delete_prefix("#!meta "))
+            next
+          end
+
+          if (tier = /\A\#!tier (\S+) (\d+)\z/.match(line))
+            current = Set.new
+            tiers[tier[1]] = current
+            declared[tier[1]] = tier[2].to_i
+            next
+          end
+
+          if line.start_with?("#!")
+            raise FormatError,
+                  "unrecognised directive #{line[0, 40].inspect} at line " \
+                  "#{index + 1}; the asset format changed without its number " \
+                  "changing"
+          end
+
+          raise FormatError, "entry at line #{index + 1} appears before any \#!tier" if current.nil?
+
+          current << line
+        end
+
+        declared.each do |name, count|
+          actual = tiers.fetch(name).size
+          next if actual == count
+
+          raise FormatError,
+                "tier #{name} declares #{count} entries and parsed #{actual}. A " \
+                "short read here removes public figures from the keep list, so " \
+                "an essay about a historical figure comes back with their name " \
+                "redacted."
+        end
+
+        [format, meta, tiers]
+      end
+
+      # Load and cache the gazetteer.
+      def load(directory: nil)
+        return @cached if @cached && directory.nil?
+
+        dir = Pathname.new(directory || locate)
+        asset_path = dir.join(ASSET_FILENAME)
+        compressed = asset_path.binread
+        sha256 = Digest::SHA256.hexdigest(compressed)
+
+        verify_against_manifest(dir, asset_path, sha256)
+
+        format, meta, tiers = parse(Zlib::GzipReader.new(StringIO.new(compressed)).read)
+        gazetteer = Gazetteer.new(format: format, meta: meta, tiers: tiers,
+                                  sha256: sha256, path: asset_path.to_s)
+        @cached = gazetteer if directory.nil?
+        gazetteer
+      end
+
+      # Forget the cached gazetteer. For tests.
+      def reset_cache
+        @cached = nil
+      end
+
+      private
+
+      def verify_against_manifest(dir, asset_path, sha256)
+        manifest_path = dir.join(MANIFEST_FILENAME)
+        return unless manifest_path.file?
+
+        entry = JSON.parse(manifest_path.read).dig("assets", ASSET_FILENAME)
+        return if entry.nil? || entry["sha256"] == sha256
+
+        raise FormatError,
+              "#{asset_path} sha256 #{sha256} does not match the manifest's " \
+              "#{entry['sha256']}. The asset was modified or truncated in " \
+              "transit; every front door must load identical bytes or " \
+              "\"byte-identical output\" is not a claim anybody can make."
+      end
+    end
+  end
+end
+
+require "stringio"