vicary 0.2.8 → 0.2.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 70cca39c8c91256c36ed5952eb12710da4125bd605f65dc4c1c71de8e0d77395
-  data.tar.gz: 879ab66dc00f074accad84a80887ac48a93fc9fe09968bc04f819bee1e0e20a8
+  metadata.gz: 2417f5e9aa64949da91c1ca4cf5cc1dac1b8a4bdc70929576bbc7b64d8e629f9
+  data.tar.gz: 67cd930a6714621ed7872054fac572b87a210940b1644032a845b16d2b2bbfb3
 SHA512:
-  metadata.gz: ecf9bbecf10214d42fc5e495d6b3b161f7d1f1f1a080fb13631c379f06dd609ec182065685060d63f733890c468eacb0b854e371a2fec6b3fdb5bebdd9f45f03
-  data.tar.gz: 704e8aaf854c08b3128299dc425e543c00cf3789909dd98ccbb13509d9c80020d72d991a8e4c690424f9508ba8dfef6a5c9da1c9ca8f513a642407a33bfdf5b5
+  metadata.gz: dfe4bc4a8eb71174d969e41f270eadf8c7170a97e54870fc27b8f2610a8ce2e6b976e39cba2ed87435652d8ca366ca4fb9d90795f773bd8b20db6017a3bce99b
+  data.tar.gz: 3bef064a891171f7668d8773e813c6efedea6f25385c5da72a4141279662838b623bcefe4a06de28866efdf7f53b04051fa74a02ad6f9c35acc70b9eba1b0f9a

data/README.md

@@ -69,7 +69,7 @@ gzip, which `zlib` reads, so that hazard does not arise on the default path.
 
 Two of the last three read the corpus the repository now ships in
 `conformance/corpora/`, so they measure on a bare checkout with no environment
-set: 100% carrier recall and 8.150 over-fired spans per essay against a ≤ 8.15
+set: 100% carrier recall and 7.400 over-fired spans per essay against a ≤ 7.40
 bar, identical to Python and TypeScript. `VICARY_EVAL_CORPUS_TSV` is an override
 for a different corpus, not a requirement.
 

data/assets/MANIFEST.json

@@ -20,14 +20,21 @@
         "title": 38024
       }
     },
-    "stop_words.txt": {
-      "bytes": 3996,
-      "entries": 421,
+    "stop_words_never_capitalised.txt": {
+      "bytes": 8017,
+      "entries": 499,
       "format": 1,
-      "min_package_version": "0.2.0",
-      "sha256": "f4dfe89490efe47c7520dac3b9185f39c0eeb397f3ab33f5d1c1211591dfcf30"
+      "min_package_version": "0.2.10",
+      "sha256": "23798cb88672c126d71a9d0988b9adf0aa044bdc291847012d76deae71fe1d33"
+    },
+    "stop_words_sometimes_capitalised.txt": {
+      "bytes": 6351,
+      "entries": 295,
+      "format": 1,
+      "min_package_version": "0.2.10",
+      "sha256": "d6c842a9eeaaa74d739a9daea2439d2f69f8b97e7e87157caafd685fd11a32c8"
     }
   },
   "manifest_version": 1,
-  "written_by": "vicary 0.2.0"
+  "written_by": "vicary 0.2.10"
 }

data/assets/stop_words_never_capitalised.txt

@@ -0,0 +1,126 @@
+#!lexicon 1
+#!list stop_words_never_capitalised 499
+# Ordinary words that must never become name candidates, and that English never
+# capitalises in the middle of a sentence.
+#
+# HALF A LIST. The stoplist every front door vetoes candidates against is this
+# file UNION `stop_words_sometimes_capitalised.txt`, and neither half is the
+# stoplist on its own. Loading only one under-vetoes, which makes the redactor
+# more aggressive — the failure that looks privacy-safe and corrupts prose. The
+# `#!list` count on each file is per-file; the union is what code must build.
+#
+# Why there are two files. One consumer asks a second question of these words:
+# `capitalises_ordinary_words` decides a document's capitals are untrustworthy
+# when it finds a mid-sentence capital on a stop word, on the premise that a
+# capital there was never orthographic. That premise is true of the words below
+# and false of the words in the sibling file — "in July" and "my Dad" are
+# correctly capitalised English, and reading them as sloppiness suppressed
+# `Alvarez` in "We stayed with the Alvarez family in July." So the *veto* wants
+# every stop word and the *signal* wants only these, and one list could not
+# serve both.
+#
+# The line between the two files is orthographic, not thematic, because a
+# thematic line is an argument every new word reopens:
+#
+#   here    function words, verbs, adverbs, interjections and contractions —
+#           a capital on one mid-sentence is a mistake, never a construction;
+#   sibling nouns, adjectives, numerals, and the proper-noun categories —
+#           English capitalises these inside proper names ("Lincoln School",
+#           "Great Lakes", "First Baptist"), so a capital is uninformative.
+#
+# Language-neutral on purpose. This list is the only thing standing between
+# candidate generation and "mask every capitalised word", so all three front doors
+# have to work from the same words — a stoplist transliterated by hand into a
+# second language is a second detector wearing the first one's name, and the
+# difference shows up as prose corruption in one language and not the others,
+# which no parity check on masked output would catch.
+#
+# Skewed toward over-inclusion deliberately. A missed name is one span and shows
+# up in the recall number; a wrongly-masked common word corrupts every essay that
+# uses it and shows up nowhere unless somebody reads the prose.
+#
+# Format: `#!` lines are directives, `#` lines are comments, and every other line
+# contributes whitespace-separated words. The count on the `#!list` directive is
+# the number of DISTINCT words after case-folding — authored and generated
+# together, since a reader loads one file — and each front door asserts it
+# against what it parsed. That is not ceremony: a short read here silently makes
+# the redactor MORE aggressive, which looks privacy-safe and passes any check that
+# only asks whether something was masked.
+#
+# A word appearing twice is not an error — the groupings below are thematic and
+# overlap ("else", "may", "us"), and enforcing uniqueness in the source would make
+# the list harder to read for no benefit.
+#
+# Write the SINGULAR only. Plurals are generated into the region at the bottom by
+# `python -m vicary_build lexicon`, which is also what stops one from being added
+# without being priced: the generator drops any form an American bears as a
+# surname, and a hand-written plural is not priced by anything.
+  a an the this that these those there here it its it's
+  i me my mine myself we us our ours ourselves you your yours
+  he him his she her hers they them their theirs who whom whose which what
+  and or but so because although though however therefore thus hence yet
+  if then else when while until since before after during once whenever
+  for from to into onto out off over under above below between among across
+  through around about against along beside beyond within without
+  at by in on up down near finally
+  is am are was were be been being have has had having do does did doing
+  can could will would shall should might must let
+  not no nor none nothing never always sometimes often usually rarely
+  all any both each every few many more most much several some such
+  also even just only really very too still again ever else quite rather
+  call called come came go went get got give gave take took make made
+  see saw look looked think thought know knew say said tell told ask asked
+  want wanted need needed try tried help helped work worked feel felt
+  find found keep kept leave left put set start started stop stopped
+  remember remembered learn learned teach taught write wrote read
+  everyone everybody someone somebody anyone anybody nobody everything
+  something anything
+  yes yeah ok okay maybe perhaps well now today tomorrow yesterday
+  im ive ill id dont cant wont didnt isnt aint thats theres whats
+  as than instead unless whether either neither plus versus etc
+  getting making looking thinking talking playing living walking running
+  sitting standing growing learning moving trying using
+  back away together alone everywhere somewhere anywhere nowhere
+
+# `may`, `march` and `august` are ordinary words too, and they are NOT here:
+# they live in the sibling file, because a mid-sentence `May` is a month far
+# more often than it is a modal. The veto is the union, so candidate generation
+# is unchanged either way; what moves is whether the capital counts as evidence
+# that the writer is sloppy, and on a month it does not.
+
+# >>> generated inflections — written by `python -m vicary_build lexicon`
+# Bare plurals of the words above, so a capitalised `Sets` or `Parties` is
+# vetoed by the same lookup its singular is. Hand-written pairs used to do this
+# and only for the 35 somebody thought of.
+#
+# Two subtractions are applied here and neither is optional. A form borne as an
+# American surname is dropped, because a stop word puts every family bearing it
+# beyond the redactor (`may` would claim `Mays`, `will` would claim `Wills`).
+# A form that is a common given name is dropped for a sharper reason: a stop
+# word wins over the given-name tier, so `we` -> `wes` would stop redacting a
+# child called Wes. Plurals still written out by hand above are the ones a
+# subtraction removes from here — they are load-bearing, not leftovers.
+#
+# Regenerate with `python -m vicary_build lexicon`; do not edit below by hand.
+  abouts aboves acrosses afters agains againsts aints alones alongs alsos
+  althoughs alwayses amongs ands anies ans anybodies anyones anythings
+  anywheres arounds ases askeds asks ats aways becauses beens befores beings
+  belows bes besides betweens beyonds boths buts calleds calls cames cans
+  cants coulds didnts dids doeses doings donts durings eithers elses etcs
+  everies everybodies everyones everythings everywheres feels fews finallies
+  finds froms gaves gets gives gos gots growings hads hases havings helpeds
+  hences hims hises howevers ids ifs ills ins insteads intos ises isnts justs
+  keeps kepts knews knows learneds learnings learns leaves lefts lets lookeds
+  lookings looks makes maybes mights mosts movings muches musts myselfs
+  neededs neithers nobodies nos nothings nots nowheres nows offs oftens okays
+  onces onlies ons ontos ourselveses outs perhapses playings pluses puts
+  quites rarelies rathers reads reallies remembereds remembers runnings saids
+  saws says sets severals shalls shes shoulds sinces sittings somebodies
+  someones somethings sometimeses somewheres standings starteds stoppeds
+  suches talkings taughts teaches tells thans thems thens therefores thes
+  theses thinkings thinks thises thoses thoughs thoughts throughs thuses
+  todays togethers tolds tomorrows toos trieds tries tryings unders unlesses
+  untils ups uses usings usuallies veries versuses walkings wanteds wants
+  wases wents weres whenevers whens whethers whiches whoms whos whoses withins
+  withouts wonts workeds woulds writes wrotes yeahs yeses yesterdays yets yous
+# <<< end generated inflections

data/assets/stop_words_sometimes_capitalised.txt

@@ -0,0 +1,109 @@
+#!lexicon 1
+#!list stop_words_sometimes_capitalised 295
+# Ordinary words that must never become name candidates, but that correct
+# English does capitalise mid-sentence.
+#
+# HALF A LIST. The stoplist every front door vetoes candidates against is this
+# file UNION `stop_words_never_capitalised.txt`, and neither half is the
+# stoplist on its own. Read that file's header for the split and the leak that
+# forced it; in one line: the veto wants every stop word, and the
+# sloppy-capitaliser signal wants only the never-capitalised half, because
+# "in July" and "my Dad" are correct English and told us nothing about the
+# writer.
+#
+# Three kinds of word are here, and the test for all three is the same — is
+# there a correct English sentence with this word capitalised in the middle of
+# it?
+#
+# * **Nouns and adjectives and numerals.** English capitalises them inside
+#   proper names: "Lincoln School", "Great Lakes", "First Baptist Church",
+#   "New Year's Day". `School` fired the signal on four NWP papers and every
+#   one of them was a school's name.
+# * **Months, weekdays and honorifics.** Always capitalised. `July` is what
+#   suppressed `Alvarez` in "We stayed with the Alvarez family in July."
+# * **Named categories that identify nobody** — holidays and observances,
+#   languages, nationalities and ethnic groups, religions and their scriptures.
+#   These are on the stoplist at all because `English` was masked in a 34-word
+#   paper by an English-language learner writing about learning English.
+#
+# Two rules govern what may be added to the named-category block, and both are
+# load-bearing.
+#
+# 1. Nothing that `is_common_given_name` already carries. Easter, Valentine,
+#    German, Latina, Christian, Jesus and Quran were all proposed and all
+#    excluded on that test: a stop word wins over the given-name tier, so
+#    listing one of those would stop redacting a real child called Christian.
+# 2. Priced before adding, not after. The 50 category words below claim 138,506
+#    American surname bearers between them — 0.072% of the census population,
+#    against the 1.25% ceiling the bare-surname exposure gate enforces. French
+#    (63,991) and English (46,393) are most of it. Measured 2026-09-10;
+#    re-measure before extending, because the cheap-looking additions are the
+#    common ones.
+#
+# Scope of that block is deliberately narrow. Public FIGURES, PLACES and WORKS
+# belong in the gazetteer, which already carries `Che Guevara` and `Australia`;
+# settlements are excluded from it on purpose and stay excluded. This list is
+# only for the categories no gazetteer tier models at all.
+#
+# Same format and the same bias as the sibling file: singulars only, plurals
+# generated at the bottom by `python -m vicary_build lexicon`, count asserted on
+# read.
+  another other same different new old good bad better best worst
+  great big small long short high low young happy sad hard easy
+  one two three four five six seven eight nine ten hundred thousand million
+  next last first second third
+  right wrong true false sure certain important special favorite
+  people person thing time day year week month hour minute
+  school class teacher student friend family home house mom dad mother father
+  parent brother sister grandma grandpa
+  life world way place part kind sort lot
+  january february march april may june july august september october
+  november december monday tuesday wednesday thursday friday saturday sunday
+  mr mrs ms dr pm usa u.s tv
+
+# Plurals the generated region below cannot carry. Each is also an American
+# surname, so the census veto drops it — and a stop word that quietly stops
+# existing is a recall regression with no diff to read. Written out by hand for
+# that reason, not as a leftover: check
+# `test_the_built_list_still_carries_every_word_it_used_to` before touching one.
+  brothers days friends schools times ways weeks years
+
+# The named categories. See rules 1 and 2 in the header before adding one.
+  christmas halloween thanksgiving hanukkah passover ramadan eid diwali
+  kwanzaa valentines
+  english spanish french chinese japanese korean italian russian arabic
+  hindi portuguese vietnamese greek polish irish scottish
+  hispanic latino asian african american european mexican
+  canadian indian native jewish muslim catholic protestant buddhist
+  hindu nazi
+  god bible koran torah allah
+
+# >>> generated inflections — written by `python -m vicary_build lexicon`
+# Bare plurals of the words above, so a capitalised `Sets` or `Parties` is
+# vetoed by the same lookup its singular is. Hand-written pairs used to do this
+# and only for the 35 somebody thought of.
+#
+# Two subtractions are applied here and neither is optional. A form borne as an
+# American surname is dropped, because a stop word puts every family bearing it
+# beyond the redactor (`may` would claim `Mays`, `will` would claim `Wills`).
+# A form that is a common given name is dropped for a sharper reason: a stop
+# word wins over the given-name tier, so `we` -> `wes` would stop redacting a
+# child called Wes. Plurals still written out by hand above are the ones a
+# subtraction removes from here — they are load-bearing, not leftovers.
+#
+# Regenerate with `python -m vicary_build lexicon`; do not edit below by hand.
+  africans allahs americans anothers aprils arabics asians augusts bads bests
+  bigs buddhists canadians catholics certains chineses christmases classes
+  dads decembers differents diwalis drs easies eids eights englishes europeans
+  falses families fathers favorites februaries firsts fours frenches fridays
+  gods grandmas grandpas greats greeks halloweens hanukkahs happies highs
+  hindis hindus hispanics hours houses hundreds importants indians irishes
+  italians januaries japaneses jewishes julies korans koreans kwanzaas lasts
+  latinos lifes lots marches mexicans millions minutes moms mondays months
+  mothers mses muslims natives nazis nexts novembers octobers others parents
+  parts passovers places pms polishes portugueses protestants ramadans rights
+  russians sads saturdays scottishes seconds septembers sevens sisters sixes
+  sorts spanishes specials students sundays sures teachers tens thanksgivings
+  things thirds thousands threes thursdays torahs trues tuesdays tvs twos usas
+  valentineses vietnameses wednesdays worsts wrongs
+# <<< end generated inflections

data/lib/vicary/candidates.rb

@@ -119,8 +119,30 @@ module Vicary
     # load-at-import is that a host may `require "vicary"` to read
     # {Vicary::VERSION} without a vendored asset, and raising there would fail a
     # program that never redacts anything.
+    #
+    # **The union of two files, and every veto must use the union.** The split is
+    # not thematic: `stop_words_never_capitalised` holds the words a mid-sentence
+    # capital is a *mistake* on, and `stop_words_sometimes_capitalised` the ones
+    # correct English capitalises inside a proper name ("Lincoln School"), in a
+    # date ("in July") or as a nationality. Only a document-level *signal* needs
+    # that distinction; candidate generation needs all 794 and always did.
     def self.stop_words
-      @stop_words ||= Lexicon.load("stop_words")
+      @stop_words ||= Lexicon.stop_words
+    end
+
+    # The half of the stoplist a mid-sentence capital is never orthographic on.
+    # Its own list because reading the whole stoplist for that question counts
+    # "in July" as evidence that the writer capitalises sloppily, which it is
+    # not — see the file header in `asset/lexicon/`.
+    def self.never_capitalised
+      @never_capitalised ||= Lexicon.load(Lexicon::NEVER_CAPITALISED)
+    end
+
+    # The other half: stop words English does capitalise mid-sentence — months,
+    # weekdays, honorifics, nationalities, and the nouns and adjectives that sit
+    # inside proper names.
+    def self.sometimes_capitalised
+      @sometimes_capitalised ||= Lexicon.load(Lexicon::SOMETIMES_CAPITALISED)
     end
 
     # Contraction and possessive tails. `[A-Z][A-Za-z'’]*` matches "I'm" as one
@@ -383,6 +405,14 @@ module Vicary
     # document level, and that is what `INCONSISTENT` is for.
     MARKS_PROPER_NOUNS_MIN = 2
 
+    # How many ordinary words a document must capitalise mid-sentence before its
+    # capitals stop counting as testimony. One is enough, and the reason it is
+    # not a rate: a word on the never-capitalised half of the stoplist is not a
+    # name AND is not a word English capitalises, so a capital on one is neither
+    # orthographic nor ambiguous. A floor of one is only defensible while that
+    # list stays that clean, which is what the sibling lexicon is for.
+    STRAY_CAPITALS_MIN = 1
+
     # A sentence opening on a lower-case letter, which is the writer telling us
     # directly that they are not keeping standard capitalisation. Matched at the
     # start of the text as well as after a sentence break.
@@ -645,9 +675,32 @@ module Vicary
       end
 
       # Whether `token` is an ordinary word that must never become a candidate.
+      #
+      # The one place inflection is folded at runtime, and it folds exactly one
+      # thing: a trailing clitic, so `Nazi's` is looked up as `nazi`. Plurals are
+      # NOT folded here — they are written into the asset by
+      # `python -m vicary_build lexicon`, which drops any form an American bears
+      # as a surname before emitting it, so all three front doors inherit the
+      # behaviour from the same bytes instead of implementing it three times.
       def stop?(token)
-        word = without_clitic(strip(token.downcase, ".,"))
-        stop_words.include?(strip(word, "'’"))
+        stop_words.include?(fold(token))
+      end
+
+      # `token` as the stoplists are keyed: lower-cased, clitic and edge
+      # punctuation off. Extracted so the two lookups cannot fold differently.
+      def fold(token)
+        strip(without_clitic(strip(token.downcase, ".,")), "'’")
+      end
+
+      # Whether a capital on `token` mid-sentence is a mistake rather than a
+      # construction — the narrow half of the stoplist.
+      #
+      # The question {.capitalises_ordinary_words?} needs and {.stop?} does not
+      # answer: every word here is a stop word, but a stop word may be a month, a
+      # nationality or the noun in "Lincoln School", and English capitalises all
+      # three correctly.
+      def never_capitalised?(token)
+        never_capitalised.include?(fold(token))
       end
 
       # Words split on whitespace, the way Python's bare `str.split()` does.
@@ -1095,6 +1148,82 @@ module Vicary
           !corroborated?(tokens, written_as_a_capital, is_given)
       end
 
+      # Whether this document capitalises words that cannot be names.
+      #
+      # Counts mid-sentence capitals landing on the **never-capitalised** half of
+      # the stoplist, which is a narrower question than {.stop?} answers and the
+      # reason that list is a file of its own.
+      #
+      # It used to read the whole stoplist, and that was wrong in a way no
+      # threshold fixes. The stoplist carries months, weekdays, honorifics,
+      # nationalities, religions and ordinary nouns — `English` is on it because
+      # a 34-word paper by an English-language learner had the word masked — and
+      # every one of those is *correctly* capitalised. A document writing "in
+      # July" tripped this having told us nothing, which suppressed `Alvarez` in
+      # "We stayed with the Alvarez family in July."
+      #
+      # Unlike `marks_proper_nouns?`, which counts every mid-sentence capital and
+      # therefore counts the names too, this cannot be satisfied by a document
+      # that simply names a lot of people. Headings are excluded because title
+      # case capitalises every word in one.
+      def capitalises_ordinary_words?(text, headings = [])
+        count = 0
+        each_match(text, MID_SENTENCE_CAP) do |m|
+          word = m[1]
+          next unless never_capitalised?(word)
+
+          start = m.begin(1)
+          finish = start + word.length
+          next if headings.any? { |h_start, h_end| start < h_end && finish > h_start }
+
+          count += 1
+        end
+        count >= STRAY_CAPITALS_MIN
+      end
+
+      # The mid-sentence guard: drop a lone capital a sloppy capitaliser chose.
+      #
+      # The mirror of {.suppressed_as_an_unevidenced_capital?}, and it exists
+      # because that rule guards the smaller hole. A sentence-initial capital is
+      # orthographically required, so it proves nothing — that is the rule
+      # already there, and on the NWP corpus it is **4 of 66** false-positive
+      # spans. A *mid-sentence* capital was taken as sufficient evidence on its
+      # own, and that is **41 of 66**.
+      #
+      # So a mid-sentence capital stops being self-sufficient, but **only in a
+      # document that has shown its capitals are unreliable** — see
+      # {.capitalises_ordinary_words?}. In every other document nothing changes.
+      #
+      # Two channels can still keep the span, and the document's own
+      # capitalisation is deliberately not one of them: `written_as_a_capital`
+      # *is* the mid-sentence capital, so consulting it here would be the span
+      # vouching for itself. What is left is evidence from outside the document —
+      # the given-name tier, and a first-person relation in the local context
+      # ("my friend Cade"), which is testimony that the token names a person
+      # whatever its case.
+      #
+      # **The measured weakness, which is an equity exposure and not just a
+      # coverage one.** Of the true catches this leaves intact on the NWP corpus,
+      # three — `Amy`, `Barry`, `Whitney` — survive on the given-name tier alone.
+      # Given-name coverage is systematically thinner for less common and
+      # non-Anglo names, so the population this rule is most likely to leak is
+      # not a random sample of children.
+      def suppressed_as_a_stray_mid_sentence_capital?(tokens, start, finish, text, starts,
+                                                      is_given)
+        return false unless tokens.length == 1
+        # A sentence-initial capital belongs to the other rule, which already
+        # consults a channel this one must not.
+        return false if starts.any? { |s| s <= start && start <= s + 2 }
+
+        stripped = strip(tokens[0].downcase, ".,'’")
+        return false if is_given.call(stripped)
+
+        folded = without_clitic(stripped)
+        return false if folded != stripped && is_given.call(folded)
+
+        !names_someone_in_the_writers_life?(text, start, finish)
+      end
+
       # ---------------------------------------------------------------------
       # Titles, and the relations that override them
       # ---------------------------------------------------------------------
@@ -1576,6 +1705,7 @@ module Vicary
         settlement = options[:settlement]
         headings_are_orthographic = options.fetch(:headings_are_orthographic, true)
         title_relation_refusal = options.fetch(:title_relation_refusal, true)
+        mid_sentence_corroboration = options.fetch(:mid_sentence_corroboration, true)
 
         blocked = each_match(text, PROTECTED).map { |m| [m.begin(0), m.begin(0) + m[0].length] }
         starts = sentence_starts(text)
@@ -1610,6 +1740,10 @@ module Vicary
         end
 
         written_as_a_capital = mid_sentence_capitals(text, starts, headings)
+        # A property of the whole document, read once, for the same reason
+        # `habit` is: two call sites computing it separately could disagree.
+        stray_capitals = mid_sentence_corroboration && !given_name.nil? &&
+                         capitalises_ordinary_words?(text, headings)
 
         out = []
         each_match(text, CANDIDATE_RE) do |m|
@@ -1638,6 +1772,15 @@ module Vicary
               next
             end
 
+            # ...and the other half of the same question, for the capital the
+            # rule above reads as evidence. Only in a document that has shown its
+            # capitals are worth less than that.
+            if stray_capitals &&
+               suppressed_as_a_stray_mid_sentence_capital?(run, start, start + joined.length,
+                                                           text, starts, given_name)
+              next
+            end
+
             # A *trailing* apostrophe is the closing quote, not part of the name.
             # The candidate pattern treats `'` as a name character so O'Brien
             # survives, which also means "words like 'Terrence'" arrives as

data/lib/vicary/corpus.rb

@@ -104,6 +104,10 @@ module Vicary
 
     Metrics = Struct.new(
       :essays, :recall_held_out, :recall_held_out_passed, :recall_held_out_total,
+      # Every REDACT span, not only the held-out ones. Held-out recall is blind
+      # to a span the detector was SHOWN, and a suppression rule that drops one
+      # leaves held-out recall at 100%.
+      :recall_all, :recall_all_passed, :recall_all_total,
       :over_fire_spans_per_essay, :over_fire_spans_total,
       :asap_rewrites_per_essay, :latency_p50_ms, :latency_p95_ms,
       :latency_pooled_median_ms,
@@ -485,6 +489,8 @@ module Vicary
 
         held_out = outcomes.select { |o| o.held_out && o.verdict != "keep" }
         passed = held_out.count(&:passed)
+        all_redact = outcomes.reject { |o| o.verdict == "keep" }
+        all_passed = all_redact.count(&:passed)
         sorted = latencies.sort
         at = lambda do |q|
           next 0.0 if sorted.empty?
@@ -497,6 +503,9 @@ module Vicary
           recall_held_out: held_out.empty? ? 0.0 : 100.0 * passed / held_out.size,
           recall_held_out_passed: passed,
           recall_held_out_total: held_out.size,
+          recall_all: all_redact.empty? ? 0.0 : 100.0 * all_passed / all_redact.size,
+          recall_all_passed: all_passed,
+          recall_all_total: all_redact.size,
           over_fire_spans_per_essay: cases.empty? ? 0.0 : over_fire.to_f / cases.size,
           over_fire_spans_total: over_fire,
           asap_rewrites_per_essay: cases.empty? ? 0.0 : rewrites.to_f / cases.size,

data/lib/vicary/gates.rb

@@ -294,7 +294,8 @@ module Vicary
       # a caller that wants those gates supplies the number, and one that does
       # not gets NOT MEASURED rather than a load.
       def measure(spec, gate_spec, asset_entries: nil, bare_surname_exposure: nil,
-                  held_out_recall_carrier: nil, over_fire_per_essay: nil,
+                  held_out_recall_carrier: nil, all_span_recall_carrier: nil,
+                  over_fire_per_essay: nil,
                   latency_regression_pct: nil, latency_regression_detail: nil,
                   corpus_id: nil)
         outcomes = []
@@ -365,6 +366,14 @@ module Vicary
                       "#{round3(held_out_recall_carrier)}% of held-out REDACT spans in carrier essays"
                     end,
           },
+          "all_span_recall_carrier" => {
+            value: all_span_recall_carrier,
+            detail: if all_span_recall_carrier.nil?
+                      no_corpus
+                    else
+                      "#{round3(all_span_recall_carrier)}% of ALL REDACT spans in carrier essays"
+                    end,
+          },
           "over_fire_prose" => {
             value: over_fire_per_essay,
             detail: if over_fire_per_essay.nil?

data/lib/vicary/lexicon.rb

@@ -46,6 +46,25 @@ module Vicary
     # Filename suffix. Named so a second list costs a file rather than a refactor.
     SUFFIX = ".txt"
 
+    # The two files whose union is the stoplist. Named here rather than at each
+    # call site because the *veto* needs both and exactly one consumer needs one
+    # — a document-level signal that reads a mid-sentence capital as testimony
+    # about the writer, and that must not read "in July" that way. A call site
+    # free to load one half is free to narrow the veto by accident, which makes
+    # the redactor more aggressive: privacy-safe to look at, prose-corrupting in
+    # fact, and invisible to any check that only asks whether something was
+    # masked.
+    NEVER_CAPITALISED = "stop_words_never_capitalised"
+    SOMETIMES_CAPITALISED = "stop_words_sometimes_capitalised"
+    STOP_WORD_LISTS = [NEVER_CAPITALISED, SOMETIMES_CAPITALISED].freeze
+
+    # Every word that must never become a name candidate: both halves, unioned.
+    # Word for word the list that shipped as a single `stop_words.txt` through
+    # 0.2.9.
+    def self.stop_words
+      STOP_WORD_LISTS.map { |name| load(name) }.reduce(:|)
+    end
+
     # A lexicon is absent, unreadable, or not the shape this reader understands.
     #
     # Its own class rather than a bare RuntimeError for the same reason

data/lib/vicary/version.rb

@@ -6,5 +6,5 @@ module Vicary
   # Shared across all three front doors on purpose: one detector, one number. A
   # gem 0.3.0 that corresponds to nothing on PyPI cannot be reasoned about, and
   # the parity claim is between *versions*, not between package names.
-  VERSION = "0.2.8"
+  VERSION = "0.2.11"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: vicary
 version: !ruby/object:Gem::Version
-  version: 0.2.8
+  version: 0.2.11
 platform: ruby
 authors:
 - Blake Thomas
@@ -25,7 +25,8 @@ files:
 - README.md
 - assets/MANIFEST.json
 - assets/notability.txt.gz
-- assets/stop_words.txt
+- assets/stop_words_never_capitalised.txt
+- assets/stop_words_sometimes_capitalised.txt
 - lib/vicary.rb
 - lib/vicary/asset.rb
 - lib/vicary/candidates.rb

data/assets/stop_words.txt

@@ -1,63 +0,0 @@
-#!lexicon 1
-#!list stop_words 421
-# Ordinary words that must never become name candidates.
-#
-# Language-neutral on purpose. This list is the only thing standing between
-# candidate generation and "mask every capitalised word", so all three front doors
-# have to work from the same 421 words — a stoplist transliterated by hand into a
-# second language is a second detector wearing the first one's name, and the
-# difference shows up as prose corruption in one language and not the others,
-# which no parity check on masked output would catch.
-#
-# Skewed toward over-inclusion deliberately. A missed name is one span and shows
-# up in the recall number; a wrongly-masked common word corrupts every essay that
-# uses it and shows up nowhere unless somebody reads the prose.
-#
-# Format: `#!` lines are directives, `#` lines are comments, and every other line
-# contributes whitespace-separated words. The count on the `#!list` directive is
-# the number of DISTINCT words after case-folding, and each front door asserts it
-# against what it parsed. That is not ceremony: a short read here silently makes
-# the redactor MORE aggressive, which looks privacy-safe and passes any check that
-# only asks whether something was masked.
-#
-# A word appearing twice is not an error — the groupings below are thematic and
-# overlap ("else", "may", "us"), and enforcing uniqueness in the source would make
-# the list harder to read for no benefit.
-  a an the this that these those there here it its it's
-  i me my mine myself we us our ours ourselves you your yours
-  he him his she her hers they them their theirs who whom whose which what
-  and or but so because although though however therefore thus hence yet
-  if then else when while until since before after during once whenever
-  for from to into onto out off over under above below between among across
-  through around about against along beside besides beyond within without
-  at by in on up down near next last first second third finally
-  is am are was were be been being have has had having do does did doing
-  can could will would shall should may might must let lets
-  not no nor none nothing never always sometimes often usually rarely
-  all any both each every few many more most much several some such
-  another other others same different new old good bad better best worst
-  great big small long short high low young happy sad hard easy
-  one two three four five six seven eight nine ten hundred thousand million
-  also even just only really very too still again ever else quite rather
-  call called come came go went get got give gave take took make made
-  see saw look looked think thought know knew say said tell told ask asked
-  want wanted need needed try tried help helped work worked feel felt
-  find found keep kept leave left put set start started stop stopped
-  remember remembered learn learned teach taught write wrote read
-  everyone everybody someone somebody anyone anybody nobody everything
-  something anything people person thing things time times day days
-  year years week weeks month months hour hours minute minutes
-  school schools class classes teacher teachers student students friend
-  friends family families home house mom dad mother father parent parents
-  brother sister sisters brothers grandma grandpa
-  life world way ways place places part parts kind sort lot lots
-  yes yeah ok okay maybe perhaps well now today tomorrow yesterday
-  january february march april may june july august september october
-  november december monday tuesday wednesday thursday friday saturday sunday
-  mr mrs ms dr am pm usa us u.s tv
-  im ive ill id dont cant wont didnt isnt aint thats theres whats
-  as than instead unless whether either neither plus versus etc
-  getting making looking thinking talking playing living walking running
-  sitting standing growing learning moving trying using
-  back away together alone everywhere somewhere anywhere nowhere
-  right wrong true false sure certain important special favorite