vicary 0.2.14 → 0.2.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +13 -0
- data/lib/vicary/candidates.rb +115 -79
- data/lib/vicary/conformance.rb +17 -0
- data/lib/vicary/placeholders.rb +196 -0
- data/lib/vicary/version.rb +1 -1
- data/lib/vicary.rb +1 -0
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4e78fbd85fa4dde2e481bbf8b5349ceda35e4ab98fbf9ad0b336761be3f6379d
|
|
4
|
+
data.tar.gz: b9ad9cd2ebeaffa8b438c4b051d378edb47cfc86f1117cce68732f6cd3f2cdd3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 9ef2a213c75f9cf539620a393b20e7f2521173fb14a2bfbc2d6753654a8a19f19875b143a6b4240cc4ce09b01667cccc4d1da8d2e4c52d774bc9678c52eb13cb
|
|
7
|
+
data.tar.gz: 7102716c3687292aff85bc8191e0fbbe65ef457930f9fcce2c5b78e58ed3134a8e69d5b07f728a09438f0aeda84f97753a82419e91478ebd401ba99df5c2dd47
|
data/README.md
CHANGED
|
@@ -45,6 +45,19 @@ replaced. A map that is absent, or that covers only some of the placeholders in
|
|
|
45
45
|
the text, yields **no** spans and translates as the identity — a caller can
|
|
46
46
|
handle "no spans" and cannot detect a wrong offset.
|
|
47
47
|
|
|
48
|
+
Before a reader sees it, check that no placeholder the host did not restore
|
|
49
|
+
survived — echoed by a model, or paraphrased as `[NAME_1]`, `{Name_1}` or a bare
|
|
50
|
+
`NAME_1`:
|
|
51
|
+
|
|
52
|
+
```ruby
|
|
53
|
+
show(Vicary.assert_no_placeholders(feedback)) # raises PlaceholderLeakError
|
|
54
|
+
show(Vicary.scrub_placeholders(feedback).text) # or drop each sentence carrying one
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
The error names the kinds and the count, never the text.
|
|
58
|
+
`Vicary.find_placeholders` and `Vicary.has_placeholders?` answer without raising;
|
|
59
|
+
`scrub_placeholders(text, quote: true)` elides instead, for a verbatim quote.
|
|
60
|
+
|
|
48
61
|
## Checking it
|
|
49
62
|
|
|
50
63
|
Three layers, because each catches what the one above it cannot.
|
data/lib/vicary/candidates.rb
CHANGED
|
@@ -186,42 +186,32 @@ module Vicary
|
|
|
186
186
|
# of the commonest surname shapes there are.
|
|
187
187
|
PIECE = /[^'’\-‐-―]+/
|
|
188
188
|
|
|
189
|
-
# The word
|
|
190
|
-
#
|
|
191
|
-
# at
|
|
189
|
+
# The only shape a word with an interior capital can carry: a capital with a
|
|
190
|
+
# word character in front of it. Every capital {Candidates.interior_capital?}
|
|
191
|
+
# is allowed to fire on sits at word-index 1 or later, so it is preceded
|
|
192
|
+
# either by the word's own first character or by one of `[A-Za-z'’-]`, and
|
|
193
|
+
# both are inside this class. A document that fails this has no interior
|
|
194
|
+
# capital and needs no word scan at all; a document that passes has told the
|
|
195
|
+
# scan exactly where to look. That makes this a filter and never a second
|
|
196
|
+
# opinion — the answer stays that method's.
|
|
192
197
|
#
|
|
193
|
-
#
|
|
194
|
-
#
|
|
195
|
-
#
|
|
196
|
-
#
|
|
197
|
-
#
|
|
198
|
-
#
|
|
199
|
-
#
|
|
200
|
-
#
|
|
201
|
-
#
|
|
202
|
-
# has every earlier character in `[a-z'’-]` by construction, so the engine
|
|
203
|
-
# matches from the token's own start, and the trailing class then runs
|
|
204
|
-
# greedily to the token's own end.
|
|
205
|
-
#
|
|
206
|
-
# Why this is worth a second pattern: the scan is per-document and the words
|
|
207
|
-
# it is looking for are rare. On the twenty-essay gate corpus exactly ONE
|
|
208
|
-
# document contains an interior capital, so the other nineteen used to be
|
|
209
|
-
# tokenised to the last word — and a regex spent on each — to return false.
|
|
210
|
-
INTERIOR_CAP_WORD = /[A-Za-z][a-z'’-]*[A-Z][A-Za-z'’-]*/
|
|
211
|
-
|
|
212
|
-
# The two-character shape {INTERIOR_CAP_WORD} cannot match without: a
|
|
213
|
-
# capital with a word character in front of it. Every capital that pattern
|
|
214
|
-
# fires on is preceded either by its own first character or by one of
|
|
215
|
-
# `[a-z'’-]`, and both are inside this class, so a document that fails
|
|
216
|
-
# this has no interior capital and needs no word scan at all.
|
|
217
|
-
#
|
|
218
|
-
# It exists because the precise pattern backtracks. `[a-z'’-]*[A-Z]` is
|
|
219
|
-
# retried at every letter of every word that has no capital after it, which
|
|
220
|
-
# is most words in most documents; this is a two-character scan that cannot
|
|
221
|
-
# backtrack. On the twenty-essay gate corpus it clears thirteen documents
|
|
222
|
-
# outright at a twentieth of the cost.
|
|
198
|
+
# Two characters, and it cannot backtrack. It replaced a precise word
|
|
199
|
+
# pattern (`[A-Za-z][a-z'’-]*[A-Z][A-Za-z'’-]*`) that did: `[a-z'’-]*[A-Z]`
|
|
200
|
+
# is retried at every letter of every word that has no capital after it,
|
|
201
|
+
# which is most words in most documents, and running it over the whole text
|
|
202
|
+
# cost threefold what running this one and walking out from its hits costs —
|
|
203
|
+
# 57 µs per document against 17.5 on the twenty-essay gate corpus. The walk
|
|
204
|
+
# is in {Candidates.capitalises_inside_a_word?}; the word it reaches is the
|
|
205
|
+
# same substring {WORD_TOKEN} would have produced, because the walk uses
|
|
206
|
+
# that token's own continuation class.
|
|
223
207
|
INTERIOR_CAP_HINT = /[A-Za-z'’-][A-Z]/
|
|
224
208
|
|
|
209
|
+
# The complement of {WORD_TOKEN}'s continuation class, and its first-character
|
|
210
|
+
# class. The interior-capital walk reads outwards from a hint rather than
|
|
211
|
+
# forwards from a word, so it needs the boundary rather than the token.
|
|
212
|
+
NOT_WORD_CHARACTER = /[^A-Za-z'’-]/
|
|
213
|
+
WORD_FIRST_CHARACTER = /[A-Za-z]/
|
|
214
|
+
|
|
225
215
|
# Where a sentence begins: start of text, after terminal punctuation and any
|
|
226
216
|
# closing quote, after a line break, or immediately inside an *opening*
|
|
227
217
|
# quote. A capital in one of these positions is required by orthography, so
|
|
@@ -383,6 +373,11 @@ module Vicary
|
|
|
383
373
|
# One row of the precedence table: a tag, and what it decides.
|
|
384
374
|
PrecedenceRow = Struct.new(:tag, :mask, :kind)
|
|
385
375
|
|
|
376
|
+
# What one pass over a document's words says about how the writer cases
|
|
377
|
+
# them. See {Candidates.document_casing}; either half is empty when the
|
|
378
|
+
# caller did not ask for it.
|
|
379
|
+
DocumentCasing = Struct.new(:written_as_a_capital, :written_in_lower_case)
|
|
380
|
+
|
|
386
381
|
# The precedence table. The first row whose tag the span carries decides both
|
|
387
382
|
# the mask/keep verdict and the placeholder, and that is the whole
|
|
388
383
|
# classification policy.
|
|
@@ -1085,16 +1080,63 @@ module Vicary
|
|
|
1085
1080
|
# Horses" vouch for "Horses" as a name — the heading corroborating itself,
|
|
1086
1081
|
# one line removed.
|
|
1087
1082
|
def mid_sentence_capitals(text, starts, headings = [])
|
|
1088
|
-
|
|
1083
|
+
document_casing(text, starts, headings, lower_case: false).written_as_a_capital
|
|
1084
|
+
end
|
|
1085
|
+
|
|
1086
|
+
# Both halves of the document's casing testimony, from ONE pass over its
|
|
1087
|
+
# words.
|
|
1088
|
+
#
|
|
1089
|
+
# {.mid_sentence_capitals} and {.written_in_lower_case} ask opposite
|
|
1090
|
+
# questions of the same tokens — one reads the words a writer capitalised
|
|
1091
|
+
# where orthography would not have, the other the words they left
|
|
1092
|
+
# lower-case — and a token answers exactly one of them, because a word's
|
|
1093
|
+
# first character is either a capital or it is not. Run separately they
|
|
1094
|
+
# tokenised every document twice.
|
|
1095
|
+
#
|
|
1096
|
+
# That second tokenisation is what this exists to remove, and it is worth
|
|
1097
|
+
# a method rather than a comment: the pair was most of what the two
|
|
1098
|
+
# orthography channels added to the release latency gate.
|
|
1099
|
+
#
|
|
1100
|
+
# Each half is opt-in because each has a caller that does not want it. The
|
|
1101
|
+
# lower-case set is evidence only where the writer marks proper nouns at
|
|
1102
|
+
# all (see `lower_cased` in {.find_candidates}), and asking for a set that
|
|
1103
|
+
# will be discarded is the cost this method was written to avoid.
|
|
1104
|
+
def document_casing(text, starts = Set.new, headings = [], capitals: true, lower_case: true)
|
|
1105
|
+
capitalised = Set.new
|
|
1106
|
+
lowered = Set.new
|
|
1089
1107
|
each_match(text, WORD_TOKEN) do |m|
|
|
1090
1108
|
token = m[0]
|
|
1091
|
-
|
|
1109
|
+
# A byte comparison rather than `token[0].match?(/[A-Z]/)`, which
|
|
1110
|
+
# allocated a one-character string and ran a regex for every word of
|
|
1111
|
+
# every document. Exact, not an approximation: {WORD_TOKEN}'s first
|
|
1112
|
+
# character is `[A-Za-z]`, so it is ASCII, a single byte, and
|
|
1113
|
+
# lower-case exactly when it is not upper-case — which is what splits
|
|
1114
|
+
# the two halves, at one test rather than two.
|
|
1115
|
+
first = token.getbyte(0)
|
|
1116
|
+
if first >= 97 && first <= 122
|
|
1117
|
+
next unless lower_case
|
|
1118
|
+
|
|
1119
|
+
# {.strip} copies the string whether or not it takes anything off,
|
|
1120
|
+
# and almost no word ends in an apostrophe. Same value either way —
|
|
1121
|
+
# a strip that removes nothing returns an equal string, and the set
|
|
1122
|
+
# holds values.
|
|
1123
|
+
#
|
|
1124
|
+
# Only the tail is worth testing. {.strip} works in from both ends,
|
|
1125
|
+
# and the head was just established to be a lower-case ASCII letter,
|
|
1126
|
+
# so the leading pass stops on the first character every time.
|
|
1127
|
+
lowered_token = token.downcase
|
|
1128
|
+
lowered_token = strip(lowered_token, "'’") if lowered_token.end_with?("'", "’")
|
|
1129
|
+
lowered << lowered_token
|
|
1130
|
+
next
|
|
1131
|
+
end
|
|
1132
|
+
next unless capitals
|
|
1133
|
+
next if starts.include?(m.begin(0))
|
|
1092
1134
|
next if token.length > 1 && upper?(token)
|
|
1093
1135
|
next if overlaps?(headings, m.begin(0), m.begin(0) + token.length)
|
|
1094
1136
|
|
|
1095
|
-
|
|
1137
|
+
capitalised << strip(token.downcase, "'’")
|
|
1096
1138
|
end
|
|
1097
|
-
|
|
1139
|
+
DocumentCasing.new(capitalised, lowered)
|
|
1098
1140
|
end
|
|
1099
1141
|
|
|
1100
1142
|
# ---------------------------------------------------------------------
|
|
@@ -1334,13 +1376,27 @@ module Vicary
|
|
|
1334
1376
|
# NWP corpus 14 papers carry an interior capital and 2 of them (`SPecial`,
|
|
1335
1377
|
# `AFter`) carry it only inside a heading.
|
|
1336
1378
|
#
|
|
1337
|
-
# The scan is
|
|
1338
|
-
# is a filter and not a change of answer — see that
|
|
1379
|
+
# The scan is driven off {INTERIOR_CAP_HINT} rather than run over every
|
|
1380
|
+
# word token, which is a filter and not a change of answer — see that
|
|
1381
|
+
# pattern. Each hit names a capital and the walk reaches the word carrying
|
|
1382
|
+
# it; the hit need not be interior itself (`'ChoaCh'` hints at its own
|
|
1383
|
+
# first letter), so the WHOLE token is handed over rather than the
|
|
1384
|
+
# position being judged here.
|
|
1339
1385
|
def capitalises_inside_a_word?(text)
|
|
1340
|
-
|
|
1341
|
-
|
|
1342
|
-
|
|
1343
|
-
|
|
1386
|
+
pos = 0
|
|
1387
|
+
while (hint = INTERIOR_CAP_HINT.match(text, pos))
|
|
1388
|
+
capital = hint.begin(0) + 1
|
|
1389
|
+
boundary = text.rindex(NOT_WORD_CHARACTER, capital - 1)
|
|
1390
|
+
# A word begins on a letter, so a run of leading apostrophes or
|
|
1391
|
+
# hyphens the walk crossed belongs to no word — `'Abc` tokenises to
|
|
1392
|
+
# `Abc`. The capital is itself a letter, so this never runs past it.
|
|
1393
|
+
start = text.index(WORD_FIRST_CHARACTER, boundary.nil? ? 0 : boundary + 1)
|
|
1394
|
+
stop = text.index(NOT_WORD_CHARACTER, capital + 1) || text.length
|
|
1395
|
+
return true if interior_capital?(text[start...stop])
|
|
1396
|
+
|
|
1397
|
+
# Past the whole word: it has just been read in full, so a second
|
|
1398
|
+
# capital inside it would ask the same question again.
|
|
1399
|
+
pos = stop
|
|
1344
1400
|
end
|
|
1345
1401
|
false
|
|
1346
1402
|
end
|
|
@@ -1359,29 +1415,7 @@ module Vicary
|
|
|
1359
1415
|
# letters, which is what keeps the rule free of any imported collision: it
|
|
1360
1416
|
# consults no list, so it cannot inherit one's mistakes.
|
|
1361
1417
|
def written_in_lower_case(text)
|
|
1362
|
-
|
|
1363
|
-
# `scan` rather than `each_match`, and a byte comparison rather than
|
|
1364
|
-
# `token[0].match?(/[a-z]/)` — which allocated a one-character string
|
|
1365
|
-
# and ran a regex for every word of every document. Exact, not an
|
|
1366
|
-
# approximation: {WORD_TOKEN}'s first character is `[A-Za-z]`, so it is
|
|
1367
|
-
# ASCII and a single byte by construction.
|
|
1368
|
-
text.scan(WORD_TOKEN) do |token|
|
|
1369
|
-
first = token.getbyte(0)
|
|
1370
|
-
next unless first >= 97 && first <= 122
|
|
1371
|
-
|
|
1372
|
-
# {.strip} copies the string whether or not it takes anything off,
|
|
1373
|
-
# and almost no word ends in an apostrophe. Same value either way — a
|
|
1374
|
-
# strip that removes nothing returns an equal string, and the set
|
|
1375
|
-
# holds values.
|
|
1376
|
-
#
|
|
1377
|
-
# Only the tail is worth testing. {.strip} works in from both ends,
|
|
1378
|
-
# and the head was just established to be a lower-case ASCII letter,
|
|
1379
|
-
# so the leading pass stops on the first character every time.
|
|
1380
|
-
lowered = token.downcase
|
|
1381
|
-
lowered = strip(lowered, "'’") if lowered.end_with?("'", "’")
|
|
1382
|
-
out << lowered
|
|
1383
|
-
end
|
|
1384
|
-
out
|
|
1418
|
+
document_casing(text, capitals: false).written_in_lower_case
|
|
1385
1419
|
end
|
|
1386
1420
|
|
|
1387
1421
|
# The mid-sentence guard: drop a lone capital a sloppy capitaliser chose.
|
|
@@ -1947,21 +1981,23 @@ module Vicary
|
|
|
1947
1981
|
blocked.any? { |block_start, block_end| start < block_end && finish > block_start }
|
|
1948
1982
|
end
|
|
1949
1983
|
|
|
1950
|
-
|
|
1951
|
-
#
|
|
1952
|
-
#
|
|
1984
|
+
# Both casing questions, read once from one pass over the words, for the
|
|
1985
|
+
# same reason `habit` is read once: two call sites computing it
|
|
1986
|
+
# separately could disagree.
|
|
1987
|
+
#
|
|
1988
|
+
# The lower-case half is asked for only where the writer marks proper
|
|
1989
|
+
# nouns at all: that rule reads the ABSENCE of a capital as testimony,
|
|
1990
|
+
# and the habit states in as many words that an absence means nothing in
|
|
1991
|
+
# a LOWERCASE or SILENT document. Running it there would suppress every
|
|
1992
|
+
# capital the writer did manage.
|
|
1993
|
+
casing = document_casing(
|
|
1994
|
+
text, starts, headings,
|
|
1995
|
+
lower_case: case_variance && !given_name.nil? && marks_proper_nouns?(habit),
|
|
1996
|
+
)
|
|
1997
|
+
written_as_a_capital = casing.written_as_a_capital
|
|
1998
|
+
lower_cased = casing.written_in_lower_case
|
|
1953
1999
|
stray_capitals = mid_sentence_corroboration && !given_name.nil? &&
|
|
1954
2000
|
capitalises_ordinary_words?(text, headings)
|
|
1955
|
-
# The mirror of `written_as_a_capital`, read once for the same reason.
|
|
1956
|
-
# Empty unless the writer marks proper nouns at all: this rule reads the
|
|
1957
|
-
# ABSENCE of a capital as testimony, and the habit states in as many
|
|
1958
|
-
# words that an absence means nothing in a LOWERCASE or SILENT document.
|
|
1959
|
-
# Running it there would suppress every capital the writer did manage.
|
|
1960
|
-
lower_cased = if case_variance && !given_name.nil? && marks_proper_nouns?(habit)
|
|
1961
|
-
written_in_lower_case(text)
|
|
1962
|
-
else
|
|
1963
|
-
Set.new
|
|
1964
|
-
end
|
|
1965
2001
|
|
|
1966
2002
|
out = []
|
|
1967
2003
|
each_match(text, CANDIDATE_RE) do |m|
|
data/lib/vicary/conformance.rb
CHANGED
|
@@ -181,6 +181,23 @@ module Vicary
|
|
|
181
181
|
raw
|
|
182
182
|
end
|
|
183
183
|
|
|
184
|
+
# The placeholder guard's contract: the pattern source, the cases and the
|
|
185
|
+
# reference's answers. Absent is an error, as for `spans.json`: a guard
|
|
186
|
+
# checked against nothing reports the same green as one that agrees.
|
|
187
|
+
def load_placeholders(dir = nil)
|
|
188
|
+
dir = Pathname.new(dir || directory)
|
|
189
|
+
path = dir.join("placeholders.json")
|
|
190
|
+
unless path.file?
|
|
191
|
+
raise SpecError,
|
|
192
|
+
"no placeholders.json at #{path}. The ports would check their " \
|
|
193
|
+
"placeholder guard against nothing."
|
|
194
|
+
end
|
|
195
|
+
|
|
196
|
+
raw = JSON.parse(path.read)
|
|
197
|
+
require_version(raw["document_version"], "placeholders.json")
|
|
198
|
+
raw
|
|
199
|
+
end
|
|
200
|
+
|
|
184
201
|
# Score an implementation against every frame.
|
|
185
202
|
#
|
|
186
203
|
# The block receives (sentence, identity) — the same input every Python arm
|
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Vicary
|
|
4
|
+
# Text bound for a reader still carries a redaction placeholder.
|
|
5
|
+
#
|
|
6
|
+
# Names the kinds and the count, never the text around them: the message is
|
|
7
|
+
# going to a log, and the text is exactly what was not supposed to leave.
|
|
8
|
+
class PlaceholderLeakError < ArgumentError
|
|
9
|
+
attr_reader :kinds, :count
|
|
10
|
+
|
|
11
|
+
def initialize(kinds, count)
|
|
12
|
+
@kinds = kinds.freeze
|
|
13
|
+
@count = count
|
|
14
|
+
noun = count == 1 ? "placeholder" : "placeholders"
|
|
15
|
+
super("#{count} redaction #{noun} would reach a reader " \
|
|
16
|
+
"(kinds: #{kinds.join(', ')})")
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
# The host boundary: assert that no placeholder reaches a reader — the Ruby
|
|
21
|
+
# port of `python/src/vicary/placeholders.py`.
|
|
22
|
+
#
|
|
23
|
+
# vicary masks what it redacts as typed placeholders (`{NAME_1}`,
|
|
24
|
+
# `{LOCATION_2}`, `{EMAIL}`) and a host restores the ones a reader may see.
|
|
25
|
+
# Everything else must never be displayed, and two ways one gets out anyway
|
|
26
|
+
# are invisible to the masker: a model echoes a placeholder the host's restore
|
|
27
|
+
# map does not cover, or paraphrases one — `[NAME_1]`, `{Name_1}`,
|
|
28
|
+
# `{ NAME_1 }`, a bare `NAME_1`.
|
|
29
|
+
#
|
|
30
|
+
# {assert_no_placeholders} raises before the text is shown;
|
|
31
|
+
# {scrub_placeholders} removes them (a prose sentence carrying one is dropped,
|
|
32
|
+
# a quote elides it with `…`). Nothing is rewritten back into a word here.
|
|
33
|
+
#
|
|
34
|
+
# The pattern source is byte-identical in all three front doors, written in
|
|
35
|
+
# the dialect the engines share — ASCII classes rather than `\b`/`\w`/`\s`,
|
|
36
|
+
# and `\A`/`\z` rather than `^`/`$`, which anchor per line in Ruby.
|
|
37
|
+
# `test/placeholders_test.rb` compares it with `conformance/placeholders.json`.
|
|
38
|
+
module Placeholders
|
|
39
|
+
KINDS = %w[
|
|
40
|
+
ADDRESS AGE AWS_ACCESS_KEY AWS_SECRET_KEY
|
|
41
|
+
CREDIT_DEBIT_CARD_CVV CREDIT_DEBIT_CARD_EXPIRY
|
|
42
|
+
CREDIT_DEBIT_CARD_NUMBER DATE_OF_BIRTH DRIVER_ID EMAIL
|
|
43
|
+
IP_ADDRESS LICENSE_PLATE LOCATION MAC_ADDRESS NAME
|
|
44
|
+
ORGANIZATION PASSWORD PERSON PHONE PIN SCHOOL URL
|
|
45
|
+
US_BANK_ACCOUNT_NUMBER US_BANK_ROUTING_NUMBER US_PASSPORT_NUMBER
|
|
46
|
+
US_SOCIAL_SECURITY_NUMBER USERNAME ZIP_CODE
|
|
47
|
+
].freeze
|
|
48
|
+
private_constant :KINDS
|
|
49
|
+
|
|
50
|
+
# Every placeholder kind vicary or its Guardrail arm can mint, plus
|
|
51
|
+
# `PERSON`, which is what a model writes when it paraphrases `NAME`.
|
|
52
|
+
# Longest first, so a match reports the whole kind.
|
|
53
|
+
PLACEHOLDER_KINDS = KINDS.sort_by { |kind| [-kind.length, kind] }.freeze
|
|
54
|
+
|
|
55
|
+
# The kind reported for a `{UPPER_CASE}` token that names no known kind.
|
|
56
|
+
OTHER_KIND = "OTHER"
|
|
57
|
+
|
|
58
|
+
WORD = "[A-Za-z0-9_]"
|
|
59
|
+
private_constant :WORD
|
|
60
|
+
|
|
61
|
+
# `NAME` -> `[Nn][Aa][Mm][Ee]`: case-insensitive in every dialect.
|
|
62
|
+
def self.fold(kind)
|
|
63
|
+
kind.chars.map { |c| c.match?(/[A-Z]/) ? "[#{c}#{c.downcase}]" : c }.join
|
|
64
|
+
end
|
|
65
|
+
private_class_method :fold
|
|
66
|
+
|
|
67
|
+
# Anything placeholder-shaped: vicary's own `{KIND}` / `{KIND_n}` for any
|
|
68
|
+
# upper-case kind; a known kind in any case inside `{}`, `[]` or `<>` with
|
|
69
|
+
# stray spaces or tabs; or a bare upper-case `KIND_n` as a whole word.
|
|
70
|
+
PLACEHOLDER_LIKE = Regexp.new(
|
|
71
|
+
"\\{[A-Z][A-Z_]*(?:_[0-9]+)?\\}" \
|
|
72
|
+
"|[{\\[<][ \\t]*(?:#{PLACEHOLDER_KINDS.map { |k| fold(k) }.join('|')})" \
|
|
73
|
+
"(?:_[0-9]+)?[ \\t]*[}\\]>]" \
|
|
74
|
+
"|(?<!#{WORD})(?:#{PLACEHOLDER_KINDS.join('|')})_[0-9]+(?!#{WORD})"
|
|
75
|
+
)
|
|
76
|
+
|
|
77
|
+
# A sentence ends at `.`, `!` or `?` followed by whitespace — but not after
|
|
78
|
+
# an honorific, an initial, `e.g.`, `i.e.` or `vs.`.
|
|
79
|
+
SENTENCE_END = Regexp.new(
|
|
80
|
+
"(?<![A-Z][a-z]\\.)" \
|
|
81
|
+
"(?<!(?<!#{WORD})[A-Z][a-z][a-z]\\.)" \
|
|
82
|
+
"(?<!(?<!#{WORD})[A-Z]\\.)" \
|
|
83
|
+
"(?<!(?<!#{WORD})e\\.g\\.)" \
|
|
84
|
+
"(?<!(?<!#{WORD})i\\.e\\.)" \
|
|
85
|
+
"(?<!(?<!#{WORD})vs\\.)" \
|
|
86
|
+
"(?<=[.!?])[ \\t\\n\\r\\f\\v]+"
|
|
87
|
+
)
|
|
88
|
+
|
|
89
|
+
# What stands in for a placeholder when every sentence of a prose field
|
|
90
|
+
# carries one. Any other kind reads "this".
|
|
91
|
+
GENERIC_STAND_INS = {
|
|
92
|
+
"NAME" => "someone",
|
|
93
|
+
"PERSON" => "someone",
|
|
94
|
+
"LOCATION" => "a place",
|
|
95
|
+
"SCHOOL" => "the school",
|
|
96
|
+
"ORGANIZATION" => "an organization"
|
|
97
|
+
}.freeze
|
|
98
|
+
|
|
99
|
+
TRIM = /\A[ \t\n\r\f\v]+|[ \t\n\r\f\v]+\z/
|
|
100
|
+
TRIM_TOKEN = /\A[{}\[\]<> \t\n\r\f\v]+|[{}\[\]<> \t\n\r\f\v]+\z/
|
|
101
|
+
ELISION = /[ \t\n\r\f\v]*…[ \t\n\r\f\v]*/
|
|
102
|
+
ELISION_BEFORE_PUNCT = / … ([.,;:!?])/
|
|
103
|
+
NUMBER_SUFFIX = /_[0-9]+\z/
|
|
104
|
+
private_constant :TRIM, :TRIM_TOKEN, :ELISION, :ELISION_BEFORE_PUNCT,
|
|
105
|
+
:NUMBER_SUFFIX
|
|
106
|
+
|
|
107
|
+
# One placeholder found in text. `start`/`end` count characters.
|
|
108
|
+
Match = Struct.new(:text, :kind, :start, :end) do
|
|
109
|
+
def to_h
|
|
110
|
+
{ "text" => text, "kind" => kind, "start" => start, "end" => self.end }
|
|
111
|
+
end
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
# Text with every placeholder removed, and how many were.
|
|
115
|
+
ScrubResult = Struct.new(:text, :removed)
|
|
116
|
+
|
|
117
|
+
class << self
|
|
118
|
+
# The kind a placeholder token names: `[ Name_2 ]` -> `NAME`; else
|
|
119
|
+
# {OTHER_KIND}.
|
|
120
|
+
def kind_of(token)
|
|
121
|
+
kind = token.gsub(TRIM_TOKEN, "").upcase.sub(NUMBER_SUFFIX, "")
|
|
122
|
+
PLACEHOLDER_KINDS.include?(kind) ? kind : OTHER_KIND
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
# Every placeholder in `text`, in order.
|
|
126
|
+
def find(text)
|
|
127
|
+
return [] if text.nil? || text.empty?
|
|
128
|
+
|
|
129
|
+
text.to_enum(:scan, PLACEHOLDER_LIKE).map do
|
|
130
|
+
m = Regexp.last_match
|
|
131
|
+
Match.new(m[0], kind_of(m[0]), m.begin(0), m.end(0))
|
|
132
|
+
end
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
# Whether `text` carries any placeholder.
|
|
136
|
+
def any?(text)
|
|
137
|
+
!text.nil? && !text.empty? && PLACEHOLDER_LIKE.match?(text)
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
# Return `text` unchanged, or raise {Vicary::PlaceholderLeakError}. Call
|
|
141
|
+
# it on the last string a host builds before a reader sees it.
|
|
142
|
+
def assert_none(text)
|
|
143
|
+
found = find(text)
|
|
144
|
+
unless found.empty?
|
|
145
|
+
raise PlaceholderLeakError.new(found.map(&:kind).uniq.sort, found.length)
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
text
|
|
149
|
+
end
|
|
150
|
+
|
|
151
|
+
# Remove every placeholder from `text`. Prose (the default) drops each
|
|
152
|
+
# sentence carrying one, or — when every sentence does — replaces each
|
|
153
|
+
# with its {GENERIC_STAND_INS} entry; `quote: true` elides each as `…`.
|
|
154
|
+
def scrub(text, quote: false)
|
|
155
|
+
return ScrubResult.new(text, 0) unless any?(text)
|
|
156
|
+
|
|
157
|
+
removed = text.scan(PLACEHOLDER_LIKE).length
|
|
158
|
+
if quote
|
|
159
|
+
out = text.gsub(PLACEHOLDER_LIKE, "…").gsub(ELISION, " … ")
|
|
160
|
+
.gsub(ELISION_BEFORE_PUNCT, ' …\1').gsub(TRIM, "")
|
|
161
|
+
return ScrubResult.new(out, removed)
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
kept = text.gsub(TRIM, "").split(SENTENCE_END).reject { |s| any?(s) }
|
|
165
|
+
return ScrubResult.new(kept.join(" "), removed) unless kept.empty?
|
|
166
|
+
|
|
167
|
+
out = text.gsub(PLACEHOLDER_LIKE) do |token|
|
|
168
|
+
GENERIC_STAND_INS.fetch(kind_of(token), "this")
|
|
169
|
+
end
|
|
170
|
+
ScrubResult.new(out, removed)
|
|
171
|
+
end
|
|
172
|
+
end
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
class << self
|
|
176
|
+
# Every placeholder in `text` — see {Vicary::Placeholders}.
|
|
177
|
+
def find_placeholders(text)
|
|
178
|
+
Placeholders.find(text)
|
|
179
|
+
end
|
|
180
|
+
|
|
181
|
+
# Whether `text` carries any placeholder.
|
|
182
|
+
def has_placeholders?(text)
|
|
183
|
+
Placeholders.any?(text)
|
|
184
|
+
end
|
|
185
|
+
|
|
186
|
+
# Return `text`, or raise {Vicary::PlaceholderLeakError}.
|
|
187
|
+
def assert_no_placeholders(text)
|
|
188
|
+
Placeholders.assert_none(text)
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
# Remove every placeholder from `text`.
|
|
192
|
+
def scrub_placeholders(text, quote: false)
|
|
193
|
+
Placeholders.scrub(text, quote: quote)
|
|
194
|
+
end
|
|
195
|
+
end
|
|
196
|
+
end
|
data/lib/vicary/version.rb
CHANGED
|
@@ -6,5 +6,5 @@ module Vicary
|
|
|
6
6
|
# Shared across all three front doors on purpose: one detector, one number. A
|
|
7
7
|
# gem 0.3.0 that corresponds to nothing on PyPI cannot be reasoned about, and
|
|
8
8
|
# the parity claim is between *versions*, not between package names.
|
|
9
|
-
VERSION = "0.2.
|
|
9
|
+
VERSION = "0.2.16"
|
|
10
10
|
end
|
data/lib/vicary.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: vicary
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.16
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Blake Thomas
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
11
|
+
date: 2026-10-19 00:00:00.000000000 Z
|
|
12
12
|
dependencies: []
|
|
13
13
|
description: |
|
|
14
14
|
Finds the names a student writes about — classmates, teachers, relatives — and
|
|
@@ -38,6 +38,7 @@ files:
|
|
|
38
38
|
- lib/vicary/latency_baseline.rb
|
|
39
39
|
- lib/vicary/lexicon.rb
|
|
40
40
|
- lib/vicary/minter.rb
|
|
41
|
+
- lib/vicary/placeholders.rb
|
|
41
42
|
- lib/vicary/redact.rb
|
|
42
43
|
- lib/vicary/spans.rb
|
|
43
44
|
- lib/vicary/structured.rb
|