twfilter 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9fa81150f2ea0cdd0793e84dc5bca8b568fe7ab699693b90a6023a3933e00e40
+  data.tar.gz: 1264fea5c38f14348f7b2767acaf1583c89b793c341eebfd5864e106f38070a9
+SHA512:
+  metadata.gz: dd27ae0c7be37abc15e95a0b014d07cb41c8096fd4ad8a98b5574309e3706155f53086360ccea33d20e009399270f23ccd2ab08cfed514ca1f67ef114d89632a
+  data.tar.gz: 7aa05c02c19e57255e452def5ba72ea17aa253d5ea4aa485fc51eba965740bb9a10ccd940be1e5e3b791e959d4962f3727e452b5cb6960556fbc5bf2dbc3d2ed

data/CHANGELOG.md

@@ -0,0 +1,37 @@
+# Changelog
+
+## 0.1.0 — 2026-08-01
+
+First release.
+
+- Provenance analysis over five independent checks, each reporting typed findings with a
+  severity rather than a boolean (`TWFilter::Report`, `TWFilter::Finding`)
+- MOE character-chart tiering: 常用 4 808, 次常用 6 343, 罕用 18 356, plus 12 dictionary
+  headword characters outside all three charts. Any Han character outside that inventory is
+  rejected unconditionally (`TWFilter::Checks::Script`)
+- Simplified-form detection derived from the OpenCC `STCharacters` / `TSCharacters`
+  round-trip, minus 16 forms that appear in an MOE chart despite occurring in the
+  simplification tables
+- Mainland traditional orthography: 裏着衞爲説麽綫墻眞啓衆産囯麪
+- Mainland lexical markers graded by attestation against a native reference corpus:
+  149 rejecting, 38 marking, 6 exception masks (`TWFilter::Checks::Lexicon`)
+- Hong Kong, written-Cantonese and Singapore markers: 17 characters and 19 word forms
+- 80 non-Taiwanese toponyms and institutions, tagged rather than rejected, with an address
+  guard for the Taipei streets named after mainland cities
+- Erhua two-window analysis over 兒 as a root morpheme (`TWFilter::Checks::Erhua`)
+- Literary Chinese density over 矣哉汝吾 (`TWFilter::Checks::Wenyan`)
+- Positive Taiwan evidence over lexicon, sentence-final particles, ROC institutions and the
+  有 + V perfective outside negation (`TWFilter::Evidence`)
+- Punctuation normalization to 教育部《重訂標點符號手冊》: quotation brackets, the
+  U+2027 separator, six-dot ellipsis, box-drawing dash, width folding, invisible removal
+  (`TWFilter::Punctuation`)
+- Sentence segmentation over terminator and closer classes, linear in input length
+  (`TWFilter::Sentences`)
+- Block-level judgement with a contamination tolerance and an evidence floor
+  (`TWFilter::Block`)
+- Three policies — `corpus`, `publishable`, `permissive` — as immutable value objects
+  (`TWFilter::Policy`)
+- Reference tables resolvable from an external directory, with a checksum manifest and a
+  provenance fingerprint for recording alongside derived measurements (`TWFilter::Tables`)
+- A stated membership rule per table, enforced against the shipped data
+  (`spec/twfilter/membership_spec.rb`)

data/LICENSE

@@ -0,0 +1,19 @@
+MIT License
+
+Copyright (c) 2026 Den Patin <hi@dpat.in>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this
+software and associated documentation files (the "Software"), to deal in the Software
+without restriction, including without limitation the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
+to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or
+substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
+FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

data/NOTICES.md

@@ -0,0 +1,93 @@
+# Notices
+
+The code in this gem is MIT. The reference tables under `lib/twfilter/data/` are not all
+mine. This file records the provenance and terms of each. **Every table may be used in
+commercial products**, subject to the attribution stated below.
+
+## Table provenance
+
+| Table | Source | Terms |
+| --- | --- | --- |
+| `moe_common.txt`, `moe_secondary.txt`, `moe_rare.txt` | 全字庫 CNS 11643, field `CNS_source`, entries marked 常用國字標準字體表 / 次常用國字標準字體表 / 罕用字體表 | OGDL v1.0 |
+| `moe_exception.txt` | Characters occurring as headwords in the MOE dictionaries but in none of the three charts | Factual inventory; see below |
+| `simplified_only.txt` | Computed from OpenCC `STCharacters` ∖ `TSCharacters` | Apache-2.0 |
+| `mainland_hard.tsv`, `mainland_soft.tsv` | Candidate pairs from OpenCC `TWPhrases` and from hand curation; retention and banding computed by me | Apache-2.0 |
+| all other tables | Authored by me | MIT |
+
+## OGDL v1.0 — 政府資料開放授權條款 第1版
+
+Applies to the three character charts, obtained from 全字庫 (數位發展部), which offers its
+data under OGDL v1.0 or SIL OFL 1.1 at the user's option. I take OGDL v1.0.
+
+Grants: unrestricted use, reproduction, adaptation, distribution and public transmission,
+including for commercial purposes and in derivative products, worldwide and without time
+limit.
+
+Requires: attribution (註明出處). **Failure to attribute voids the grant ab initio.**
+
+Attribution string:
+
+```text
+數位發展部，CNS11643中文標準交換碼全字庫網站
+```
+
+## Apache-2.0 — OpenCC
+
+Applies to `simplified_only.txt` and to the candidate pairs behind `mainland_hard.tsv` and
+`mainland_soft.tsv`. Full text in `licenses/APACHE-2.0.txt` (§4(a)). OpenCC ships no NOTICE
+file, so §4(d) does not apply.
+
+Per §4(b), the derived files are **modified**:
+
+- `simplified_only.txt` is a set difference over two OpenCC conversion tables, further
+  reduced by 16 forms standard in Taiwan. It is not an OpenCC table and does not convert.
+- `mainland_hard.tsv` and `mainland_soft.tsv` retain 187 of 780 candidate pairs. Retention,
+  banding and the attestation counts are mine. The excluded 593 are excluded because
+  `TWPhrases` is an orthographic mapping, not a provenance signal.
+
+Attribution string:
+
+```text
+OpenCC — https://github.com/BYVoid/OpenCC, Apache License 2.0
+```
+
+## MOE dictionaries — CC BY-ND 3.0 TW
+
+`moe_exception.txt` lists 12 characters found as dictionary headwords. A character
+inventory states which characters occur; it reproduces no definition, no example and no
+expression, and is therefore neither a copy nor an adaptation of the dictionaries. Taiwan
+著作權法 art. 9(1)(3) excludes 通用之符號 and 表格 from protection for want of creativity.
+
+Attribution is given regardless:
+
+```text
+中華民國教育部《重編國語辭典修訂本》《國語辭典簡編本》
+```
+
+The ND term restricts distribution of adaptations. No dictionary text is shipped here, so
+the term is not engaged. Consumers who ship dictionary *content* must satisfy it
+themselves.
+
+## My own tables
+
+`variants_used_in_taiwan.txt`, `converted_orthography.txt`, `cantonese.txt`, `wenyan.txt`,
+`mainland_exceptions.tsv`, `regional_hard.tsv`, `foreign_topics.txt`, `taiwan_markers.txt`,
+`taiwan_lexicon.txt`, `taiwan_particles.txt`, `taiwan_grammar.txt`, `erhua_headed.txt`,
+`erhua_tailed.txt` — MIT, same as the code.
+
+These are word lists. Individual words are not copyrightable; the selection and arrangement
+are mine and are released under MIT.
+
+## What a consumer must do
+
+Shipping this gem in a commercial product requires reproducing two attribution strings —
+全字庫 and OpenCC — somewhere a user can reach, and including `licenses/APACHE-2.0.txt`.
+`NOTICES.md` satisfies both if shipped as is.
+
+Nothing here restricts commercial use, field of endeavour, or the license of the work you
+build. No table carries a NonCommercial, ShareAlike or copyleft term.
+
+## Downstream output
+
+Verdicts, findings and counts produced by this gem are yours. They are measurements over
+your input, not derivatives of these tables.

data/README.md

@@ -0,0 +1,364 @@
+# twfilter
+
+Provenance analysis for Chinese text: is this span Taiwan Mandarin (臺灣華語), and if not,
+on what evidence. Pure Ruby, no runtime dependencies, no native extensions, no network,
+deterministic output.
+
+Traditional Chinese is written in Taiwan, Hong Kong and Macau, and by mainland publishers
+converting from simplified. Script alone does not identify origin. Every string below is
+written in traditional characters, and only three of them are Taiwan Mandarin.
+
+```ruby
+TWFilter.keep?("這個政策有沒有經過完整評估？")    # => true
+TWFilter.keep?("请把这个信息转发给同事")         # => false (simplified script)
+TWFilter.keep?("請把這個信息轉發給同事")         # => false (lexicon: 信息, Taiwan writes 資訊)
+TWFilter.keep?("他站在門裏面等了很久")           # => false (orthography: 裏, Taiwan writes 裡)
+TWFilter.keep?("佢哋唔知道發生咗咩事")           # => false (Cantonese particles)
+TWFilter.keep?("這事兒辦得挺漂亮的")             # => false (erhua suffix on 事)
+TWFilter.keep?("他的女兒今年考上大學")           # => true (兒 here is a root morpheme)
+```
+
+Depth matters as much as breadth. Conversion tables are not provenance signals, and
+subject matter is not provenance either:
+
+```ruby
+# 程序 is ordinary Taiwanese legal vocabulary, not a mainland form
+TWFilter.keep?("行政程序法所定之程序應由主管機關進行")    # => true
+
+# 了嗎 is attested in every native Taiwanese source; it is not a marker
+TWFilter.keep?("你吃飯了嗎？我們等一下再出發。")    # => true
+
+# Ambiguous vocabulary is flagged for review, not discarded
+TWFilter.examine("這個視頻的畫質很不錯").marks.map(&:code)    # => [:mainland_soft]
+
+# Taiwanese reporting about China is Taiwan Mandarin; the topic is tagged, not rejected
+TWFilter.examine("行政院昨天回應了北京的最新聲明").marks.map(&:code)    # => [:foreign_topic]
+
+# 南京 is also a Taipei street; an address suffix suppresses the topic tag
+TWFilter.examine("臺北市南京東路五段的捷運站出口").findings    # => []
+```
+
+Provenance is decided on orthography, character inventory, lexical choice, morphology and
+a positive evidence signal, and every criterion that fired is reported.
+
+## What it does and does not do
+
+**Does** — decide provenance of a span of text; normalize punctuation to the Taiwan
+standard; segment into sentences; report a character-difficulty tier; report every reason
+for every decision.
+
+**Does not** — segment words, tag parts of speech, detect language (assume Han script
+input), translate, convert between simplified and traditional, or judge grammaticality.
+A span rejected here is not ungrammatical; it is *not Taiwanese*.
+
+## Applications
+
+1. **Learner-facing corpora.** Extracted from [TaiwanCards](https://taiwancards.com), where
+   it gates every sentence, gloss and collocation before it can reach a card. The
+   `publishable` policy encodes the boundary of the pedagogical inventory: MOE 常用 4 808
+   only, no ambiguous vocabulary, MOE punctuation only.
+2. **Web corpus construction.** Filtering crawl output where the language identifier
+   reports `zho_Hant` but the provenance is unknown. Traditional-script crawls conflate
+   four written standards; this separates them at 10⁴ sentences per second per core.
+3. **Training-set curation.** Building or auditing a monolingual set for a model intended
+   to produce Taiwan Mandarin rather than converted mainland text. `Report#findings` is
+   machine-readable, so exclusions stay attributable.
+4. **Editorial and localization QA.** Detecting mainland vocabulary, orthography and
+   punctuation in copy that is supposed to be Taiwanese — the `mainland_soft` band is
+   designed for review queues rather than automatic rejection.
+
+## Decision procedure
+
+```mermaid
+flowchart TD
+  T["input span"] --> N["Punctuation.normalize<br/>教育部《重訂標點符號手冊》"]
+  N --> SH
+
+  subgraph SH["Checks::Shape"]
+    A1["han count ∈ policy.han_range"]
+    A2["han ratio ≥ policy.min_han_ratio"]
+    A3["inventory ⊆ Han ∪ Latin ∪ Bopomofo ∪ punctuation"]
+  end
+
+  SH --> SC
+  subgraph SC["Checks::Script — character inventory"]
+    B1["simplified-only set<br/>OpenCC STCharacters ∖ TSCharacters ∖ Taiwan variants"]
+    B2["converted orthography 裏着衞爲説麽綫墻眞啓衆産囯麪"]
+    B3["MOE tier ≤ policy.max_tier"]
+  end
+
+  SC --> LX
+  subgraph LX["Checks::Lexicon, Erhua, Wenyan"]
+    C1["mainland hard ×149 → reject"]
+    C2["mainland soft ×38 → mark"]
+    C3["Cantonese characters ×17, HK/SEA lexis ×19 → reject"]
+    C4["foreign topics ×80 → mark"]
+    C5["erhua: 兒 outside a headed or tailed window → reject"]
+    C6["literary density > 0.05 → reject"]
+  end
+
+  LX --> EV["Evidence — Taiwan lexicon, particles, institutions,<br/>有 + V perfective outside negation"]
+  EV --> R["Report{ok?, findings[], tier, han, evidence}"]
+```
+
+Positive evidence never rejects. `了嗎` is not a defect: it is attested in every native
+source measured. `有` as a perfective marker counts in favour, never against — but only
+outside negation, since `沒有看` and `有沒有看` are pan-Mandarin and carry no Taiwan signal.
+
+## Input and output
+
+**Input** — anything responding to `to_s`. Empty, whitespace-only, Latin-only and
+supplementary-plane input are reported, not raised on. Invalid byte sequences must be
+scrubbed by the caller. There is no length limit; segmentation and normalization are
+linear in input length.
+
+**Output** — `TWFilter::Report`:
+
+| member | meaning |
+| --- | --- |
+| `ok?` | true iff no finding has severity `:reject` |
+| `findings` | every criterion that fired, in check order |
+| `rejects`, `marks` | the two severity partitions; their union is `findings` |
+| `tier` | highest MOE chart index reached: 0 常用, 1 次常用, 2 罕用, `nil` if unlisted |
+| `han` | Han character count |
+| `evidence` | count of positive Taiwan signals |
+| `reasons` | `rejects` rendered as `"code: detail"` strings |
+| `to_h` | plain data, JSON-serializable |
+
+A `Finding` is `{check, code, severity, detail}`. Codes are stable across releases;
+severity depends on the policy. Nothing is ever discarded silently — `keep?` is a
+convenience over `examine`, not a separate path.
+
+## Policies
+
+Immutable value objects. Derive with `#with`.
+
+| | `corpus` | `publishable` | `permissive` |
+| --- | --- | --- | --- |
+| han count | 6–60 | 6–40 | 1–∞ |
+| han ratio | ≥ 0.65 | ≥ 0.70 | 0 |
+| character tier | ≤ 罕用 | ≤ 常用 4 808 | ≤ 罕用 |
+| soft lexical markers | mark | reject | mark |
+| foreign topics | mark | reject | mark |
+| punctuation inventory | wide | MOE strict | wide |
+| converted orthography | reject | reject | reject |
+
+```ruby
+TWFilter::Policy.corpus.with(max_tier: :secondary, han_range: (4..80))
+```
+
+## Usage
+
+```ruby
+require "twfilter"
+
+# Provenance, with reasons
+report = TWFilter.examine("请把这个信息转发给同事")
+report.ok?                    # => false
+report.reasons                # => ["simplified: 请这个转", "unlisted: 请这个转", "mainland: 信息"]
+report.findings.map(&:code)   # => [:simplified, :unlisted, :mainland]
+
+# Editorial review rather than rejection
+report = TWFilter.examine("這個程式的運行狀況非常穩定")
+report.ok?                    # => true
+report.marks.map(&:detail)    # => ["運行"]
+TWFilter::Checks::Lexicon.taiwan_form("運行")   # => "執行"
+
+# Pedagogical gate
+TWFilter.keep?("他每天都會去公園散步運動。", policy: TWFilter::Policy.publishable)  # => true
+TWFilter.keep?("饕餮紋飾在青銅器上十分常見。", policy: TWFilter::Policy.publishable)  # => false
+
+# Punctuation to the Taiwan standard
+TWFilter.normalize("他說“好”...")        # => "他說「好」……"
+TWFilter.normalize("馬丁·路德ＰＤＦ")     # => "馬丁‧路德PDF"
+
+# Segmentation: 。！？…； terminate, closers stay attached, 、 does not break
+TWFilter.sentences("他說：「好。」我走了。")   # => ["他說：「好。」", "我走了。"]
+
+# Character difficulty
+TWFilter.tier("散步")   # => 0
+TWFilter.tier("饕餮")   # => 1
+
+# Block judgement — a paragraph is accepted as a unit or not at all
+TWFilter::Block.judge(paragraph_sentences)
+
+# Table fingerprint, to record beside derived measurements
+TWFilter.provenance
+# => {version: "1.0.0", dir: "…/lib/twfilter/data", tables: 19, digest: "14cd9fd7fc97dc9a"}
+```
+
+## Placement in a pipeline
+
+Normalize before segmenting: the ellipsis and dash rules change sentence boundaries.
+Segment before examining: the shape thresholds are defined on sentences. Examine before
+deduplicating and counting, so rejected material never reaches the statistics.
+
+```text
+acquire → normalize → segment → examine → deduplicate → tokenize → count
+             │           │         │
+       Punctuation   Sentences   Report
+```
+
+`Block.judge` is an optional stage between segmentation and examination for sources whose
+documents are topically coherent: it rejects a whole window when any member fails, and
+requires a minimum density of positive evidence across the window. Use it on crawls, not
+on dictionaries.
+
+## Performance
+
+Single core, Apple M5 Max, on 199 936 sentences of Taiwanese web text (6.2 M characters):
+
+| operation | throughput | per 10⁷ sentences |
+| --- | ---: | ---: |
+| `Sentences.split` | 741 000 /s · 68 MB/s | 13 s |
+| `Punctuation.normalize` | 67 000 /s · 6.1 MB/s | 149 s |
+| `tier` | 134 000 /s · 12 MB/s | 75 s |
+| `examine` (all five checks) | **10 700 /s · 1.0 MB/s** | 934 s |
+
+Suitable for corpora of arbitrary size. Cost is linear in input length and dominated by
+`Checks::Lexicon`, which scans 264 union-compiled patterns per span; the other four checks
+together account for under a third of the total. Memory is constant — tables load once
+(≈ 3 MB resident) and every entry point is a pure function over a string, so the work
+parallelizes by `fork` or by thread without coordination.
+
+A 37.7 M-sentence corpus takes ≈ 3.5 minutes of wall time across 18 cores. In practice
+I/O, JSON parsing and deduplication dominate; this library is not usually the bottleneck.
+
+## Reference tables
+
+`lib/twfilter/data/` — plain text, one record per line, tab-separated where a record has
+fields. Diffable, reviewable and versioned with the gem.
+
+| table | rows | content |
+| --- | ---: | --- |
+| `moe_common/secondary/rare.txt` | 4 808 / 6 343 / 18 356 | 教育部標準字體表 |
+| `moe_exception.txt` | 12 | dictionary headword characters outside all three charts |
+| `simplified_only.txt` | 3 785 | derived from the OpenCC round-trip |
+| `variants_used_in_taiwan.txt` | 16 | simplified-looking forms standard in Taiwan |
+| `converted_orthography.txt` | 14 | mainland traditional variants |
+| `mainland_hard.tsv` | 149 | rejecting pairs, with attestation counts |
+| `mainland_soft.tsv` | 38 | marking pairs, ambiguous or dominated |
+| `mainland_exceptions.tsv` | 6 | superstrings in which a marker is legitimate |
+| `cantonese.txt` | 17 | written-Cantonese characters outside the 常用 chart |
+| `regional_hard.tsv` | 19 | Hong Kong, Cantonese-sequence and Singapore forms |
+| `foreign_topics.txt` | 80 | non-Taiwanese toponyms and institutions — subject matter, not provenance |
+| `taiwan_markers/lexicon/particles/grammar.txt` | 66 / 51 / 7 / 20 | positive evidence |
+| `erhua_headed/tailed.txt` | 14 / 36 | 兒 as a root morpheme |
+| `wenyan.txt` | 4 | literary particles for density estimation |
+| `MANIFEST.json` | — | row count and SHA-256 per table |
+
+**Grading.** Candidate pairs are graded by attestation ratio against a reference corpus of
+ten natively Taiwanese sources, 11 923 110 characters: *hard* when the mainland form is
+unattested and the Taiwan form occurs at least three times; *soft* when the Taiwan form
+dominates by 20 : 1 or the pair is hand-graded ambiguous; excluded otherwise. Most candidates
+are excluded — conversion tables built for orthographic mapping contain many pairs where both
+forms are ordinary Taiwanese.
+
+The reference corpus admits no source with an open authorship gate. Open web comments on
+Taiwanese channels are Taiwan-hosted but not Taiwan-authored: every mainland form attested
+anywhere in the corpus occurs there and only there, so counting them would license the
+mainland vocabulary of mainland commenters as Taiwanese.
+
+**Membership.** Each table states the rule that admits a row, and `spec/twfilter/membership_spec.rb`
+enforces it against the shipped data:
+
+| table | a row is admitted iff |
+| --- | --- |
+| `cantonese.txt` | the character is used in written Cantonese and is not in 常用國字標準字體表 |
+| `regional_hard.tsv` | the form is unattested in Taiwan and its Taiwan counterpart occurs ≥ 3 times |
+| `converted_orthography.txt` | the form is in no MOE chart and the Taiwan form outnumbers it ≥ 20 : 1 |
+| `variants_used_in_taiwan.txt` | the form looks simplified but is in an MOE chart |
+| `mainland_hard.tsv` | the mainland form is unattested and the Taiwan form occurs ≥ 3 times |
+| `mainland_soft.tsv` | the pair is a real opposition but the mainland form is attested here |
+| `mainland_exceptions.tsv` | the head term is a detector in one of the two bands |
+| `taiwan_lexicon.txt` | a mainland writer would not use this string for this referent |
+| `taiwan_markers.txt` | the name denotes an ROC institution or a Taiwanese place |
+| `taiwan_particles.txt` | the graph is the Taiwan-preferred spelling of the particle |
+| `taiwan_grammar.txt` | the construction is a Southern Min substrate feature |
+
+The rules exclude more than they admit. Subject matter is not provenance, so Taiwanese food
+names are not evidence; A-not-A interrogatives are pan-Mandarin, so `好不好` is not evidence;
+and a character in everyday Taiwanese use cannot mark another variety, however
+characteristic of it that character may be.
+
+**Topics are not provenance.** `foreign_topics.txt` is a subject-matter tag. Taiwanese media
+covers the mainland constantly, several entries are ordinary ROC administrative vocabulary,
+and Taipei streets are named after mainland cities. It marks under `corpus` and rejects only
+under `publishable`, where excluding foreign subject matter is an editorial decision.
+
+## Influencing the criteria
+
+Preferred: contribute to the published dataset. The tables are maintained at
+[huggingface.co/datasets/taiwan-corpora/twfilter-tables](https://huggingface.co/datasets/taiwan-corpora/twfilter-tables),
+where each release carries its grading evidence, and the gem vendors a copy of a specific
+version. A correction accepted there reaches every consumer at the next release, with a
+record of what changed and why — which a local edit does not.
+
+Three table sources are supported. All three are reported by `provenance`.
+
+**Bundled** — the default, versioned with the gem, the only configuration whose results are
+comparable with anyone else's.
+
+**Replacement** — a directory holding the full set, e.g. a checkout of the dataset newer
+than the gem:
+
+```ruby
+TWFilter::Tables.dir = "path/to/twfilter-tables"   # or TWFILTER_DATA_DIR
+```
+
+**Overlay** — the bundled set plus your additions. Filename suffixes select the operation:
+
+| file in the overlay directory | effect on `mainland_hard.tsv` |
+| --- | --- |
+| `mainland_hard.add.tsv` | rows appended |
+| `mainland_hard.remove.tsv` | rows removed, matched on the full row or its first field |
+| `mainland_hard.tsv` | table replaced |
+
+```ruby
+TWFilter::Tables.overlay = "path/to/house-style"    # or TWFILTER_OVERLAY_DIR
+```
+
+Row format is validated on load and a violation raises `TWFilter::InvalidTableError`:
+
+| shape | tables | row |
+| --- | --- | --- |
+| `:chars` | the MOE charts, `simplified_only`, `variants_used_in_taiwan`, `converted_orthography`, `cantonese`, `wenyan` | one Han character |
+| `:pairs` | `mainland_hard`, `mainland_soft`, `mainland_exceptions` | two or more tab-separated fields, first non-empty |
+| `:terms` | everything else, including tables you invent | one term, no tabs |
+
+Files named `*.remove.*` carry keys only and are not shape-checked. UTF-8, one record per
+line, no header, no comments; blank lines ignored.
+
+Record `provenance` alongside any measurement derived this way. Two corpora filtered with
+different tables are not comparable, and the digest is what makes that detectable.
+
+Policy thresholds are the supported knob and need no table changes:
+
+```ruby
+lenient = TWFilter::Policy.corpus.with(soft_lexicon_rejects: false, foreign_topics_reject: false)
+```
+
+## Development
+
+```bash
+bundle install
+bundle exec rake          # specs and RBS validation
+bundle exec rake tables   # regenerate the tables and their manifest
+```
+
+Specs require no data, no network and no credentials.
+
+## License
+
+Code: MIT. Tables: mixed, all permitting commercial use. See [NOTICES.md](NOTICES.md) for
+per-table provenance and terms.
+
+Two attribution strings must be reproduced by anything shipping this gem:
+
+```text
+數位發展部，CNS11643中文標準交換碼全字庫網站
+OpenCC — https://github.com/BYVoid/OpenCC, Apache License 2.0
+```
+
+No table carries a NonCommercial, ShareAlike or copyleft term. Verdicts and counts produced
+by the gem are the caller's, not derivatives of the tables.

data/lib/twfilter/block.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module TWFilter
+  module Block
+    # Sentences per judged window, accepted or rejected as a unit. 300 is large enough for
+    # the evidence rate to be stable and small enough to stay within one document.
+    SIZE = 300
+
+    module_function
+
+    def judge(window, policy: Policy.corpus)
+      return [] if window.empty?
+
+      reports = window.map { |line| TWFilter.examine(line, policy: policy) }
+      return [] if reports.count { |report| !report.ok? } > policy.block_tolerance * window.length
+      return [] if Evidence.count(window.join) < policy.evidence_per_100 * window.length / 100
+
+      reports.filter_map { |report| report.text if report.ok? }
+    end
+
+    def each_kept(lines, size: SIZE, policy: Policy.corpus)
+      return to_enum(:each_kept, lines, size: size, policy: policy) unless block_given?
+
+      window = []
+      lines.each do |line|
+        window << line
+        next if window.length < size
+
+        judge(window, policy: policy).each { |kept| yield kept }
+        window = []
+      end
+
+      judge(window, policy: policy).each { |kept| yield kept } if window.any?
+    end
+  end
+end

data/lib/twfilter/checks/erhua.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module TWFilter
+  module Checks
+    module Erhua
+      NAME = :erhua
+      # 兒 is a root morpheme inside a headed or tailed window, and an erhua suffix elsewhere.
+      MARK = "兒"
+
+      class << self
+        def call(subject)
+          offender = offender(subject.text)
+          return [] if offender.nil?
+
+          [Finding.new(check: NAME, code: :erhua, detail: offender)]
+        end
+
+        def offender(text)
+          return nil if text == MARK
+
+          position = 0
+          while (position = text.index(MARK, position))
+            after = text[position, 2]
+            before = position.positive? ? text[position - 1, 2] : nil
+
+            return before || MARK unless headed.include?(after) || tailed.include?(before)
+
+            position += 1
+          end
+
+          nil
+        end
+
+        def headed = Tables.set("erhua_headed.txt")
+
+        def tailed = Tables.set("erhua_tailed.txt")
+      end
+    end
+  end
+end