swear_jar 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +22 -0
- data/LICENSE.txt +22 -0
- data/README.md +313 -0
- data/config/locales/en.yml +4 -0
- data/config/locales/pl.yml +4 -0
- data/lib/swear_jar/configuration.rb +56 -0
- data/lib/swear_jar/detector.rb +58 -0
- data/lib/swear_jar/entry.rb +82 -0
- data/lib/swear_jar/errors.rb +7 -0
- data/lib/swear_jar/fold_table.rb +115 -0
- data/lib/swear_jar/match.rb +10 -0
- data/lib/swear_jar/matcher.rb +289 -0
- data/lib/swear_jar/normalizer.rb +163 -0
- data/lib/swear_jar/pattern_builder.rb +221 -0
- data/lib/swear_jar/railtie.rb +36 -0
- data/lib/swear_jar/skeleton.rb +116 -0
- data/lib/swear_jar/slicer.rb +58 -0
- data/lib/swear_jar/swear_jar_validator.rb +44 -0
- data/lib/swear_jar/version.rb +5 -0
- data/lib/swear_jar/word_list.rb +167 -0
- data/lib/swear_jar.rb +27 -0
- data/locales/en/words.yml +266 -0
- data/locales/pl/words.yml +213 -0
- metadata +67 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 1eb1d6cca9742143877d91065075b28e5d8e80b36a83b9fc0c29e5c6b6bbfaa0
|
|
4
|
+
data.tar.gz: d802c3f370fad5cba13b2d435b898510b4600b34df902df5db6b2b5522045256
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 2936ce72650f14af1a261db23c9000ec61d3ad062870d727ecab6d2a25dbc1c7a657136cf1dea4ae58aab4d57aab3e8e0328486bf770c287a240277a3cea3f1e
|
|
7
|
+
data.tar.gz: 6b2c2e29edbc353e330b911fc146eb8877f0d413eff0430e2a6320c1f387bacc08e7349a575cf198dc5018f62ce41c129f352475ade1882ece741acfe38c78ab
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 - Unreleased
|
|
4
|
+
|
|
5
|
+
- Require Ruby 3.0+, no runtime dependencies
|
|
6
|
+
- Add `SwearJar::Detector` with `profane?` and `matches`, and the frozen `SwearJar::Match` value object (`text`, character `range`, `stem`, `locale`, `severity`, `categories`)
|
|
7
|
+
- Add Polish (`:pl`) and English (`:en`) word lists with severities (`mild`, `strong`, `severe`), categories (`vulgar`, `sexual`, `insult`, `slur`, `religious`), variants, Polish prefixes and allowlists
|
|
8
|
+
- Add detector options: `locales`, `mode` (`:words`, `:embedded`), `min_severity`, `categories`, `extra_paths`, `extra_words` and `allow`
|
|
9
|
+
- Detect obfuscation: case, missing diacritics, fullwidth and other compatibility characters, leetspeak, repeated letters, separators inside a word, spaced letters and masked letters
|
|
10
|
+
- Treat masks at the edge of a word as markers, not masked letters: `**kurwa**`, `*kurwa*`, `#kurwa` report just the word, and `censor` keeps the markers
|
|
11
|
+
- Add `Detector#censor` (one `*` per character by default, a `with:` string or a block per match)
|
|
12
|
+
- Add `SwearJar.configure`, `SwearJar.detector`, `SwearJar.profane?`, `SwearJar.matches` and `SwearJar.censor`
|
|
13
|
+
- Add a Railtie with `config.swear_jar`, and `SwearJarValidator` (`validates :name, swear_jar: true`) with `en` and `pl` messages under `errors.messages.profane`
|
|
14
|
+
- Detect invisible characters inside a word (zero-width space and joiners, soft hyphen, byte order mark and other format characters), several masks for a doubled letter (`a**hole`), spaced letters right after a one-letter word (`a k u r w a`; exact entries only after `a i o u w z`, so `p a s s` stays clean), and in `:words` mode words separated only by an invisible character (`you\u200Bidiot`)
|
|
15
|
+
- Extend a match over combining marks after its last letter; `Match#text` is always a plain `String`, also for an `ActiveSupport::SafeBuffer` input
|
|
16
|
+
- Run in linear time on adversarial input: many hits, long runs of letters or leet symbols, one long glued word, non-ASCII text
|
|
17
|
+
- `SwearJarValidator` keeps the `:profane` error type with a custom `message:`
|
|
18
|
+
- Raise `ArgumentError` for empty `locales` or `categories`, and `InvalidWordListError` for an `extra_paths` entry without wildcards that names no file, for an entry with a non-string key, and for a stem, variant or allow word with a digit or without letters, naming the file and entry; accept a single String for `allow`
|
|
19
|
+
- The Railtie builds `SwearJar.detector` at boot, so preloading app servers share it
|
|
20
|
+
- Compact `inspect` for `Detector` and the internal classes, which are marked `@api private`
|
|
21
|
+
- Add `rake bench` (a clean ~200-character message averages under 50 µs in `:words` mode)
|
|
22
|
+
- Add RuboCop configuration and GitHub Actions CI workflow
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
Copyright (c) 2026 Konrad Jurkowski
|
|
2
|
+
|
|
3
|
+
MIT License
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
|
6
|
+
a copy of this software and associated documentation files (the
|
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
|
11
|
+
the following conditions:
|
|
12
|
+
|
|
13
|
+
The above copyright notice and this permission notice shall be
|
|
14
|
+
included in all copies or substantial portions of the Software.
|
|
15
|
+
|
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,313 @@
|
|
|
1
|
+
# SwearJar
|
|
2
|
+
|
|
3
|
+
Fast profanity detector for Polish (`:pl`) and English (`:en`) text. It handles Polish inflection and common obfuscation (missing diacritics, leetspeak, repeated letters, spaced or masked letters) and returns every match with its exact range in the original text.
|
|
4
|
+
|
|
5
|
+
SwearJar only detects. Your application decides what to do with the matches (censor them, reject a name, count offences). It works as a plain Ruby library and integrates with Rails when Rails is present. Requires Ruby 3.0+. No runtime dependencies.
|
|
6
|
+
|
|
7
|
+
## Installation
|
|
8
|
+
|
|
9
|
+
Add this line to your application's Gemfile:
|
|
10
|
+
|
|
11
|
+
```ruby
|
|
12
|
+
gem 'swear_jar'
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
And then execute:
|
|
16
|
+
|
|
17
|
+
```sh
|
|
18
|
+
bundle install
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Or install it yourself as:
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
gem install swear_jar
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## Usage
|
|
28
|
+
|
|
29
|
+
```ruby
|
|
30
|
+
require 'swear_jar'
|
|
31
|
+
|
|
32
|
+
detector = SwearJar::Detector.new
|
|
33
|
+
|
|
34
|
+
detector.profane?('ty k u r r w o!') # => true (stops at the first match)
|
|
35
|
+
detector.profane?('skurcz łydki') # => false
|
|
36
|
+
|
|
37
|
+
detector.matches('Ty k u r r w o! Co za wkurwiony dzień.')
|
|
38
|
+
# => [#<struct SwearJar::Match text="k u r r w o", range=3...14, stem="kurw", locale=:pl,
|
|
39
|
+
# severity=:strong, categories=[:vulgar]>,
|
|
40
|
+
# #<struct SwearJar::Match text="wkurwiony", range=22...31, stem="kurw", locale=:pl,
|
|
41
|
+
# severity=:strong, categories=[:vulgar]>]
|
|
42
|
+
|
|
43
|
+
# Module-level shortcuts use a shared detector built with the defaults (see Configuration)
|
|
44
|
+
SwearJar.profane?('What the fuck') # => true
|
|
45
|
+
SwearJar.matches('What the fuck')
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
`nil` and `''` give `[]` and `false`. A Symbol or any other object is converted with `to_s`. The input string is never modified.
|
|
49
|
+
|
|
50
|
+
Detection takes time linear in the length of the input, but a long input still costs more: cap the length before you call the detector (see [Long input](#long-input)).
|
|
51
|
+
|
|
52
|
+
A `Detector` is frozen after `new` and safe to share between threads. All word-list loading and pattern compilation happens in `new` (about 10 ms with both bundled lists), so build a detector once and reuse it. To change the word list, build a new detector and swap the reference.
|
|
53
|
+
|
|
54
|
+
### Detector options
|
|
55
|
+
|
|
56
|
+
```ruby
|
|
57
|
+
SwearJar::Detector.new(
|
|
58
|
+
locales: %i[pl en],
|
|
59
|
+
mode: :words,
|
|
60
|
+
min_severity: :mild,
|
|
61
|
+
categories: nil,
|
|
62
|
+
extra_paths: ['config/swear_jar/*.yml'],
|
|
63
|
+
extra_words: [{ stem: 'smoczydup', locale: :pl, severity: :strong, categories: [:insult] }],
|
|
64
|
+
allow: %w[dupleks]
|
|
65
|
+
)
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
| option | default | meaning |
|
|
69
|
+
|---|---|---|
|
|
70
|
+
| `locales` | `%i[pl en]` | bundled word lists to load |
|
|
71
|
+
| `mode` | `:words` | `:words` or `:embedded`, see [Modes](#modes) |
|
|
72
|
+
| `min_severity` | `:mild` | `:mild`, `:strong` or `:severe`; `:strong` keeps `strong` and `severe` entries |
|
|
73
|
+
| `categories` | `nil` (all) | keep entries that have at least one of these: `:vulgar`, `:sexual`, `:insult`, `:slur`, `:religious` |
|
|
74
|
+
| `extra_paths` | `[]` | glob patterns of extra word-list YAML files |
|
|
75
|
+
| `extra_words` | `[]` | extra entries as hashes; each needs a `locale:` |
|
|
76
|
+
| `allow` | `[]` | extra allowlisted words |
|
|
77
|
+
|
|
78
|
+
An unknown locale, mode, severity or category raises `ArgumentError`, and so does an empty `locales` or `categories` list (it would find nothing). A malformed word list raises `SwearJar::InvalidWordListError` with the file and entry in the message. An `extra_paths` entry without wildcards (`*`, `?`, `[`, `{`) must name an existing file, otherwise it raises `InvalidWordListError`; a glob may match nothing. `allow` takes an Array or a single String.
|
|
79
|
+
|
|
80
|
+
### `SwearJar::Match`
|
|
81
|
+
|
|
82
|
+
A frozen `Struct`:
|
|
83
|
+
|
|
84
|
+
| field | example | meaning |
|
|
85
|
+
|---|---|---|
|
|
86
|
+
| `text` | `"k u r r w o"` | the matched fragment of the string |
|
|
87
|
+
| `range` | `3...14` | character (not byte) range, so `string[match.range] == match.text` |
|
|
88
|
+
| `stem` | `"kurw"` | canonical stem from the word list |
|
|
89
|
+
| `locale` | `:pl` | |
|
|
90
|
+
| `severity` | `:strong` | `:mild`, `:strong` or `:severe` |
|
|
91
|
+
| `categories` | `[:vulgar]` | |
|
|
92
|
+
|
|
93
|
+
Matches are sorted by position and never overlap. A match covers the whole word that contains the stem (`wkurwiony`, not `kurw`), including combining marks after its last letter, but without punctuation around it (`kurwa!!!` gives `kurwa`). One word gives at most one match: the highest severity wins, then the longest stem. `text` is always a plain `String`, also when the input is a subclass such as Rails' `ActiveSupport::SafeBuffer`.
|
|
94
|
+
|
|
95
|
+
Every profane word is a separate match, so `kurwa kurwa kurwa` gives 3 matches. To count offences per message (for example, mute after 3 offences in 10 minutes), count messages with at least one match (`profane?`), not matches, or one long message counts as many offences.
|
|
96
|
+
|
|
97
|
+
`range` indexes the string after UTF-8 coercion. For valid UTF-8 input (the normal case) that is the input itself. If your string has invalid bytes or another encoding, coerce it yourself first, the same way the detector does, and use the ranges on the coerced string:
|
|
98
|
+
|
|
99
|
+
```ruby
|
|
100
|
+
text = text.encode('UTF-8', invalid: :replace, undef: :replace, replace: '') unless text.encoding == Encoding::UTF_8
|
|
101
|
+
text = text.scrub('')
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### Censoring
|
|
105
|
+
|
|
106
|
+
`censor` returns a copy of the text with every match replaced. By default each match becomes one `*` per character; `with:` gives a fixed replacement, and a block gets each `SwearJar::Match` and returns its replacement:
|
|
107
|
+
|
|
108
|
+
```ruby
|
|
109
|
+
text = 'Ty k u r r w o! Co za wkurwiony dzień.'
|
|
110
|
+
detector.censor(text) # => "Ty ***********! Co za ********* dzień."
|
|
111
|
+
detector.censor(text, with: '[bleep]') # => "Ty [bleep]! Co za [bleep] dzień."
|
|
112
|
+
detector.censor('dupa i kurwa') { |match| match.severity == :mild ? match.text : '[bleep]' }
|
|
113
|
+
# => "dupa i [bleep]"
|
|
114
|
+
SwearJar.censor(text, with: '[bleep]') # the global detector
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
The input is never modified, `nil` returns `nil`, and a clean text comes back as an unchanged copy. Passing both `with:` and a block raises `ArgumentError`. The result is plain text, not HTML.
|
|
118
|
+
|
|
119
|
+
To censor at display time in a Rails view with markup (`<em>`), split the raw text at the match ranges and let `safe_join` escape each piece, with markup only between them:
|
|
120
|
+
|
|
121
|
+
```ruby
|
|
122
|
+
# app/helpers/chat_helper.rb
|
|
123
|
+
def censored(text)
|
|
124
|
+
pieces = []
|
|
125
|
+
pos = 0
|
|
126
|
+
SwearJar.matches(text).each do |match|
|
|
127
|
+
pieces << text[pos...match.range.begin] << tag.em('[bleep]')
|
|
128
|
+
pos = match.range.end
|
|
129
|
+
end
|
|
130
|
+
safe_join(pieces << text[pos..])
|
|
131
|
+
end
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Do not insert tags into the raw string and call `html_safe` on the result: the rest of the user's text would then be rendered unescaped (an XSS hole).
|
|
135
|
+
|
|
136
|
+
### Modes
|
|
137
|
+
|
|
138
|
+
- `:words` (default, intended for chat): a stem matches only at the start of a word, optionally after a Polish prefix (`wkurwiony`, `spierdolić`, `zajebać`). English entries take no prefixes.
|
|
139
|
+
- `:embedded` (intended for character names and logins): a stem also matches inside a word (`xxchujxx`, `KurwiMistrz`). Entries marked `embeddable: false` still need a word boundary.
|
|
140
|
+
|
|
141
|
+
```ruby
|
|
142
|
+
SwearJar::Detector.new.profane?('xxchujxx') # => false
|
|
143
|
+
SwearJar::Detector.new(mode: :embedded).profane?('xxchujxx') # => true
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
### What is detected
|
|
147
|
+
|
|
148
|
+
Every message goes through a normalizer before matching:
|
|
149
|
+
|
|
150
|
+
- case and Unicode compatibility forms: `KURWA`, fullwidth `kurwa`
|
|
151
|
+
- Polish diacritics, written or not, and other combining marks: `gówno` and `gowno`
|
|
152
|
+
- leetspeak: `a` = `4 @`, `o` = `0`, `i` = `1 ! |`, `e` = `3`, `s` = `$ 5`, `t` = `7`, `l` = `1 |`, `b` = `8`, `g` = `9`
|
|
153
|
+
- repeated letters: `kuuuurwa`
|
|
154
|
+
- separators inside a word: `kur.wa`, `kur-wa`
|
|
155
|
+
- spaced letters: `k u r w a`, `k.u.r.w.a`, `k-u-r-w-a` (three or more single characters, one repeated separator), also right after a one-letter word that the run takes in: `a k u r w a` reports `k u r w a`. In `:words` mode, an exact entry (`ass`) is split off only after a one-letter word (`a`, `i`, `o`, `u`, `w`, `z`), so `i a s s` is reported and `p a s s` is not
|
|
156
|
+
- masked letters: `k*rwa`, `k#rwa`, `k%rwa`, `k_rwa` (`_` only between letters), and several masks for a doubled letter: `a**hole`, `pu**y`. Masks count as letters only between two letters: at the edge of a word they are markers, so `**kurwa**`, `*kurwa*`, `~~chuj~~` and `#kurwa` report just the word, and `censor` keeps the markers (`censor('**kurwa** mac', with: 'X')` → `'**X** mac'`)
|
|
157
|
+
- invisible characters inside a word, which are ignored: zero-width space, zero-width (non-)joiner, word joiner, soft hyphen, byte order mark and other format characters. They render as nothing, so `ku\u200Brwa` is one word; in `:words` mode the letter after them is also tried as a word start, so `you\u200Bidiot` reports `idiot`
|
|
158
|
+
|
|
159
|
+
## Rails
|
|
160
|
+
|
|
161
|
+
With Rails loaded, SwearJar adds `config.swear_jar` and an ActiveModel validator.
|
|
162
|
+
|
|
163
|
+
### Configuration
|
|
164
|
+
|
|
165
|
+
Set options in `config/application.rb`:
|
|
166
|
+
|
|
167
|
+
```ruby
|
|
168
|
+
config.swear_jar.locales = %i[pl en]
|
|
169
|
+
config.swear_jar.min_severity = :mild
|
|
170
|
+
config.swear_jar.extra_paths = [Rails.root.join('config/swear_jar/*.yml').to_s]
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
Or in an initializer (this also works without Rails):
|
|
174
|
+
|
|
175
|
+
```ruby
|
|
176
|
+
# config/initializers/swear_jar.rb
|
|
177
|
+
SwearJar.configure do |config|
|
|
178
|
+
config.locales = %i[pl en]
|
|
179
|
+
config.mode = :words
|
|
180
|
+
config.allow = %w[dupleks]
|
|
181
|
+
end
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Use one of the two. `config.swear_jar` is applied after the initializers run, and each `configure` call starts from the defaults, so the later one replaces the earlier one. The Railtie then builds `SwearJar.detector` during boot, so a server that preloads the app (Puma `preload_app!`) shares one detector between its workers. `configure` accepts the same options as `Detector.new`. It builds a new detector first, so invalid options raise and leave the current one in place, and then swaps it in under a mutex. `SwearJar.detector`, `SwearJar.profane?`, `SwearJar.matches` and `SwearJar.censor` use that detector. Without configuration it uses the defaults. An unknown `config.swear_jar` key raises `ArgumentError` at boot.
|
|
185
|
+
|
|
186
|
+
### Validator
|
|
187
|
+
|
|
188
|
+
```ruby
|
|
189
|
+
class Character < ApplicationRecord
|
|
190
|
+
validates :name, swear_jar: { mode: :embedded }
|
|
191
|
+
validates :bio, swear_jar: true
|
|
192
|
+
validates :motto, swear_jar: { min_severity: :strong, message: :inappropriate }
|
|
193
|
+
end
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
`swear_jar: true` uses `SwearJar.detector`. Any detector option (`mode`, `locales`, `min_severity`, `categories`, `extra_paths`, `extra_words`, `allow`) gives the validator its own detector, built once per option set and cached. Standard validator options (`message`, `if`, `unless`, `on`, `allow_nil`, `allow_blank`, `strict`) work as usual. Blank values are skipped.
|
|
197
|
+
|
|
198
|
+
The error type is `:profane` (`record.errors.added?(:name, :profane)`), with the message key `errors.messages.profane` and translations for `en` ("contains inappropriate language") and `pl` ("zawiera niedozwolone słowa"). Override it in your own locale files. A custom `message:` (a String, a Symbol translation key or a Proc) replaces only the text; the error type stays `:profane`.
|
|
199
|
+
|
|
200
|
+
Rails runs every validator of an attribute, even after an earlier one has failed, so a `length:` validation does not keep a huge value away from the detector. Skip the check for values that are too long anyway:
|
|
201
|
+
|
|
202
|
+
```ruby
|
|
203
|
+
validates :name, length: { maximum: 30 }, swear_jar: { if: -> { name.to_s.length <= 30 } }
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
## Word lists
|
|
207
|
+
|
|
208
|
+
The bundled lists are in `locales/pl/words.yml` and `locales/en/words.yml`. Extra files use the same format plus a top-level `locale:`:
|
|
209
|
+
|
|
210
|
+
```yaml
|
|
211
|
+
locale: pl
|
|
212
|
+
prefixes: [bez] # Polish only, optional
|
|
213
|
+
words:
|
|
214
|
+
- stem: kurw
|
|
215
|
+
variants: [kurew, kurest] # other spellings that report the same stem
|
|
216
|
+
severity: strong
|
|
217
|
+
categories: [vulgar]
|
|
218
|
+
- stem: ass
|
|
219
|
+
severity: mild
|
|
220
|
+
categories: [vulgar]
|
|
221
|
+
embeddable: false # never inside a word, even in :embedded mode
|
|
222
|
+
exact: true # whole word only: no prefixes, no suffixes
|
|
223
|
+
allow:
|
|
224
|
+
- duplikat # an innocent word that contains a stem
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
| key | required | default | meaning |
|
|
228
|
+
|---|---|---|---|
|
|
229
|
+
| `stem` | yes | | canonical stem, written normally (with diacritics, no leetspeak or digits) |
|
|
230
|
+
| `variants` | no | `[]` | other spellings that report the same `stem` |
|
|
231
|
+
| `severity` | yes | | `mild`, `strong` or `severe` |
|
|
232
|
+
| `categories` | yes | | one or more of `vulgar`, `sexual`, `insult`, `slur`, `religious` |
|
|
233
|
+
| `embeddable` | no | `true` | may match inside a word in `:embedded` mode |
|
|
234
|
+
| `exact` | no | `false` | must be the whole word |
|
|
235
|
+
|
|
236
|
+
Stems go through the same normalizer as the input. A stem matches with any ending, so `kurw` covers `kurwa`, `kurwy` and `kurwiszon`. An allow word cancels a match when it is found in the same word and covers the stem (`duplikat` covers `dup`).
|
|
237
|
+
|
|
238
|
+
To extend the lists, point `extra_paths` at your own files, or pass `extra_words` and `allow`:
|
|
239
|
+
|
|
240
|
+
```ruby
|
|
241
|
+
SwearJar::Detector.new(
|
|
242
|
+
extra_paths: ['config/swear_jar/*.yml'],
|
|
243
|
+
extra_words: [{ stem: 'smoczydup', locale: :pl, severity: :strong, categories: [:insult] }],
|
|
244
|
+
allow: %w[dupleks]
|
|
245
|
+
)
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
Extra files and words for a locale that is not in `locales` are ignored.
|
|
249
|
+
|
|
250
|
+
## Known limitations
|
|
251
|
+
|
|
252
|
+
- A mask (`*`, `#`, `%`, `_`) stands for one letter of the stem, or for up to as many letters as a run of the same letter has (`a*hole` and `a**hole` for `asshole`), and only for an interior one: the stem's first and last letters must be written. With the stem `kurw`, `k*rwa` and `k**wa` match, but `*urwa` (the `*` is read as a marker before `urwa`) and `k***a` do not.
|
|
253
|
+
- When spaced letters follow a one-letter word (`a k u r w a`), only the first character is split off again: after two one-letter words (`i a k u r w a`) the stem is missed in `:words` mode. `:embedded` mode does not split it off at all, so an exact or non-embeddable entry after a swallowed letter (`i a s s`) is missed there; non-exact stems (`a k u r w a`) are found anyway.
|
|
254
|
+
- A word glued to a profane one by an invisible character is reported together with it when the scanner sees the profane part first (`kurwa\u200Bmać` is one match). A Polish prefix before the invisible character counts as part of the word: `za\u200Bchuj` reports `za\u200Bchuj`.
|
|
255
|
+
- Spaced letters need at least three single characters (`k u r w a`, but not `ku r wa`). This keeps Polish one-letter words (`i`, `w`, `z`, `a`) apart.
|
|
256
|
+
- `1` and `|` stand for both `i` and `l`. When a stem has an `i` right before an `l`, the first run takes the shared character: `d1ldo` matches, `di1do` does not.
|
|
257
|
+
- Allow words are single words. A phrase such as `bastard sword` cannot be allowlisted.
|
|
258
|
+
- `:words` mode does not find a stem in the middle of a compound (`horseshit`, `clusterfuck`). Only listed compounds such as `bullshit`, `jackass` and `motherfuck` are caught.
|
|
259
|
+
|
|
260
|
+
False positives found while writing the lists. They cannot be fixed in the data without missing real profanity:
|
|
261
|
+
|
|
262
|
+
- `pedał` (bicycle pedal) is reported, because it is the same word as the slur. `pedałować` (to pedal) is allowed. English `pedal` and `pedals` are reported too.
|
|
263
|
+
- English `dupe` and `duped` (Polish `dupę` without diacritics is `dupe`), and `Dupont`.
|
|
264
|
+
- `pierdoła` (`pierdol`), `Matołek` (`matoł`), `palant` (a bat game, same word as the insult), `Jebel` (`jeb`), `Sukkot` (`suk`), Czech `sucet` (`suce`) and the surname `Ciotti` (`ciot`).
|
|
265
|
+
- `cocky`, the name `Dick`, `Cumming`, `tit` (the bird), French `douche`, British `fag` (cigarette), `cholera` (the disease), `summa cum laude`, `pussy willow`, and `the damned`.
|
|
266
|
+
- `:embedded` only: `bipedal` and `Ospedale` (`pedał`), and Japanese names such as `Yoshitsune` and `Yoshitaka` (`shit`). `Matsushita` is allowlisted.
|
|
267
|
+
|
|
268
|
+
False negatives:
|
|
269
|
+
|
|
270
|
+
- `pedałów` and `spedałowany`, because the allow word `pedałow` (for `pedałować`) covers them.
|
|
271
|
+
- Forms of the slur `ciota` that are spelled like forms of `ciotka` (`ciotek`, `ciotce`), which are allowed.
|
|
272
|
+
|
|
273
|
+
## Performance
|
|
274
|
+
|
|
275
|
+
A clean message costs one regexp scan over its normalized form. Entry identification, allowlist checks and the original-text index map run only around hits.
|
|
276
|
+
|
|
277
|
+
`bundle exec rake bench` on an arm64 Mac, Ruby 3.4.9, default detector (both locales), average per `matches` call unless noted:
|
|
278
|
+
|
|
279
|
+
| message (~175 characters) | `:words` | `:embedded` |
|
|
280
|
+
|---|---|---|
|
|
281
|
+
| clean Polish | 30 µs | 60 µs |
|
|
282
|
+
| clean English | 22 µs | 51 µs |
|
|
283
|
+
| clean, with emoji | 33 µs | 54 µs |
|
|
284
|
+
| 3 profanities (`matches`) | 111 µs | 128 µs |
|
|
285
|
+
| 3 profanities (`profane?`) | 65 µs | 71 µs |
|
|
286
|
+
|
|
287
|
+
`Detector.new` with both lists takes about 10 ms, so build one detector at boot and reuse it. Characters outside the Latin blocks (emoji, symbols) are folded once per block of 256 code points and cached for the life of the process, up to 64 blocks. A message with characters beyond that is still handled correctly, just more slowly.
|
|
288
|
+
|
|
289
|
+
### Long input
|
|
290
|
+
|
|
291
|
+
The time grows linearly with the length of the input and the number of hits, also for adversarial input (long runs of one letter or symbol, thousands of profanities, one huge glued word). On the machine above, text without matches takes at most about 60 ms per 50 000 characters (one glued word in `:embedded` mode is the slowest case), and each profanity found adds 5 to 10 µs (`'kurwa ' * 8_333` takes about 50 ms). Still, a chat message or a name has no reason to be long, so cap the length before detection (reject or truncate first). In Rails, a `length:` validation alone is not enough, see [Validator](#validator).
|
|
292
|
+
|
|
293
|
+
## Development
|
|
294
|
+
|
|
295
|
+
```sh
|
|
296
|
+
bundle install
|
|
297
|
+
bundle exec rspec
|
|
298
|
+
bundle exec rubocop
|
|
299
|
+
bundle exec rake # both
|
|
300
|
+
bundle exec rake bench
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
The public API is `SwearJar` (`configure`, `detector`, `profane?`, `matches`, `censor`), `SwearJar::Detector`, `SwearJar::Match`, `SwearJar::Configuration`, the error classes and `SwearJarValidator`. Classes marked `@api private` (`Entry`, `WordList`, `Normalizer`, `FoldTable`, `PatternBuilder`, `Matcher`, `Skeleton`, `Slicer`) may change in any release.
|
|
304
|
+
|
|
305
|
+
The spec suite includes a table-driven corpus (`spec/fixtures/pl.yml`, `spec/fixtures/en.yml`) with profane and clean cases. When you change a word list, add cases there.
|
|
306
|
+
|
|
307
|
+
## Contributing
|
|
308
|
+
|
|
309
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/msx2/swear_jar. When submitting changes, please make sure specs and RuboCop pass and document user-facing changes in [CHANGELOG.md](CHANGELOG.md).
|
|
310
|
+
|
|
311
|
+
## License
|
|
312
|
+
|
|
313
|
+
MIT, see [LICENSE.txt](LICENSE.txt).
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SwearJar
|
|
4
|
+
# Single source of truth for the default option values: +Detector.new+ reads DEFAULTS too.
|
|
5
|
+
class Configuration
|
|
6
|
+
DEFAULTS = {
|
|
7
|
+
locales: LOCALES, mode: :words, min_severity: :mild, categories: nil,
|
|
8
|
+
extra_paths: [].freeze, extra_words: [].freeze, allow: [].freeze
|
|
9
|
+
}.freeze
|
|
10
|
+
|
|
11
|
+
attr_accessor(*DEFAULTS.keys)
|
|
12
|
+
|
|
13
|
+
def initialize
|
|
14
|
+
DEFAULTS.each { |key, value| public_send(:"#{key}=", value.frozen? && value.is_a?(Array) ? value.dup : value) }
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def to_h
|
|
18
|
+
DEFAULTS.each_key.to_h { |key| [key, public_send(key)] }
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
@mutex = Mutex.new
|
|
23
|
+
@detector = nil
|
|
24
|
+
|
|
25
|
+
class << self
|
|
26
|
+
# Yields a fresh Configuration, builds the detector first (so an invalid configuration raises
|
|
27
|
+
# and leaves the current detector untouched), then swaps it in under the mutex.
|
|
28
|
+
def configure
|
|
29
|
+
config = Configuration.new
|
|
30
|
+
yield config
|
|
31
|
+
new_detector = Detector.new(**config.to_h)
|
|
32
|
+
@mutex.synchronize { @detector = new_detector }
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# Readers do not lock once the detector exists.
|
|
36
|
+
def detector
|
|
37
|
+
@detector || @mutex.synchronize { @detector ||= Detector.new(**Configuration.new.to_h) }
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def profane?(text)
|
|
41
|
+
detector.profane?(text)
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def matches(text)
|
|
45
|
+
detector.matches(text)
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def censor(text, with: nil, &block)
|
|
49
|
+
detector.censor(text, with: with, &block)
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def reset_configuration!
|
|
53
|
+
@mutex.synchronize { @detector = nil }
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
end
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SwearJar
|
|
4
|
+
# All option processing and pattern compilation happens in +new+; the detector is frozen and
|
|
5
|
+
# safe to share between threads.
|
|
6
|
+
class Detector
|
|
7
|
+
def initialize(locales: Configuration::DEFAULTS[:locales], mode: Configuration::DEFAULTS[:mode],
|
|
8
|
+
min_severity: Configuration::DEFAULTS[:min_severity],
|
|
9
|
+
categories: Configuration::DEFAULTS[:categories],
|
|
10
|
+
extra_paths: Configuration::DEFAULTS[:extra_paths],
|
|
11
|
+
extra_words: Configuration::DEFAULTS[:extra_words], allow: Configuration::DEFAULTS[:allow])
|
|
12
|
+
mode = mode.to_sym if mode.respond_to?(:to_sym)
|
|
13
|
+
word_list = WordList.load(locales: locales, min_severity: min_severity, categories: categories,
|
|
14
|
+
extra_paths: extra_paths, extra_words: extra_words, allow: allow)
|
|
15
|
+
@matcher = Matcher.new(PatternBuilder.new(word_list, mode: mode))
|
|
16
|
+
@summary = "locales=#{word_list.locales.inspect} mode=#{mode.inspect} entries=#{word_list.entries.size}"
|
|
17
|
+
freeze
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
# The default inspect would dump the compiled patterns (tens of kilobytes).
|
|
21
|
+
def inspect
|
|
22
|
+
"#<#{self.class.name} #{@summary}>"
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Stops at the first match.
|
|
26
|
+
def profane?(text)
|
|
27
|
+
@matcher.each_match(text) { return true }
|
|
28
|
+
false
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# Sorted by position and never overlapping.
|
|
32
|
+
def matches(text)
|
|
33
|
+
result = []
|
|
34
|
+
@matcher.each_match(text) { |match| result << match }
|
|
35
|
+
result
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# Each match is replaced by the block's return value for it, else by +with+, else by one `*`
|
|
39
|
+
# per character. nil stays nil.
|
|
40
|
+
def censor(text, with: nil)
|
|
41
|
+
raise ArgumentError, 'pass with: or a block, not both' if with && block_given?
|
|
42
|
+
|
|
43
|
+
original = Matcher.coerce(text)
|
|
44
|
+
return if original.nil?
|
|
45
|
+
|
|
46
|
+
result = +''
|
|
47
|
+
pos = 0
|
|
48
|
+
@matcher.each_match(original) do |match|
|
|
49
|
+
replacement = if block_given? then yield(match)
|
|
50
|
+
else with || ('*' * match.text.length)
|
|
51
|
+
end
|
|
52
|
+
result << original[pos...match.range.begin] << replacement.to_s
|
|
53
|
+
pos = match.range.end
|
|
54
|
+
end
|
|
55
|
+
result << original[pos..]
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
end
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SwearJar
|
|
4
|
+
# @api private
|
|
5
|
+
class Entry
|
|
6
|
+
KEYS = %i[stem variants severity categories embeddable exact].freeze
|
|
7
|
+
|
|
8
|
+
# +source+ names where the entry came from (a file path or `extra_words`), for error messages.
|
|
9
|
+
attr_reader :stem, :variants, :locale, :severity, :categories, :embeddable, :exact, :source
|
|
10
|
+
|
|
11
|
+
def self.build(hash, locale:, source:)
|
|
12
|
+
new(hash, locale: locale, source: source)
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def initialize(hash, locale:, source:)
|
|
16
|
+
attrs = hash.to_h { |key, value| [key.respond_to?(:to_sym) ? key.to_sym : key, value] }
|
|
17
|
+
@locale = locale
|
|
18
|
+
@source = source.to_s.dup.freeze
|
|
19
|
+
@stem = validated_stem(attrs, source)
|
|
20
|
+
where = "#{source}: entry #{@stem.inspect}"
|
|
21
|
+
reject_unknown_keys(attrs, where)
|
|
22
|
+
@variants = freeze_strings(attrs[:variants] || [])
|
|
23
|
+
@severity = validated_severity(attrs[:severity], where)
|
|
24
|
+
@categories = validated_categories(attrs[:categories], where)
|
|
25
|
+
@embeddable = validated_boolean(attrs, :embeddable, true, where)
|
|
26
|
+
@exact = validated_boolean(attrs, :exact, false, where)
|
|
27
|
+
freeze
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def spellings
|
|
31
|
+
[stem, *variants]
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def rank
|
|
35
|
+
SEVERITIES.index(severity)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
private
|
|
39
|
+
|
|
40
|
+
def validated_stem(attrs, source)
|
|
41
|
+
stem = attrs[:stem]
|
|
42
|
+
return stem.to_s.dup.freeze if stem.is_a?(String) && !stem.strip.empty?
|
|
43
|
+
|
|
44
|
+
raise InvalidWordListError, "#{source}: entry #{stem.inspect} has a missing or blank stem"
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def reject_unknown_keys(attrs, where)
|
|
48
|
+
unknown = attrs.keys - KEYS
|
|
49
|
+
raise InvalidWordListError, "#{where} has unknown keys #{unknown.inspect}" unless unknown.empty?
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def freeze_strings(list)
|
|
53
|
+
list.map { |item| item.to_s.dup.freeze }.freeze
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def validated_severity(value, where)
|
|
57
|
+
severity = value.respond_to?(:to_sym) ? value.to_sym : nil
|
|
58
|
+
return severity if SEVERITIES.include?(severity)
|
|
59
|
+
|
|
60
|
+
raise InvalidWordListError, "#{where} has unknown severity #{value.inspect}"
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def validated_categories(value, where)
|
|
64
|
+
list = Array(value).map { |c| c.respond_to?(:to_sym) ? c.to_sym : c }
|
|
65
|
+
raise InvalidWordListError, "#{where} has empty categories" if list.empty?
|
|
66
|
+
|
|
67
|
+
unknown = list - CATEGORIES
|
|
68
|
+
raise InvalidWordListError, "#{where} has unknown categories #{unknown.inspect}" unless unknown.empty?
|
|
69
|
+
|
|
70
|
+
list.freeze
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def validated_boolean(attrs, key, default, where)
|
|
74
|
+
return default unless attrs.key?(key)
|
|
75
|
+
|
|
76
|
+
value = attrs[key]
|
|
77
|
+
return value if [true, false].include?(value)
|
|
78
|
+
|
|
79
|
+
raise InvalidWordListError, "#{where} has non-boolean #{key}: #{value.inspect}"
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
end
|