sieve-redact 0.5.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- sieve_redact-0.5.0/LICENSE +21 -0
- sieve_redact-0.5.0/PKG-INFO +216 -0
- sieve_redact-0.5.0/README.md +189 -0
- sieve_redact-0.5.0/pyproject.toml +43 -0
- sieve_redact-0.5.0/setup.cfg +4 -0
- sieve_redact-0.5.0/sieve_redact.egg-info/PKG-INFO +216 -0
- sieve_redact-0.5.0/sieve_redact.egg-info/SOURCES.txt +17 -0
- sieve_redact-0.5.0/sieve_redact.egg-info/dependency_links.txt +1 -0
- sieve_redact-0.5.0/sieve_redact.egg-info/entry_points.txt +2 -0
- sieve_redact-0.5.0/sieve_redact.egg-info/requires.txt +3 -0
- sieve_redact-0.5.0/sieve_redact.egg-info/top_level.txt +1 -0
- sieve_redact-0.5.0/sieve_redact.py +1196 -0
- sieve_redact-0.5.0/tests/test_acceptance.py +445 -0
- sieve_redact-0.5.0/tests/test_v02_builtin.py +286 -0
- sieve_redact-0.5.0/tests/test_v02_config_lens.py +304 -0
- sieve_redact-0.5.0/tests/test_v02_wordunit.py +193 -0
- sieve_redact-0.5.0/tests/test_v03.py +242 -0
- sieve_redact-0.5.0/tests/test_v04.py +151 -0
- sieve_redact-0.5.0/tests/test_v05_image.py +451 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Kai IWASAKI
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sieve-redact
|
|
3
|
+
Version: 0.5.0
|
|
4
|
+
Summary: Deterministic text and image masking — byte/byte-pixel integrity outside matched regions
|
|
5
|
+
Author: Kai IWASAKI
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/neguseatama/sieve-redact
|
|
8
|
+
Project-URL: Repository, https://github.com/neguseatama/sieve-redact.git
|
|
9
|
+
Project-URL: Bug Tracker, https://github.com/neguseatama/sieve-redact/issues
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: Education
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Topic :: Security
|
|
20
|
+
Classifier: Topic :: Text Processing :: Filters
|
|
21
|
+
Requires-Python: >=3.9
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
License-File: LICENSE
|
|
24
|
+
Provides-Extra: image
|
|
25
|
+
Requires-Dist: Pillow>=10.0; extra == "image"
|
|
26
|
+
Dynamic: license-file
|
|
27
|
+
|
|
28
|
+
# Sieve Redact v0.5
|
|
29
|
+
|
|
30
|
+
A command-line tool that removes sensitive information from documents —
|
|
31
|
+
**deterministically**, while preserving **byte-for-byte integrity** of
|
|
32
|
+
everything else.
|
|
33
|
+
|
|
34
|
+
[](https://github.com/neguseatama/sieve-redact/actions/workflows/test.yml)
|
|
35
|
+
[](https://opensource.org/licenses/MIT)
|
|
36
|
+
[](https://www.python.org/)
|
|
37
|
+
|
|
38
|
+
**English** | [日本語](README.ja.md)
|
|
39
|
+
|
|
40
|
+
## 💡 Concept
|
|
41
|
+
|
|
42
|
+
Personal information in resumes, reports and logs is converted into a second
|
|
43
|
+
file that is safe to publish and share.
|
|
44
|
+
|
|
45
|
+
- **Deterministic** — Same input plus same rules always produces the same
|
|
46
|
+
output. Character selection in `noise` is derived from SHA-256; it depends
|
|
47
|
+
on nothing else — not on the environment, the clock, or any RNG.
|
|
48
|
+
- **Byte-for-byte integrity** — Bytes outside a match are left untouched,
|
|
49
|
+
down to the last byte. CRLF line endings survive. Integrity is verified
|
|
50
|
+
mechanically before output; `--strict` refuses to write on failure
|
|
51
|
+
(exit code 3).
|
|
52
|
+
- **No guessing** — The user specifies what is sensitive. The engine never
|
|
53
|
+
guesses.
|
|
54
|
+
- **Receipts** — The receipt records the facts of processing (positions,
|
|
55
|
+
lengths, modes, hashes) but never the sensitive body itself. Even
|
|
56
|
+
"no pattern remains" is reported as a measurement
|
|
57
|
+
(`pattern_appearance_in_output`), not a promise.
|
|
58
|
+
|
|
59
|
+
> The name refers to a sieve: drop only what is specified, and leave
|
|
60
|
+
> everything else unharmed, byte for byte.
|
|
61
|
+
|
|
62
|
+
Companion tool to [Sieve Lens](https://github.com/neguseatama/sieve-lens)
|
|
63
|
+
(observation of invisible content) — Lens observes, Redact masks.
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## 🔥 Features
|
|
68
|
+
|
|
69
|
+
### Six redaction modes
|
|
70
|
+
|
|
71
|
+
| MODE | Operation | Content |
|
|
72
|
+
|------|-----------|---------|
|
|
73
|
+
| `delete` | remove the region | empty string |
|
|
74
|
+
| `mosaic` | fill per character | `█` repeated for the original character count |
|
|
75
|
+
| `label` | numbered labels | prefix (default `[REDACTED]`) + sequence number |
|
|
76
|
+
| `noise` | same-class replacement | picked deterministically within the same character class (hiragana / katakana / kanji / upper / lower / digit) |
|
|
77
|
+
| `decor` | wrap | `ARG + original + ARG` |
|
|
78
|
+
| `replace` | bulk replace | ARG |
|
|
79
|
+
|
|
80
|
+
### Built-in matchers (`--builtin`)
|
|
81
|
+
|
|
82
|
+
| NAME | Target |
|
|
83
|
+
|------|--------|
|
|
84
|
+
| `postal-jp` | Japanese postal codes `NNN-NNNN` |
|
|
85
|
+
| `phone-jp` | Japanese phone numbers (`090-…` / `03-…` / `050-…` / `0120-…` / 4-digit area codes `01XX-07XX`) |
|
|
86
|
+
| `email` | Email addresses (ASCII) |
|
|
87
|
+
| `chars` | All occurrences of given codepoints (`U+200B,U+00AD` form) |
|
|
88
|
+
|
|
89
|
+
### Image masking (`--region`)
|
|
90
|
+
|
|
91
|
+
PNG images (8-bit RGB/RGBA, non-interlaced, non-animated) are masked by
|
|
92
|
+
explicit regions: `--region x,y,w,h:MODE[:ARG]` (repeatable; overlapping
|
|
93
|
+
regions are discarded in declaration order; zero-area or out-of-bounds
|
|
94
|
+
regions are usage errors; `--rule` cannot be combined with `--region`).
|
|
95
|
+
|
|
96
|
+
| MODE | Effect |
|
|
97
|
+
|------|--------|
|
|
98
|
+
| `delete` | opaque white fill |
|
|
99
|
+
| `mosaic` | block average over all channels (blocks align to the region origin; ARG = block size, default 8) |
|
|
100
|
+
| `noise` | per-pixel deterministic replacement — SHA-256(`img:x:y:R,G,B`) drives the new RGB; alpha is kept, so semi-transparent pixels keep their visibility (content changes, visibility does not) |
|
|
101
|
+
| `label` | black band with `[REDACTED]n` (bundled bitmap font) |
|
|
102
|
+
| `decor` | 2px border in the ARG color (RRGGBB hex) |
|
|
103
|
+
| `replace` | ARG text drawn on a white fill (bundled bitmap font) |
|
|
104
|
+
|
|
105
|
+
Image receipts set `byte_integrity_verified` to `null` (not applicable —
|
|
106
|
+
PNG re-encoding changes file bytes structurally) and report pixel
|
|
107
|
+
integrity outside every region as `pixel_integrity_verified`; `--strict`
|
|
108
|
+
exits 3 when it fails. File metadata (EXIF etc.) is always dropped on
|
|
109
|
+
save — a privacy-friendly default by design. Output bytes are
|
|
110
|
+
deterministic across platforms and pinned in CI (with Pillow 12.3.0;
|
|
111
|
+
future Pillow updates may change encoded bytes, never pixels).
|
|
112
|
+
|
|
113
|
+
### Word-unit matching (`+` prefix)
|
|
114
|
+
|
|
115
|
+
`太郎:label` also matches the `太郎` inside `田中太郎`; `太郎:+label`
|
|
116
|
+
matches only at word boundaries (adjacent characters of the same class, or
|
|
117
|
+
both alphanumeric, cause rejection).
|
|
118
|
+
|
|
119
|
+
### Config files and Lens integration
|
|
120
|
+
|
|
121
|
+
- `--config rules.cfg`: load rules from a file (`rule <spec>` / `builtin <spec>`
|
|
122
|
+
line format).
|
|
123
|
+
- `--from-lens catalog.json`: generate masking rules from an observation
|
|
124
|
+
catalog (JSON) produced by [Sieve Lens](https://github.com/neguseatama/sieve-lens).
|
|
125
|
+
Invisible characters become deletions; look-alike letters
|
|
126
|
+
(Cyrillic/Greek → Latin) become replacements based on the `twin` field.
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## 💻 Quick start
|
|
131
|
+
|
|
132
|
+
git clone https://github.com/neguseatama/sieve-redact.git
|
|
133
|
+
cd sieve-redact
|
|
134
|
+
|
|
135
|
+
# Mosaic a name, label phone numbers
|
|
136
|
+
python sieve_redact.py in.txt -o out.txt \
|
|
137
|
+
--rule "田中太郎:mosaic" \
|
|
138
|
+
--builtin phone-jp:label \
|
|
139
|
+
--receipt receipt.json
|
|
140
|
+
|
|
141
|
+
Example receipt (stdout) — `in.txt` contains `x秘密y`, run with
|
|
142
|
+
`--rule "秘密:delete"`:
|
|
143
|
+
|
|
144
|
+
Sieve Redact v0.5 — レシート
|
|
145
|
+
入力: 8 バイト -> 出力: 2 バイト
|
|
146
|
+
redact 件数: 1
|
|
147
|
+
#1 rule=1 matcher=literal mode=delete pos=1..7 len=6B -> 0B sha256=062a2931da68...
|
|
148
|
+
他バイト完全性: 検証済み
|
|
149
|
+
出力内パターン残留: none
|
|
150
|
+
|
|
151
|
+
The sensitive body never appears in the receipt — hashes only. In
|
|
152
|
+
image mode (`--region`) `byte_integrity_verified` is `null` (not
|
|
153
|
+
applicable) and pixel integrity is measured by
|
|
154
|
+
`pixel_integrity_verified` instead.
|
|
155
|
+
|
|
156
|
+
### Integration with Sieve Lens
|
|
157
|
+
|
|
158
|
+
# (1) Lens produces an observation catalog
|
|
159
|
+
python -m sieve_lens_ext.redact_bridge input.txt catalog.json
|
|
160
|
+
# (2) Redact masks everything
|
|
161
|
+
python sieve_redact.py input.txt -o clean.txt --from-lens catalog.json
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
## 🔬 Determinism guarantees
|
|
166
|
+
|
|
167
|
+
Character selection in `noise` is fully determined by:
|
|
168
|
+
|
|
169
|
+
char = POOL[ SHA-256("<pattern>:<byte_offset>:<original char>").digest()[0] % len(POOL) ]
|
|
170
|
+
|
|
171
|
+
No seed. No environment dependence. Same input, same output, always.
|
|
172
|
+
The bundled tests verify exact agreement with this formula by recomputation.
|
|
173
|
+
|
|
174
|
+
Overlapping matches are resolved deterministically as well: rules are tried
|
|
175
|
+
in declaration order, each rule scans leftmost-first, later matches that
|
|
176
|
+
overlap a confirmed region are discarded, and confirmed regions are applied
|
|
177
|
+
in ascending order of position.
|
|
178
|
+
|
|
179
|
+
---
|
|
180
|
+
|
|
181
|
+
## 🔬 Tests
|
|
182
|
+
|
|
183
|
+
python -m unittest discover -s tests -v
|
|
184
|
+
|
|
185
|
+
113 tests (five modes, built-in matchers, word-unit matching, config files,
|
|
186
|
+
Lens integration, CLI boundaries). No additional dependencies.
|
|
187
|
+
|
|
188
|
+
---
|
|
189
|
+
|
|
190
|
+
## ⚠️ Known limitations
|
|
191
|
+
|
|
192
|
+
1. **No regex** — patterns are literal, exact-match strings only
|
|
193
|
+
2. **No case-insensitive matching** — matching is case-sensitive
|
|
194
|
+
3. **No automatic sensitive-data detection** — the user specifies what is
|
|
195
|
+
sensitive (by design)
|
|
196
|
+
4. **Text and PNG images** — PDF and Office formats are not supported;
|
|
197
|
+
image masking is PNG-only (8-bit RGB/RGBA) and JPEG is explicitly
|
|
198
|
+
out of scope; PDF masking is out of scope (observation is Sieve
|
|
199
|
+
Lens's domain)
|
|
200
|
+
5. **Full-width digits and non-ASCII hyphens are out of scope** for
|
|
201
|
+
postal/phone
|
|
202
|
+
6. **4-digit area codes starting with 08/09 (except `0800`) and short
|
|
203
|
+
numbers such as 117/104 are out of scope** for `phone-jp` — match
|
|
204
|
+
them explicitly with `--rule`
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
## 📄 License
|
|
209
|
+
|
|
210
|
+
MIT. See [LICENSE](LICENSE).
|
|
211
|
+
|
|
212
|
+
---
|
|
213
|
+
|
|
214
|
+
## 👤 Author
|
|
215
|
+
|
|
216
|
+
* **Kai IWASAKI**
|
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
# Sieve Redact v0.5
|
|
2
|
+
|
|
3
|
+
A command-line tool that removes sensitive information from documents —
|
|
4
|
+
**deterministically**, while preserving **byte-for-byte integrity** of
|
|
5
|
+
everything else.
|
|
6
|
+
|
|
7
|
+
[](https://github.com/neguseatama/sieve-redact/actions/workflows/test.yml)
|
|
8
|
+
[](https://opensource.org/licenses/MIT)
|
|
9
|
+
[](https://www.python.org/)
|
|
10
|
+
|
|
11
|
+
**English** | [日本語](README.ja.md)
|
|
12
|
+
|
|
13
|
+
## 💡 Concept
|
|
14
|
+
|
|
15
|
+
Personal information in resumes, reports and logs is converted into a second
|
|
16
|
+
file that is safe to publish and share.
|
|
17
|
+
|
|
18
|
+
- **Deterministic** — Same input plus same rules always produces the same
|
|
19
|
+
output. Character selection in `noise` is derived from SHA-256; it depends
|
|
20
|
+
on nothing else — not on the environment, the clock, or any RNG.
|
|
21
|
+
- **Byte-for-byte integrity** — Bytes outside a match are left untouched,
|
|
22
|
+
down to the last byte. CRLF line endings survive. Integrity is verified
|
|
23
|
+
mechanically before output; `--strict` refuses to write on failure
|
|
24
|
+
(exit code 3).
|
|
25
|
+
- **No guessing** — The user specifies what is sensitive. The engine never
|
|
26
|
+
guesses.
|
|
27
|
+
- **Receipts** — The receipt records the facts of processing (positions,
|
|
28
|
+
lengths, modes, hashes) but never the sensitive body itself. Even
|
|
29
|
+
"no pattern remains" is reported as a measurement
|
|
30
|
+
(`pattern_appearance_in_output`), not a promise.
|
|
31
|
+
|
|
32
|
+
> The name refers to a sieve: drop only what is specified, and leave
|
|
33
|
+
> everything else unharmed, byte for byte.
|
|
34
|
+
|
|
35
|
+
Companion tool to [Sieve Lens](https://github.com/neguseatama/sieve-lens)
|
|
36
|
+
(observation of invisible content) — Lens observes, Redact masks.
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## 🔥 Features
|
|
41
|
+
|
|
42
|
+
### Six redaction modes
|
|
43
|
+
|
|
44
|
+
| MODE | Operation | Content |
|
|
45
|
+
|------|-----------|---------|
|
|
46
|
+
| `delete` | remove the region | empty string |
|
|
47
|
+
| `mosaic` | fill per character | `█` repeated for the original character count |
|
|
48
|
+
| `label` | numbered labels | prefix (default `[REDACTED]`) + sequence number |
|
|
49
|
+
| `noise` | same-class replacement | picked deterministically within the same character class (hiragana / katakana / kanji / upper / lower / digit) |
|
|
50
|
+
| `decor` | wrap | `ARG + original + ARG` |
|
|
51
|
+
| `replace` | bulk replace | ARG |
|
|
52
|
+
|
|
53
|
+
### Built-in matchers (`--builtin`)
|
|
54
|
+
|
|
55
|
+
| NAME | Target |
|
|
56
|
+
|------|--------|
|
|
57
|
+
| `postal-jp` | Japanese postal codes `NNN-NNNN` |
|
|
58
|
+
| `phone-jp` | Japanese phone numbers (`090-…` / `03-…` / `050-…` / `0120-…` / 4-digit area codes `01XX-07XX`) |
|
|
59
|
+
| `email` | Email addresses (ASCII) |
|
|
60
|
+
| `chars` | All occurrences of given codepoints (`U+200B,U+00AD` form) |
|
|
61
|
+
|
|
62
|
+
### Image masking (`--region`)
|
|
63
|
+
|
|
64
|
+
PNG images (8-bit RGB/RGBA, non-interlaced, non-animated) are masked by
|
|
65
|
+
explicit regions: `--region x,y,w,h:MODE[:ARG]` (repeatable; overlapping
|
|
66
|
+
regions are discarded in declaration order; zero-area or out-of-bounds
|
|
67
|
+
regions are usage errors; `--rule` cannot be combined with `--region`).
|
|
68
|
+
|
|
69
|
+
| MODE | Effect |
|
|
70
|
+
|------|--------|
|
|
71
|
+
| `delete` | opaque white fill |
|
|
72
|
+
| `mosaic` | block average over all channels (blocks align to the region origin; ARG = block size, default 8) |
|
|
73
|
+
| `noise` | per-pixel deterministic replacement — SHA-256(`img:x:y:R,G,B`) drives the new RGB; alpha is kept, so semi-transparent pixels keep their visibility (content changes, visibility does not) |
|
|
74
|
+
| `label` | black band with `[REDACTED]n` (bundled bitmap font) |
|
|
75
|
+
| `decor` | 2px border in the ARG color (RRGGBB hex) |
|
|
76
|
+
| `replace` | ARG text drawn on a white fill (bundled bitmap font) |
|
|
77
|
+
|
|
78
|
+
Image receipts set `byte_integrity_verified` to `null` (not applicable —
|
|
79
|
+
PNG re-encoding changes file bytes structurally) and report pixel
|
|
80
|
+
integrity outside every region as `pixel_integrity_verified`; `--strict`
|
|
81
|
+
exits 3 when it fails. File metadata (EXIF etc.) is always dropped on
|
|
82
|
+
save — a privacy-friendly default by design. Output bytes are
|
|
83
|
+
deterministic across platforms and pinned in CI (with Pillow 12.3.0;
|
|
84
|
+
future Pillow updates may change encoded bytes, never pixels).
|
|
85
|
+
|
|
86
|
+
### Word-unit matching (`+` prefix)
|
|
87
|
+
|
|
88
|
+
`太郎:label` also matches the `太郎` inside `田中太郎`; `太郎:+label`
|
|
89
|
+
matches only at word boundaries (adjacent characters of the same class, or
|
|
90
|
+
both alphanumeric, cause rejection).
|
|
91
|
+
|
|
92
|
+
### Config files and Lens integration
|
|
93
|
+
|
|
94
|
+
- `--config rules.cfg`: load rules from a file (`rule <spec>` / `builtin <spec>`
|
|
95
|
+
line format).
|
|
96
|
+
- `--from-lens catalog.json`: generate masking rules from an observation
|
|
97
|
+
catalog (JSON) produced by [Sieve Lens](https://github.com/neguseatama/sieve-lens).
|
|
98
|
+
Invisible characters become deletions; look-alike letters
|
|
99
|
+
(Cyrillic/Greek → Latin) become replacements based on the `twin` field.
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
|
|
103
|
+
## 💻 Quick start
|
|
104
|
+
|
|
105
|
+
git clone https://github.com/neguseatama/sieve-redact.git
|
|
106
|
+
cd sieve-redact
|
|
107
|
+
|
|
108
|
+
# Mosaic a name, label phone numbers
|
|
109
|
+
python sieve_redact.py in.txt -o out.txt \
|
|
110
|
+
--rule "田中太郎:mosaic" \
|
|
111
|
+
--builtin phone-jp:label \
|
|
112
|
+
--receipt receipt.json
|
|
113
|
+
|
|
114
|
+
Example receipt (stdout) — `in.txt` contains `x秘密y`, run with
|
|
115
|
+
`--rule "秘密:delete"`:
|
|
116
|
+
|
|
117
|
+
Sieve Redact v0.5 — レシート
|
|
118
|
+
入力: 8 バイト -> 出力: 2 バイト
|
|
119
|
+
redact 件数: 1
|
|
120
|
+
#1 rule=1 matcher=literal mode=delete pos=1..7 len=6B -> 0B sha256=062a2931da68...
|
|
121
|
+
他バイト完全性: 検証済み
|
|
122
|
+
出力内パターン残留: none
|
|
123
|
+
|
|
124
|
+
The sensitive body never appears in the receipt — hashes only. In
|
|
125
|
+
image mode (`--region`) `byte_integrity_verified` is `null` (not
|
|
126
|
+
applicable) and pixel integrity is measured by
|
|
127
|
+
`pixel_integrity_verified` instead.
|
|
128
|
+
|
|
129
|
+
### Integration with Sieve Lens
|
|
130
|
+
|
|
131
|
+
# (1) Lens produces an observation catalog
|
|
132
|
+
python -m sieve_lens_ext.redact_bridge input.txt catalog.json
|
|
133
|
+
# (2) Redact masks everything
|
|
134
|
+
python sieve_redact.py input.txt -o clean.txt --from-lens catalog.json
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## 🔬 Determinism guarantees
|
|
139
|
+
|
|
140
|
+
Character selection in `noise` is fully determined by:
|
|
141
|
+
|
|
142
|
+
char = POOL[ SHA-256("<pattern>:<byte_offset>:<original char>").digest()[0] % len(POOL) ]
|
|
143
|
+
|
|
144
|
+
No seed. No environment dependence. Same input, same output, always.
|
|
145
|
+
The bundled tests verify exact agreement with this formula by recomputation.
|
|
146
|
+
|
|
147
|
+
Overlapping matches are resolved deterministically as well: rules are tried
|
|
148
|
+
in declaration order, each rule scans leftmost-first, later matches that
|
|
149
|
+
overlap a confirmed region are discarded, and confirmed regions are applied
|
|
150
|
+
in ascending order of position.
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## 🔬 Tests
|
|
155
|
+
|
|
156
|
+
python -m unittest discover -s tests -v
|
|
157
|
+
|
|
158
|
+
113 tests (five modes, built-in matchers, word-unit matching, config files,
|
|
159
|
+
Lens integration, CLI boundaries). No additional dependencies.
|
|
160
|
+
|
|
161
|
+
---
|
|
162
|
+
|
|
163
|
+
## ⚠️ Known limitations
|
|
164
|
+
|
|
165
|
+
1. **No regex** — patterns are literal, exact-match strings only
|
|
166
|
+
2. **No case-insensitive matching** — matching is case-sensitive
|
|
167
|
+
3. **No automatic sensitive-data detection** — the user specifies what is
|
|
168
|
+
sensitive (by design)
|
|
169
|
+
4. **Text and PNG images** — PDF and Office formats are not supported;
|
|
170
|
+
image masking is PNG-only (8-bit RGB/RGBA) and JPEG is explicitly
|
|
171
|
+
out of scope; PDF masking is out of scope (observation is Sieve
|
|
172
|
+
Lens's domain)
|
|
173
|
+
5. **Full-width digits and non-ASCII hyphens are out of scope** for
|
|
174
|
+
postal/phone
|
|
175
|
+
6. **4-digit area codes starting with 08/09 (except `0800`) and short
|
|
176
|
+
numbers such as 117/104 are out of scope** for `phone-jp` — match
|
|
177
|
+
them explicitly with `--rule`
|
|
178
|
+
|
|
179
|
+
---
|
|
180
|
+
|
|
181
|
+
## 📄 License
|
|
182
|
+
|
|
183
|
+
MIT. See [LICENSE](LICENSE).
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
## 👤 Author
|
|
188
|
+
|
|
189
|
+
* **Kai IWASAKI**
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=61.0"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "sieve-redact"
|
|
7
|
+
dynamic = ["version"]
|
|
8
|
+
description = "Deterministic text and image masking — byte/byte-pixel integrity outside matched regions"
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
authors = [{ name = "Kai IWASAKI" }]
|
|
11
|
+
license = { text = "MIT" }
|
|
12
|
+
classifiers = [
|
|
13
|
+
"Development Status :: 3 - Alpha",
|
|
14
|
+
"Intended Audience :: Developers",
|
|
15
|
+
"Intended Audience :: Education",
|
|
16
|
+
"License :: OSI Approved :: MIT License",
|
|
17
|
+
"Programming Language :: Python :: 3",
|
|
18
|
+
"Programming Language :: Python :: 3.9",
|
|
19
|
+
"Programming Language :: Python :: 3.10",
|
|
20
|
+
"Programming Language :: Python :: 3.11",
|
|
21
|
+
"Programming Language :: Python :: 3.12",
|
|
22
|
+
"Topic :: Security",
|
|
23
|
+
"Topic :: Text Processing :: Filters"
|
|
24
|
+
]
|
|
25
|
+
requires-python = ">=3.9"
|
|
26
|
+
dependencies = []
|
|
27
|
+
|
|
28
|
+
[project.urls]
|
|
29
|
+
Homepage = "https://github.com/neguseatama/sieve-redact"
|
|
30
|
+
Repository = "https://github.com/neguseatama/sieve-redact.git"
|
|
31
|
+
"Bug Tracker" = "https://github.com/neguseatama/sieve-redact/issues"
|
|
32
|
+
|
|
33
|
+
[project.optional-dependencies]
|
|
34
|
+
image = ["Pillow>=10.0"]
|
|
35
|
+
|
|
36
|
+
[project.scripts]
|
|
37
|
+
sieve-redact = "sieve_redact:main"
|
|
38
|
+
|
|
39
|
+
[tool.setuptools]
|
|
40
|
+
py-modules = ["sieve_redact"]
|
|
41
|
+
|
|
42
|
+
[tool.setuptools.dynamic]
|
|
43
|
+
version = { attr = "sieve_redact.VERSION" }
|
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sieve-redact
|
|
3
|
+
Version: 0.5.0
|
|
4
|
+
Summary: Deterministic text and image masking — byte/byte-pixel integrity outside matched regions
|
|
5
|
+
Author: Kai IWASAKI
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/neguseatama/sieve-redact
|
|
8
|
+
Project-URL: Repository, https://github.com/neguseatama/sieve-redact.git
|
|
9
|
+
Project-URL: Bug Tracker, https://github.com/neguseatama/sieve-redact/issues
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: Education
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Topic :: Security
|
|
20
|
+
Classifier: Topic :: Text Processing :: Filters
|
|
21
|
+
Requires-Python: >=3.9
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
License-File: LICENSE
|
|
24
|
+
Provides-Extra: image
|
|
25
|
+
Requires-Dist: Pillow>=10.0; extra == "image"
|
|
26
|
+
Dynamic: license-file
|
|
27
|
+
|
|
28
|
+
# Sieve Redact v0.5
|
|
29
|
+
|
|
30
|
+
A command-line tool that removes sensitive information from documents —
|
|
31
|
+
**deterministically**, while preserving **byte-for-byte integrity** of
|
|
32
|
+
everything else.
|
|
33
|
+
|
|
34
|
+
[](https://github.com/neguseatama/sieve-redact/actions/workflows/test.yml)
|
|
35
|
+
[](https://opensource.org/licenses/MIT)
|
|
36
|
+
[](https://www.python.org/)
|
|
37
|
+
|
|
38
|
+
**English** | [日本語](README.ja.md)
|
|
39
|
+
|
|
40
|
+
## 💡 Concept
|
|
41
|
+
|
|
42
|
+
Personal information in resumes, reports and logs is converted into a second
|
|
43
|
+
file that is safe to publish and share.
|
|
44
|
+
|
|
45
|
+
- **Deterministic** — Same input plus same rules always produces the same
|
|
46
|
+
output. Character selection in `noise` is derived from SHA-256; it depends
|
|
47
|
+
on nothing else — not on the environment, the clock, or any RNG.
|
|
48
|
+
- **Byte-for-byte integrity** — Bytes outside a match are left untouched,
|
|
49
|
+
down to the last byte. CRLF line endings survive. Integrity is verified
|
|
50
|
+
mechanically before output; `--strict` refuses to write on failure
|
|
51
|
+
(exit code 3).
|
|
52
|
+
- **No guessing** — The user specifies what is sensitive. The engine never
|
|
53
|
+
guesses.
|
|
54
|
+
- **Receipts** — The receipt records the facts of processing (positions,
|
|
55
|
+
lengths, modes, hashes) but never the sensitive body itself. Even
|
|
56
|
+
"no pattern remains" is reported as a measurement
|
|
57
|
+
(`pattern_appearance_in_output`), not a promise.
|
|
58
|
+
|
|
59
|
+
> The name refers to a sieve: drop only what is specified, and leave
|
|
60
|
+
> everything else unharmed, byte for byte.
|
|
61
|
+
|
|
62
|
+
Companion tool to [Sieve Lens](https://github.com/neguseatama/sieve-lens)
|
|
63
|
+
(observation of invisible content) — Lens observes, Redact masks.
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## 🔥 Features
|
|
68
|
+
|
|
69
|
+
### Six redaction modes
|
|
70
|
+
|
|
71
|
+
| MODE | Operation | Content |
|
|
72
|
+
|------|-----------|---------|
|
|
73
|
+
| `delete` | remove the region | empty string |
|
|
74
|
+
| `mosaic` | fill per character | `█` repeated for the original character count |
|
|
75
|
+
| `label` | numbered labels | prefix (default `[REDACTED]`) + sequence number |
|
|
76
|
+
| `noise` | same-class replacement | picked deterministically within the same character class (hiragana / katakana / kanji / upper / lower / digit) |
|
|
77
|
+
| `decor` | wrap | `ARG + original + ARG` |
|
|
78
|
+
| `replace` | bulk replace | ARG |
|
|
79
|
+
|
|
80
|
+
### Built-in matchers (`--builtin`)
|
|
81
|
+
|
|
82
|
+
| NAME | Target |
|
|
83
|
+
|------|--------|
|
|
84
|
+
| `postal-jp` | Japanese postal codes `NNN-NNNN` |
|
|
85
|
+
| `phone-jp` | Japanese phone numbers (`090-…` / `03-…` / `050-…` / `0120-…` / 4-digit area codes `01XX-07XX`) |
|
|
86
|
+
| `email` | Email addresses (ASCII) |
|
|
87
|
+
| `chars` | All occurrences of given codepoints (`U+200B,U+00AD` form) |
|
|
88
|
+
|
|
89
|
+
### Image masking (`--region`)
|
|
90
|
+
|
|
91
|
+
PNG images (8-bit RGB/RGBA, non-interlaced, non-animated) are masked by
|
|
92
|
+
explicit regions: `--region x,y,w,h:MODE[:ARG]` (repeatable; overlapping
|
|
93
|
+
regions are discarded in declaration order; zero-area or out-of-bounds
|
|
94
|
+
regions are usage errors; `--rule` cannot be combined with `--region`).
|
|
95
|
+
|
|
96
|
+
| MODE | Effect |
|
|
97
|
+
|------|--------|
|
|
98
|
+
| `delete` | opaque white fill |
|
|
99
|
+
| `mosaic` | block average over all channels (blocks align to the region origin; ARG = block size, default 8) |
|
|
100
|
+
| `noise` | per-pixel deterministic replacement — SHA-256(`img:x:y:R,G,B`) drives the new RGB; alpha is kept, so semi-transparent pixels keep their visibility (content changes, visibility does not) |
|
|
101
|
+
| `label` | black band with `[REDACTED]n` (bundled bitmap font) |
|
|
102
|
+
| `decor` | 2px border in the ARG color (RRGGBB hex) |
|
|
103
|
+
| `replace` | ARG text drawn on a white fill (bundled bitmap font) |
|
|
104
|
+
|
|
105
|
+
Image receipts set `byte_integrity_verified` to `null` (not applicable —
|
|
106
|
+
PNG re-encoding changes file bytes structurally) and report pixel
|
|
107
|
+
integrity outside every region as `pixel_integrity_verified`; `--strict`
|
|
108
|
+
exits 3 when it fails. File metadata (EXIF etc.) is always dropped on
|
|
109
|
+
save — a privacy-friendly default by design. Output bytes are
|
|
110
|
+
deterministic across platforms and pinned in CI (with Pillow 12.3.0;
|
|
111
|
+
future Pillow updates may change encoded bytes, never pixels).
|
|
112
|
+
|
|
113
|
+
### Word-unit matching (`+` prefix)
|
|
114
|
+
|
|
115
|
+
`太郎:label` also matches the `太郎` inside `田中太郎`; `太郎:+label`
|
|
116
|
+
matches only at word boundaries (adjacent characters of the same class, or
|
|
117
|
+
both alphanumeric, cause rejection).
|
|
118
|
+
|
|
119
|
+
### Config files and Lens integration
|
|
120
|
+
|
|
121
|
+
- `--config rules.cfg`: load rules from a file (`rule <spec>` / `builtin <spec>`
|
|
122
|
+
line format).
|
|
123
|
+
- `--from-lens catalog.json`: generate masking rules from an observation
|
|
124
|
+
catalog (JSON) produced by [Sieve Lens](https://github.com/neguseatama/sieve-lens).
|
|
125
|
+
Invisible characters become deletions; look-alike letters
|
|
126
|
+
(Cyrillic/Greek → Latin) become replacements based on the `twin` field.
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## 💻 Quick start
|
|
131
|
+
|
|
132
|
+
git clone https://github.com/neguseatama/sieve-redact.git
|
|
133
|
+
cd sieve-redact
|
|
134
|
+
|
|
135
|
+
# Mosaic a name, label phone numbers
|
|
136
|
+
python sieve_redact.py in.txt -o out.txt \
|
|
137
|
+
--rule "田中太郎:mosaic" \
|
|
138
|
+
--builtin phone-jp:label \
|
|
139
|
+
--receipt receipt.json
|
|
140
|
+
|
|
141
|
+
Example receipt (stdout) — `in.txt` contains `x秘密y`, run with
|
|
142
|
+
`--rule "秘密:delete"`:
|
|
143
|
+
|
|
144
|
+
Sieve Redact v0.5 — レシート
|
|
145
|
+
入力: 8 バイト -> 出力: 2 バイト
|
|
146
|
+
redact 件数: 1
|
|
147
|
+
#1 rule=1 matcher=literal mode=delete pos=1..7 len=6B -> 0B sha256=062a2931da68...
|
|
148
|
+
他バイト完全性: 検証済み
|
|
149
|
+
出力内パターン残留: none
|
|
150
|
+
|
|
151
|
+
The sensitive body never appears in the receipt — hashes only. In
|
|
152
|
+
image mode (`--region`) `byte_integrity_verified` is `null` (not
|
|
153
|
+
applicable) and pixel integrity is measured by
|
|
154
|
+
`pixel_integrity_verified` instead.
|
|
155
|
+
|
|
156
|
+
### Integration with Sieve Lens
|
|
157
|
+
|
|
158
|
+
# (1) Lens produces an observation catalog
|
|
159
|
+
python -m sieve_lens_ext.redact_bridge input.txt catalog.json
|
|
160
|
+
# (2) Redact masks everything
|
|
161
|
+
python sieve_redact.py input.txt -o clean.txt --from-lens catalog.json
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
## 🔬 Determinism guarantees
|
|
166
|
+
|
|
167
|
+
Character selection in `noise` is fully determined by:
|
|
168
|
+
|
|
169
|
+
char = POOL[ SHA-256("<pattern>:<byte_offset>:<original char>").digest()[0] % len(POOL) ]
|
|
170
|
+
|
|
171
|
+
No seed. No environment dependence. Same input, same output, always.
|
|
172
|
+
The bundled tests verify exact agreement with this formula by recomputation.
|
|
173
|
+
|
|
174
|
+
Overlapping matches are resolved deterministically as well: rules are tried
|
|
175
|
+
in declaration order, each rule scans leftmost-first, later matches that
|
|
176
|
+
overlap a confirmed region are discarded, and confirmed regions are applied
|
|
177
|
+
in ascending order of position.
|
|
178
|
+
|
|
179
|
+
---
|
|
180
|
+
|
|
181
|
+
## 🔬 Tests
|
|
182
|
+
|
|
183
|
+
python -m unittest discover -s tests -v
|
|
184
|
+
|
|
185
|
+
113 tests (five modes, built-in matchers, word-unit matching, config files,
|
|
186
|
+
Lens integration, CLI boundaries). No additional dependencies.
|
|
187
|
+
|
|
188
|
+
---
|
|
189
|
+
|
|
190
|
+
## ⚠️ Known limitations
|
|
191
|
+
|
|
192
|
+
1. **No regex** — patterns are literal, exact-match strings only
|
|
193
|
+
2. **No case-insensitive matching** — matching is case-sensitive
|
|
194
|
+
3. **No automatic sensitive-data detection** — the user specifies what is
|
|
195
|
+
sensitive (by design)
|
|
196
|
+
4. **Text and PNG images** — PDF and Office formats are not supported;
|
|
197
|
+
image masking is PNG-only (8-bit RGB/RGBA) and JPEG is explicitly
|
|
198
|
+
out of scope; PDF masking is out of scope (observation is Sieve
|
|
199
|
+
Lens's domain)
|
|
200
|
+
5. **Full-width digits and non-ASCII hyphens are out of scope** for
|
|
201
|
+
postal/phone
|
|
202
|
+
6. **4-digit area codes starting with 08/09 (except `0800`) and short
|
|
203
|
+
numbers such as 117/104 are out of scope** for `phone-jp` — match
|
|
204
|
+
them explicitly with `--rule`
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
## 📄 License
|
|
209
|
+
|
|
210
|
+
MIT. See [LICENSE](LICENSE).
|
|
211
|
+
|
|
212
|
+
---
|
|
213
|
+
|
|
214
|
+
## 👤 Author
|
|
215
|
+
|
|
216
|
+
* **Kai IWASAKI**
|