depii 0.1.0__tar.gz

depii-0.1.0/.gitignore

@@ -0,0 +1,15 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+venv/
+build/
+dist/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.hypothesis/
+.coverage
+htmlcov/
+.idea/
+.vscode/

depii-0.1.0/CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+This project follows [Semantic Versioning](https://semver.org/). Entity names and the fields of `Span` and `Result` are part of the public interface.
+
+## 0.1.0 (2026-10-13)
+
+First version.
+
+- Detection engine: Unicode normalisation with offsets mapped back to the original text, scoring with context words, deterministic overlap resolution.
+- Global detectors: email, phone, credit card, IBAN, BIC, IPv4 and IPv6, MAC address, URL, IMEI, VIN, coordinates, crypto wallets, date of birth.
+- Secret detectors: cloud and service tokens, JWTs, private key blocks, bearer tokens, `key = value` assignments.
+- Country packs: `us`, `in`, `gb`, `de`, `fr`, `es`, `it`, `nl`, `eu`. Third-party packs load through the `depii.locales` entry-point group.
+- Operators: `Replace`, `Mask`, `Hash` (HMAC-SHA256, key required), `Tokenize` with a reversible `Vault`, `Drop`, `Keep`.
+- `redact_obj` for nested data, with field-name hints.
+- `PIIFilter` for the `logging` module.
+- Command line: `redact` (text, JSON Lines, CSV), `scan`, `entities`.
+- `redact_dataframe` for pandas (extra `pandas`).
+- `SpacyDetector` for names, places and organisations (extra `ner`).

depii-0.1.0/LICENSE

@@ -0,0 +1,177 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS

depii-0.1.0/PKG-INFO

@@ -0,0 +1,285 @@
+Metadata-Version: 2.5
+Name: depii
+Version: 0.1.0
+Summary: Detect and redact personally identifiable information (PII) in text, nested data, logs, files and DataFrames.
+Project-URL: Repository, https://github.com/Boredem125/piiredact
+Project-URL: Issues, https://github.com/Boredem125/piiredact/issues
+Project-URL: Changelog, https://github.com/Boredem125/piiredact/blob/main/CHANGELOG.md
+Author: depii contributors
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: anonymization,data-protection,dpdp,gdpr,logging,pii,privacy,redaction
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Security
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: Filters
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: all
+Requires-Dist: pandas>=2.0; extra == 'all'
+Requires-Dist: phonenumbers>=8.13; extra == 'all'
+Requires-Dist: spacy>=3.7; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: hypothesis>=6.100; extra == 'dev'
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: pandas-stubs; extra == 'dev'
+Requires-Dist: pandas>=2.0; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: ner
+Requires-Dist: spacy>=3.7; extra == 'ner'
+Provides-Extra: pandas
+Requires-Dist: pandas>=2.0; extra == 'pandas'
+Provides-Extra: phone
+Requires-Dist: phonenumbers>=8.13; extra == 'phone'
+Description-Content-Type: text/markdown
+
+# depii
+
+[![CI](https://github.com/Boredem125/piiredact/actions/workflows/ci.yml/badge.svg)](https://github.com/Boredem125/piiredact/actions/workflows/ci.yml)
+
+Detect and redact personally identifiable information (PII) in text, nested data, logs, files and DataFrames.
+
+- **No dependencies** in the core. Pure Python, 3.10 and later.
+- **Validated, not just matched.** Card numbers, IBANs, Aadhaar, NHS numbers and other identifiers are checked against their check digits, so a random run of digits is not flagged.
+- **Country packs** for the United States, India, the United Kingdom, Germany, France, Spain, Italy, the Netherlands and the EU, plus a plug-in mechanism for more.
+- **Secrets too.** API keys, tokens, private keys and `password=...` assignments.
+- **Several ways in:** strings, nested dicts and lists, a `logging` filter, a command line tool, and pandas.
+
+```python
+>>> import depii
+>>> depii.redact("Mail jane@example.com or call +1 415 555 0132")
+'Mail [EMAIL] or call [PHONE]'
+
+```
+
+## Read this first
+
+No tool finds every piece of PII, and this one does not either. Names, street addresses and free-form descriptions ("the tall man from flat 4") cannot be found by patterns at all. Treat the output as reduced-risk data, not as anonymous data.
+
+Using this library does not by itself make a system compliant with GDPR, HIPAA, India's DPDP Act or any other law. Measure recall on a sample of your own data before you depend on it; [Evaluation](#evaluation) shows how.
+
+## Install
+
+```bash
+pip install depii
+```
+
+Optional extras: `depii[pandas]` for DataFrames, `depii[ner]` for names and places through spaCy, `depii[phone]` for stricter validation of international phone numbers, `depii[all]` for all three.
+
+## Usage
+
+### Text
+
+```python
+>>> from depii import Redactor, Mask
+>>> redactor = Redactor(policy={"CREDIT_CARD": Mask(keep_last=4)})
+>>> result = redactor.redact("jane@example.com paid with 4111 1111 1111 1111")
+>>> result.text
+'[EMAIL] paid with **** **** **** 1111'
+>>> result.stats
+{'EMAIL': 1, 'CREDIT_CARD': 1}
+
+```
+
+Build a `Redactor` once and reuse it. It is immutable and safe to share between threads.
+
+`scan` reports what was found without changing the text. The `repr` of a span leaves the matched value out, so spans can be logged:
+
+```python
+>>> depii.scan("SSN 078-05-1120")
+[Span(start=4, end=15, entity='US_SSN', score=1.0, detector='us_ssn')]
+
+```
+
+### Options
+
+| Option | Meaning |
+|---|---|
+| `locales` | Country packs to load, for example `["us", "in"]`. Default: all installed. Loading only the packs you need reduces false positives. |
+| `entities` / `exclude_entities` | Limit detection to, or exempt, some entity types. |
+| `policy` | Entity type to operator. Others use `default_operator`. |
+| `min_score` | Detections below this score are ignored. Default 0.5. |
+| `strict` | Prefer over-redaction: lower the threshold and drop the need for a context word. |
+| `allow` | Exact values never to redact, such as your own support address. |
+| `detectors` | Extra detectors of your own. |
+
+### Operators
+
+| Operator | `4111 1111 1111 1111` becomes |
+|---|---|
+| `Replace()` (default) | `[CREDIT_CARD]` |
+| `Mask(keep_last=4)` | `**** **** **** 1111` |
+| `Hash(key)` | `[CREDIT_CARD:` 16 hex digits `]` |
+| `Tokenize(vault)` | `[CREDIT_CARD_1]`, reversible with `vault.restore(text)` |
+| `Drop()` | nothing |
+| `Keep()` | unchanged |
+
+`Hash` needs a secret key of at least 16 bytes and uses HMAC-SHA256. An unkeyed hash of a phone number or national ID can be reversed by trying every possible value, so it is not offered. `Mask` never reveals more than half of a value. A `Vault` holds the original values: protect it as you would the unredacted data.
+
+### Nested data
+
+```python
+>>> depii.redact_obj({
+...     "note": "call jane@example.com",
+...     "user": {"firstName": "Jane", "password": "hunter2", "age": 41},
+... })
+{'note': 'call [EMAIL]', 'user': {'firstName': '[PERSON]', 'password': '[SECRET]', 'age': 41}}
+
+```
+
+Field names are used as hints. A field called `password`, `ssn`, `userEmail` or `phone_number` has its whole value redacted, even when the value does not look like PII. Pass `key_hints=False` to turn this off. The input object is not modified.
+
+### Logging
+
+```python
+import logging
+from depii import PIIFilter
+
+handler = logging.StreamHandler()
+handler.addFilter(PIIFilter())
+logging.getLogger().addHandler(handler)
+```
+
+Attach the filter to handlers, not loggers: a filter on a logger does not see records that propagate up from child loggers. The message, its arguments, exception text, stack info and `extra` fields are all redacted. If redaction itself fails, the message is replaced with a fixed notice instead of being written unredacted.
+
+### Command line
+
+```bash
+depii redact notes.txt -o clean.txt
+cat app.log | depii redact - > clean.log
+depii redact events.jsonl --format jsonl -o clean.jsonl
+depii redact people.csv --format csv --operator mask --keep-last 4 -o clean.csv
+depii scan notes.txt
+depii entities --locales us,in
+```
+
+`scan` prints `file:line:column: ENTITY (score)` for each finding and exits with status 1 when anything is found, so it can gate a commit or a CI job. It does not print the matched values unless you pass `--show-values`.
+
+For `--operator hash`, the key is read from the environment variable `DEPII_HASH_KEY`, so that it does not appear in the process list or shell history.
+
+Text input is streamed in chunks, so large files do not need to fit in memory. A file that is one single very long line is the exception.
+
+### pandas
+
+```python
+from depii.frames import redact_dataframe
+
+clean = redact_dataframe(df)  # every text column
+clean = redact_dataframe(df, columns=["notes"])  # only these
+```
+
+Column names are used as hints in the same way as field names.
+
+### Names and places
+
+Patterns cannot find names. The optional spaCy detector can, at the cost of a model download and slower runs:
+
+```bash
+pip install "depii[ner]"
+python -m spacy download en_core_web_sm
+```
+
+```python
+from depii import Redactor
+from depii.detectors.ner import SpacyDetector
+
+redactor = Redactor(detectors=[SpacyDetector()])
+```
+
+It emits `PERSON`, `LOCATION` and `ORGANIZATION`. Its accuracy is that of the spaCy model you load.
+
+### Your own detectors
+
+```python
+>>> from depii import PatternDetector, Redactor
+>>> employee_id = PatternDetector(
+...     "EMPLOYEE_ID",
+...     r"\bE\d{5}\b",
+...     score=0.4,
+...     context=("employee", "staff"),
+...     require_context=True,
+... )
+>>> Redactor(detectors=[employee_id]).redact("staff E12345 badge").text
+'staff [EMPLOYEE_ID] badge'
+
+```
+
+A country pack is a module that exports `DETECTORS`. A separate package can register one under the `depii.locales` entry-point group, and it is then loaded like a built-in pack:
+
+```toml
+[project.entry-points."depii.locales"]
+br = "depii_br:DETECTORS"
+```
+
+## What is detected
+
+| Pack | Entity types |
+|---|---|
+| Always on | `EMAIL`, `PHONE`, `CREDIT_CARD`, `IBAN`, `BIC`, `IP_ADDRESS`, `MAC_ADDRESS`, `URL`, `IMEI`, `VIN`, `GEO_COORDINATES`, `CRYPTO_WALLET`, `DATE_OF_BIRTH`, `SECRET` |
+| `us` | `US_SSN`, `US_ITIN`, `US_EIN`, `US_PASSPORT`, `US_ROUTING_NUMBER`, `US_ZIP` |
+| `in` | `IN_AADHAAR`, `IN_PAN`, `IN_GSTIN`, `IN_VOTER_ID`, `IN_PASSPORT`, `IN_IFSC`, `IN_UPI_ID`, `IN_PIN_CODE` |
+| `gb` | `GB_NINO`, `GB_NHS_NUMBER`, `GB_POSTCODE`, `GB_DRIVING_LICENCE` |
+| `de` | `DE_TAX_ID`, `DE_ID_CARD` |
+| `fr` | `FR_NIR` |
+| `es` | `ES_DNI`, `ES_NIE` |
+| `it` | `IT_FISCAL_CODE` |
+| `nl` | `NL_BSN` |
+| `eu` | `EU_VAT` |
+
+Field-name hints can also produce `PERSON`, `LOCATION`, `PASSPORT`, `BANK_ACCOUNT`, `NATIONAL_ID`, `TAX_ID` and `DRIVING_LICENCE`.
+
+### Behaviour worth knowing
+
+- **Context words.** Shapes that are too common to trust alone need a nearby word: a bare nine-digit number is an SSN only next to "SSN", a date is a date of birth only next to "born" or "DOB", and so on. `strict=True` removes this requirement.
+- **Context reaches across sentences.** The window is about 48 characters before a value and 24 after. A second date shortly after "born 3 March 1985" is redacted as well.
+- **URLs.** A URL is redacted when it carries credentials or a query string. A plain link is left alone unless `strict=True`.
+- **Aadhaar.** A bare twelve-digit number starting with 2 to 9 that passes the Verhoeff check is treated as an Aadhaar number. About one in twelve arbitrary twelve-digit numbers does. If your data has no Indian identifiers, leave the `in` pack out.
+- **Labels can be ambiguous.** A number that is valid as two things gets one label. It is redacted either way.
+- **Unicode.** Full-width digits, non-Latin digits, unusual dashes and zero-width characters inside a value are handled. Internationalised email addresses are not.
+- **Not handled:** images, PDFs, audio, and PII that is written out in words ("jane at example dot com").
+
+## Evaluation
+
+```bash
+python benchmarks/evaluate.py
+python benchmarks/throughput.py
+```
+
+`evaluate.py` scores the detectors on a generated corpus: valid identifiers planted in sentences, plus numeric text that holds no PII. On that corpus every listed type currently reaches a recall and precision of 1.0. This shows that the detectors agree with the formats they claim to support. It does not predict recall on real text, which is messier than anything a generator produces. To measure on your own data, label a sample and pass it to `synth.evaluate` in `tests/synth.py`.
+
+Throughput on one development laptop (Python 3.12, all packs loaded), for orientation only:
+
+| Input | Speed |
+|---|---|
+| Prose without digits | about 3.6 MB/s |
+| Text dense with identifiers | about 0.3 MB/s |
+| One short log line | 40 to 200 microseconds |
+
+Loading fewer packs is the simplest way to go faster.
+
+## Development
+
+```bash
+python -m venv .venv
+.venv/bin/pip install -e ".[dev]"     # .venv\Scripts\pip on Windows
+ruff check . && ruff format --check .
+mypy
+pytest
+```
+
+The test suite includes property tests (Hypothesis) and a timing test that guards every pattern against catastrophic regex backtracking. Test data is generated: do not add real personal data to this repository.
+
+## Licence
+
+Apache-2.0. See [LICENSE](LICENSE).