nyms 0.0.2__tar.gz

nyms-0.0.2/.gitattributes

@@ -0,0 +1 @@
+*.ipynb linguist-documentation

nyms-0.0.2/.gitignore

@@ -0,0 +1,119 @@
+.claude/handoffs/
+.claude/scratch/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+
+.DS_Store
+# C extensions
+*.so
+
+# TLS certificates
+## Ignore all PEM files anywhere
+*.pem
+## Also ignore any certs directory
+certs/
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+_build
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+# PyCharm
+.idea

nyms-0.0.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Thor Whalen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

nyms-0.0.2/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.5
+Name: nyms
+Version: 0.0.2
+Summary: Reversible pseudonymisation of text: deterministic detection of personal and sensitive identifiers, keyed surrogate generation (readable pseudonyms included), and re-identification through a mapping store.
+Project-URL: Homepage, https://github.com/thorwhalen/nyms
+Project-URL: Repository, https://github.com/thorwhalen/nyms
+Project-URL: Documentation, https://thorwhalen.github.io/nyms
+Author: Thor Whalen
+License-Expression: MIT
+License-File: LICENSE
+Keywords: anonymization,cpr,danish,de-identification,gdpr,pii,privacy,pseudonymisation,redaction
+Requires-Python: >=3.10
+Requires-Dist: cw>=0.1
+Requires-Dist: dol>=0.3
+Provides-Extra: crypto
+Requires-Dist: cryptography>=42; extra == 'crypto'
+Provides-Extra: dev
+Requires-Dist: cryptography>=42; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: sphinx-rtd-theme>=1.0; extra == 'docs'
+Requires-Dist: sphinx>=6.0; extra == 'docs'
+Description-Content-Type: text/markdown
+
+# nyms
+
+Reversible pseudonymisation of text before it reaches an AI. A deterministic (non-AI) pass finds personal and sensitive identifiers in Danish and English text, replaces them with surrogates, remembers the mapping per project so the same person is `[PERSON_1]` in every document, and puts the originals back into whatever the AI answers.
+
+```bash
+pip install nyms            # add "nyms[crypto]" for an encrypted mapping store
+```
+
+```python
+from nyms import Pseudonymiser
+
+p = Pseudonymiser("case-2026", names=["Rasmus Østergaard Jensen"])
+r = p.pseudonymise("Rasmus Jensen (CPR 010190-1234) skylder 12.500 kr. Rasmus Jensens advokat ringede.")
+r.text
+# '[PERSON_1] (CPR [CPR_1]) skylder [AMOUNT_1]. [PERSON_1]s advokat ringede.'
+p.reidentify("Svar: [PERSON_1] skal betale [AMOUNT_1].").text
+# 'Svar: Rasmus Jensen skal betale 12.500 kr.'
+p.check("Rasmus Jensen bor her")      # what would leak if this were sent
+# ['Rasmus Jensen']
+```
+
+The same four verbs from the shell:
+
+```bash
+nyms pseudonymise --path letter.md --project case-2026 --names "Rasmus Østergaard Jensen" --out letter.pseud.md
+nyms reidentify --path answer.md --project case-2026 --out answer.clear.md
+nyms check --path draft.md --project case-2026       # {"ok": false, "leaks": [...]} if an original is in it
+nyms detect "Rasmus Jensen, tlf. 12 34 56 78"        # every span and the policy's decision; nothing stored
+nyms readable 010190-1234 --project case-2026        # a keyed hash as two readable words
+```
+
+Content is positional text or `--path` (a file, or `-` for stdin), never guessed from one another.
+
+```bash
+```
+
+`r.report` says what was replaced (by type and by detector), what was detected and kept (dates, postcodes and URLs, under the default policy) and which detectors ran, and ends with the caveat that this is pseudonymisation, not anonymisation.
+
+## What it finds
+
+Danish: CPR numbers (the date part must be valid; the pre-2007 modulus-11 check only raises confidence, because numbers issued since October 2007 need not pass it), CVR numbers (modulus-11 and context), DK VAT numbers, phone numbers, bank account numbers (registreringsnummer and kontonummer), IBAN (mod-97), street addresses with floor and postcode, postcode and town, amounts (`12.500 kr.`, `DKK 12.500`, `36.000,-`), dates, court and journal numbers. English: addresses, phone numbers, amounts (`£4,250.00`, `$5,000`), dates. Both: e-mail, URLs, IBAN, credentials (API keys, tokens, private keys), organisation names by legal form (`ApS`, `A/S`, `Ltd`, `GmbH`), names after a title (`Hr.`, `advokat`, `Mr`, `Dr`), names from a bundled Danish and English first-name list, names you pass in (`names=`), and a low-confidence heuristic for capitalised sequences.
+
+Once a value has been replaced anywhere in a project, every later occurrence is replaced too, including a Danish genitive (`Rasmus Jensens` becomes `[PERSON_1]s`), ALL-CAPS headings, and a surname or first name alone when it identifies exactly one known person (`Jensen blev` becomes `[PERSON_1] blev`). An identifier inside something the policy keeps (a CPR number in a URL) is still replaced. Pseudonymising already pseudonymised text changes nothing.
+
+It does not find: names it has never seen and the heuristic misses, roles and rare facts that identify without an identifier, inflected compounds, OCR-damaged identifiers. The report's `kept` list and the `check` command exist so a reader sees what stayed.
+
+## Choosing what is sensitive
+
+A `Policy` decides after detection, so the report can say what it saw and kept. `Policy.default()` replaces credentials, persons, CPR, e-mail, phone, addresses, IBAN, bank accounts, case numbers, CVR, organisations and amounts; `Policy.identifiers()` leaves amounts and organisations; `Policy.strict()` replaces everything detected, dates and postcodes included. `include`, `exclude`, per-type `min_score`, `allow` (never replace) and `deny` (always replace) lists refine any of them. The taxonomy behind the types, with tiers and the equivalent Presidio, Google DLP and HIPAA names, is `nyms.taxonomy.ENTITY_TYPES`.
+
+## Choosing the surrogates
+
+| Strategy | Example | When |
+|---|---|---|
+| `Counter()` (default) | `[PERSON_1]`, `[CPR_2]` | the form LLMs alter least in the one published benchmark; numbered per project |
+| `KeyedWords(key)` | `Alder Brook`, `[CPR:mustard-suitcase]` | the same surrogate on any machine that has the key, no shared store needed; readable |
+| `RandomWords()` | `Lunar Cobweb` | a classic tokenisation vault: random, readable, no key |
+| `Aliases({...}, fallback=...)` | `the landlord` | you choose the pseudonym for some values yourself |
+
+`KeyedWords` is an HMAC-SHA-256 under the project key rendered as words from the EFF short wordlist (1,296 words, 10.3 bits each). The renderer stands alone as `nyms.readable(value, key=...)` and `nyms readable`. There is no unkeyed option: a plain hash of a CPR number is undone by enumerating the ten billion candidates.
+
+Re-identification is tolerant of what models do to bracketed tokens (case, spacing, a dropped bracket, a genitive after it); name-shaped surrogates and aliases are matched exactly, so ordinary prose is left alone. It never guesses: tokens it did not issue come back in `unmatched`.
+
+## Where the mapping and the key live
+
+The mapping store (surrogate, original, type) is the key to the text. By default it is a set of JSON files under `~/.local/share/nyms/<project>/mapping/`, and the project key is 32 random bytes in `~/.config/nyms/keys/<project>.key` (mode 0600), created on first use; `NYMS_KEY` in the environment overrides it, and `Pseudonymiser(..., key=...)` accepts any bytes. With `pip install "nyms[crypto]"` and `encrypt=True` the records are Fernet-encrypted under the key before they hit disk; without it they are plaintext in your user directory, which is the trade-off of needing no dependency. Lose the store and `[PERSON_1]` cannot be reversed; lose the key and an encrypted store cannot be read. Back both up, apart from each other.
+
+Any `MutableMapping` works as the store (`store=`), so [dol](https://github.com/i2mint/dol) backends (SQLite, S3) drop in, and `store={}` keeps a run in memory.
+
+## For AI agents
+
+<!-- epythet:agentic-readme:start -->
+`nyms` ships a skill for coding agents, in [Agent Skills](https://agentskills.io) format:
+
+```bash
+gh skill install thorwhalen/nyms nyms --agent claude-code     # or copilot, cursor, codex, gemini
+```
+
+| Skill | Use it to |
+|---|---|
+| `nyms` | pseudonymise documents before they reach a model, re-identify the answer, and check that nothing leaks |
+
+The skill folder also ships inside the wheel: `python -c "import nyms, os; print(os.path.join(nyms.__path__[0], 'data', 'skills'))"`.
+<!-- epythet:agentic-readme:end -->
+
+## Design
+
+The research behind the design (terminology and regulation, detection libraries and methods, surrogates and key management, pipeline architecture) is in [`docs/research/`](docs/research/), with a [synthesis](docs/research/deidentification-synthesis.md); the decisions are in [`docs/adr/`](docs/adr/). Six seams, each one keyword argument of `Pseudonymiser`: `detectors`, `post_detectors`, `policy`, `strategy`, `store`, `key`. The model-based second pass is the `post_detectors` seam: any callable `text -> spans`, run on the already-pseudonymised text with the surrogates protected; none ships in v1.
+
+The EFF wordlist is CC BY 3.0 (see `nyms/data/wordlists/NOTICE.md`). Everything else is MIT.

nyms-0.0.2/README.md

@@ -0,0 +1,90 @@
+# nyms
+
+Reversible pseudonymisation of text before it reaches an AI. A deterministic (non-AI) pass finds personal and sensitive identifiers in Danish and English text, replaces them with surrogates, remembers the mapping per project so the same person is `[PERSON_1]` in every document, and puts the originals back into whatever the AI answers.
+
+```bash
+pip install nyms            # add "nyms[crypto]" for an encrypted mapping store
+```
+
+```python
+from nyms import Pseudonymiser
+
+p = Pseudonymiser("case-2026", names=["Rasmus Østergaard Jensen"])
+r = p.pseudonymise("Rasmus Jensen (CPR 010190-1234) skylder 12.500 kr. Rasmus Jensens advokat ringede.")
+r.text
+# '[PERSON_1] (CPR [CPR_1]) skylder [AMOUNT_1]. [PERSON_1]s advokat ringede.'
+p.reidentify("Svar: [PERSON_1] skal betale [AMOUNT_1].").text
+# 'Svar: Rasmus Jensen skal betale 12.500 kr.'
+p.check("Rasmus Jensen bor her")      # what would leak if this were sent
+# ['Rasmus Jensen']
+```
+
+The same four verbs from the shell:
+
+```bash
+nyms pseudonymise --path letter.md --project case-2026 --names "Rasmus Østergaard Jensen" --out letter.pseud.md
+nyms reidentify --path answer.md --project case-2026 --out answer.clear.md
+nyms check --path draft.md --project case-2026       # {"ok": false, "leaks": [...]} if an original is in it
+nyms detect "Rasmus Jensen, tlf. 12 34 56 78"        # every span and the policy's decision; nothing stored
+nyms readable 010190-1234 --project case-2026        # a keyed hash as two readable words
+```
+
+Content is positional text or `--path` (a file, or `-` for stdin), never guessed from one another.
+
+```bash
+```
+
+`r.report` says what was replaced (by type and by detector), what was detected and kept (dates, postcodes and URLs, under the default policy) and which detectors ran, and ends with the caveat that this is pseudonymisation, not anonymisation.
+
+## What it finds
+
+Danish: CPR numbers (the date part must be valid; the pre-2007 modulus-11 check only raises confidence, because numbers issued since October 2007 need not pass it), CVR numbers (modulus-11 and context), DK VAT numbers, phone numbers, bank account numbers (registreringsnummer and kontonummer), IBAN (mod-97), street addresses with floor and postcode, postcode and town, amounts (`12.500 kr.`, `DKK 12.500`, `36.000,-`), dates, court and journal numbers. English: addresses, phone numbers, amounts (`£4,250.00`, `$5,000`), dates. Both: e-mail, URLs, IBAN, credentials (API keys, tokens, private keys), organisation names by legal form (`ApS`, `A/S`, `Ltd`, `GmbH`), names after a title (`Hr.`, `advokat`, `Mr`, `Dr`), names from a bundled Danish and English first-name list, names you pass in (`names=`), and a low-confidence heuristic for capitalised sequences.
+
+Once a value has been replaced anywhere in a project, every later occurrence is replaced too, including a Danish genitive (`Rasmus Jensens` becomes `[PERSON_1]s`), ALL-CAPS headings, and a surname or first name alone when it identifies exactly one known person (`Jensen blev` becomes `[PERSON_1] blev`). An identifier inside something the policy keeps (a CPR number in a URL) is still replaced. Pseudonymising already pseudonymised text changes nothing.
+
+It does not find: names it has never seen and the heuristic misses, roles and rare facts that identify without an identifier, inflected compounds, OCR-damaged identifiers. The report's `kept` list and the `check` command exist so a reader sees what stayed.
+
+## Choosing what is sensitive
+
+A `Policy` decides after detection, so the report can say what it saw and kept. `Policy.default()` replaces credentials, persons, CPR, e-mail, phone, addresses, IBAN, bank accounts, case numbers, CVR, organisations and amounts; `Policy.identifiers()` leaves amounts and organisations; `Policy.strict()` replaces everything detected, dates and postcodes included. `include`, `exclude`, per-type `min_score`, `allow` (never replace) and `deny` (always replace) lists refine any of them. The taxonomy behind the types, with tiers and the equivalent Presidio, Google DLP and HIPAA names, is `nyms.taxonomy.ENTITY_TYPES`.
+
+## Choosing the surrogates
+
+| Strategy | Example | When |
+|---|---|---|
+| `Counter()` (default) | `[PERSON_1]`, `[CPR_2]` | the form LLMs alter least in the one published benchmark; numbered per project |
+| `KeyedWords(key)` | `Alder Brook`, `[CPR:mustard-suitcase]` | the same surrogate on any machine that has the key, no shared store needed; readable |
+| `RandomWords()` | `Lunar Cobweb` | a classic tokenisation vault: random, readable, no key |
+| `Aliases({...}, fallback=...)` | `the landlord` | you choose the pseudonym for some values yourself |
+
+`KeyedWords` is an HMAC-SHA-256 under the project key rendered as words from the EFF short wordlist (1,296 words, 10.3 bits each). The renderer stands alone as `nyms.readable(value, key=...)` and `nyms readable`. There is no unkeyed option: a plain hash of a CPR number is undone by enumerating the ten billion candidates.
+
+Re-identification is tolerant of what models do to bracketed tokens (case, spacing, a dropped bracket, a genitive after it); name-shaped surrogates and aliases are matched exactly, so ordinary prose is left alone. It never guesses: tokens it did not issue come back in `unmatched`.
+
+## Where the mapping and the key live
+
+The mapping store (surrogate, original, type) is the key to the text. By default it is a set of JSON files under `~/.local/share/nyms/<project>/mapping/`, and the project key is 32 random bytes in `~/.config/nyms/keys/<project>.key` (mode 0600), created on first use; `NYMS_KEY` in the environment overrides it, and `Pseudonymiser(..., key=...)` accepts any bytes. With `pip install "nyms[crypto]"` and `encrypt=True` the records are Fernet-encrypted under the key before they hit disk; without it they are plaintext in your user directory, which is the trade-off of needing no dependency. Lose the store and `[PERSON_1]` cannot be reversed; lose the key and an encrypted store cannot be read. Back both up, apart from each other.
+
+Any `MutableMapping` works as the store (`store=`), so [dol](https://github.com/i2mint/dol) backends (SQLite, S3) drop in, and `store={}` keeps a run in memory.
+
+## For AI agents
+
+<!-- epythet:agentic-readme:start -->
+`nyms` ships a skill for coding agents, in [Agent Skills](https://agentskills.io) format:
+
+```bash
+gh skill install thorwhalen/nyms nyms --agent claude-code     # or copilot, cursor, codex, gemini
+```
+
+| Skill | Use it to |
+|---|---|
+| `nyms` | pseudonymise documents before they reach a model, re-identify the answer, and check that nothing leaks |
+
+The skill folder also ships inside the wheel: `python -c "import nyms, os; print(os.path.join(nyms.__path__[0], 'data', 'skills'))"`.
+<!-- epythet:agentic-readme:end -->
+
+## Design
+
+The research behind the design (terminology and regulation, detection libraries and methods, surrogates and key management, pipeline architecture) is in [`docs/research/`](docs/research/), with a [synthesis](docs/research/deidentification-synthesis.md); the decisions are in [`docs/adr/`](docs/adr/). Six seams, each one keyword argument of `Pseudonymiser`: `detectors`, `post_detectors`, `policy`, `strategy`, `store`, `key`. The model-based second pass is the `post_detectors` seam: any callable `text -> spans`, run on the already-pseudonymised text with the surrogates protected; none ships in v1.
+
+The EFF wordlist is CC BY 3.0 (see `nyms/data/wordlists/NOTICE.md`). Everything else is MIT.

nyms-0.0.2/docs/adr/0001-pseudonymisation-not-anonymisation.md

@@ -0,0 +1,35 @@
+# ADR 0001: nyms does pseudonymisation, and says so
+
+*2026-11-08 · status: accepted · research: [terminology-and-regulation](../research/terminology-and-regulation.md), [synthesis](../research/deidentification-synthesis.md)*
+
+## Context
+
+The package replaces personal and sensitive identifiers in text before the text reaches an AI service, and can put them back. The user's words for this were "redaction", "mapping to keys", "anonymising". The established vocabulary differs, and the difference carries legal weight: under GDPR Art. 4(5) data that can be re-attributed with separately held additional information is **pseudonymised** and remains personal data (Recital 26); only data that no reasonably likely means can re-attribute is anonymous. The EDPB's 2025 guidelines on pseudonymisation use the same terms and name the mapping and keys the **additional information**, kept outside the **pseudonymisation domain** [1].
+
+## Decision
+
+- The package, its API and its documentation say **pseudonymise** / **re-identify**, never "anonymise". The report of every run ends with a fixed caveat sentence (`nyms.core.CAVEAT`) saying so.
+- Vocabulary, used consistently: **detector** (finds spans), **span** (character interval with entity type and score), **policy** (which spans are acted on), **surrogate** (the replacement; a *pseudonym* when it names a person), **surrogate strategy** (how surrogates are generated), **mapping store** (surrogate to original, per **project**), **key** (the project's secret), **re-identification** (restoring originals), **leak check** (originals found in outgoing text).
+- British spelling throughout (`pseudonymise`), matching the EU sources that define the terms.
+- What nyms claims: identifiers it detected were replaced before the text left the machine, and the mapping stayed local. What it never claims: that the output is anonymous, or a numeric risk score. Residual risk is reported as coverage (what was detected, replaced, kept, and which detectors ran), not as a probability.
+
+## The seams (architecture-first, turn 1)
+
+| # | Seam | v1 default (no new dependency) | Replacement it allows |
+|---|---|---|---|
+| 1 | `detectors` | regex, checksum and gazetteer detectors (`nyms.detect.default_detectors`); any callable `text -> spans` | a Presidio adapter, more gazetteers |
+| 1b | `post_detectors` | none: stage 2 is off | the model-based second pass (spaCy/DaCy, GLiNER, an LLM through `aix` or `creel`), run on the already-pseudonymised text with the surrogates protected |
+| 2 | `policy` | `Policy.default()` (tiers 0 and 1, plus CVR, ORG, AMOUNT) | `strict()`, `identifiers()`, a serialised policy file |
+| 3 | `strategy` | `Counter` (`[PERSON_1]`) | `KeyedWords`, `RandomWords`, `Aliases`, format-preserving encryption |
+| 4 | `store` | `dol.JsonFiles` under `~/.local/share/nyms/<project>/mapping` | any `MutableMapping`; `encrypt=True` for Fernet at rest; SQLite, S3 through `dol` |
+| 5 | `key` | `resolve_key`: env var, then a 0600 key file, then a passphrase | `keyring`, `config2py`, `age`, a KMS |
+
+Surface for v1: Python facade, shipped agent skill, CLI (`cw`). The review of 2026-11-08 found the model pass could not be added as a plain detector (it runs on different text), so the stage became its own seam, `post_detectors`, with no implementation: the user asked for the choice, and the default choice is no second pass. MCP and HTTP: the tools module is the SSOT they would wrap; not built. NOT seams: the span data model, the report shape, the tolerant re-identification matcher, the overlap rule's signature.
+
+## Consequences
+
+Users and agents read "pseudonymised" and know the mapping is a key. The client-facing risk guide (a separate document in another repository) owns the legal framing and the advice on what not to send; nyms owns the numbers a run can compute.
+
+## REFERENCES
+
+1. [EDPB, Guidelines 01/2025 on Pseudonymisation](https://www.edpb.europa.eu/system/files/2025-01/edpb_guidelines_202501_pseudonymisation_en.pdf)

nyms-0.0.2/docs/adr/0002-sensitivity-policy-and-entity-taxonomy.md

@@ -0,0 +1,25 @@
+# ADR 0002: sensitivity policy and entity taxonomy
+
+*2026-11-08 · status: accepted · research: [terminology-and-regulation](../research/terminology-and-regulation.md) §3 and §5*
+
+## Context
+
+The user asked to choose what counts as sensitive, "before a sensitivity detection process, or even after one, once a list of items has been extracted", and for amounts and other non-identifying sensitive values to be in scope. Existing tools pick entities before detection (Presidio's `entities=` argument) and threshold scores after; Google DLP attaches a sensitivity level to each type; HIPAA's 18 classes and GDPR Art. 9 define legal sensitivity. No tool has Danish entity types.
+
+## Decision
+
+**Taxonomy** (`nyms.taxonomy.ENTITY_TYPES`): a small registry with three axes per type: identifier class (credential, direct, quasi, sensitive, organisation), tier (0 block, 1 replace by default, 2 policy decides) and aliases to Presidio, DLP and HIPAA names. Fifteen types in v1: CREDENTIAL, PERSON, CPR, EMAIL, PHONE, ADDRESS, IBAN, BANK_ACCOUNT, CASE_ID (tiers 0 and 1); CVR, ORG, AMOUNT, DATE, POSTCODE_CITY, URL (tier 2). The registry is a dict so a caller adds a type by assignment.
+
+**Policy** (`nyms.policy.Policy`): a dataclass evaluated **after** detection: `include`/`exclude` types, `min_score` (global and per type), `allow` (never replace) and `deny` (always replace) lists on span text, and a `keep` hook. All detectors run; the policy decides; the run report lists what was detected and kept. The same object also filters detectors **before** detection (`detector_filter`, `prefilter=True` on the facade) for callers who want to skip work. Presets: `default` (tiers 0 and 1 plus CVR, ORG, AMOUNT), `identifiers` (tiers 0 and 1), `strict` (everything detected, no threshold). `with_names(...)` adds the parties to the deny list. Policies serialise to and from dicts.
+
+**Order of operations in a run**: every detector's spans, overlap resolution (ADR 0003), the policy's split into replaced and kept, then a second pass with the project's known values (ADR 0003). Tier 0 (credentials) is replaced whatever `include` says unless excluded explicitly.
+
+## Alternatives considered
+
+- Policy before detection only: cheaper, but the report could not say what was seen and kept. Rejected as the only mode; offered as `prefilter`.
+- Per-rule action lists (replace, mask, redact, generalise) as in the research's proposed object: deferred. v1 has two actions, replace and keep. Masking and date generalisation are additions at this seam.
+- Art. 9 as a boolean flag per type: deferred with the above; the identifier class carries "sensitive" for now.
+
+## Consequences
+
+Dates, postcodes and URLs stay in the text by default and appear in `report["kept"]`, so a user who needs them replaced sees them and switches to `strict` or adds the type. Amounts are replaced by default because the first use case is a legal and financial dispute; `identifiers` keeps them.

nyms-0.0.2/docs/adr/0003-detector-strategies.md

@@ -0,0 +1,27 @@
+# ADR 0003: detector strategies: a deterministic pass first, a model pass as a seam
+
+*2026-11-08 · status: accepted · research: [detection-libraries-and-methods](../research/detection-libraries-and-methods.md), [pipeline-architecture-and-agent-practice](../research/pipeline-architecture-and-agent-practice.md)*
+
+## Context
+
+The user's requirement: a non-AI pass runs before any content reaches a model, with an optional heavier model-based pass afterwards. The research found no library with Danish recognisers (Presidio has none; Google DLP has only the CPR type), that the CPR modulus-11 check is unreliable for numbers issued since October 2007, that recall is the metric de-identification systems optimise, and that the best Danish NER options (spaCy `da_core_news`, DaCy, GLiNER PII checkpoints) are heavy, optional dependencies.
+
+## Decision
+
+- **Native deterministic detectors** (`nyms.detect`): a `Detector` is a compiled regex plus an optional validator (checksum, calendar, stop list) plus context words that raise the score. v1 ships: CPR (date check required, mod-11 only raises the score), CVR (mod-11 plus context), DK VAT, e-mail, Danish and international phone, IBAN (mod-97 required), Danish bank account, Danish and English addresses, Danish postcode and town, amounts (prefix or suffix currency, Danish thousand separators), dates (numeric, Danish and English textual), case and journal numbers, URLs, credentials (vendor prefixes and one assignment heuristic, after `openloops.egress`), organisation names by legal-form suffix, titled names, a capitalised-sequence name heuristic at a low score, a bundled first-name gazetteer (Danish and English) and the caller's known names.
+- **Overlap resolution** (`nyms.spans.resolve_overlaps`): strong spans (score at or above 0.6) outrank weak ones whatever their length; among equals the longer, then the higher score, then the earlier. A function, so it is testable and replaceable.
+- **Two passes for consistency**: after the first pass enters new values into the mapping, a second pass adds a detector per entity type for every value the project already knows, so an occurrence scored too low (a name at a sentence start, a company in a heading) is replaced once the value was accepted anywhere, in this document or an earlier one. This is the benchmark engine's "propagate to every mention" behaviour.
+- **Genitive handling**: detectors never strip anything (a surname may end in `s`: "Niels Thomas"). The mapping store maps `<known value>s` to the known value's surrogate plus the suffix, and the facade writes `[PERSON_1]s`; a genitive seen before its base form is taken literally and relinked when the base form appears.
+- **Nested promotion**: an accepted span inside a span the policy keeps (a CPR number inside a URL) is still replaced; the kept span stays in the report.
+- **Name components**: a surname or first name that identifies exactly one known person (from `names=` or the mapping) stands for that person at score 0.8, so "Jensen blev" becomes `[PERSON_1] blev`. Capitalised words include all-capitals headings (`METTE HOLM`), hyphenated names and initials (`H.C. Andersen`). Letters are the Latin letters of Unicode's blocks up to Latin Extended-B, and text is NFC-normalised first.
+- **Surrogate protection**: the project's own surrogates are detected as a pseudo-type that wins overlap resolution and is never acted on, so pseudonymising already pseudonymised text is idempotent.
+- **Model-based second pass**: the `post_detectors` seam (added after the adversarial review showed a plain detector could not do it, since it runs on different text). Any callable `text -> Iterable[Span]`; it runs on the already-pseudonymised text, so the model never sees what the first pass caught, with the surrogates protected from re-detection; what it finds is replaced in that text and added to the mapping, and the report lists it under its own name. Default: none. Candidates: DaCy or spaCy Danish models, GLiNER PII (Apache-2.0), an LLM through `aix` or `creel`. Piiranha and the ai4privacy data are excluded by licence.
+- **Recall over precision**, with the policy threshold as the user's dial; the heuristic exists because a missed name leaks and a redundant replacement only costs readability.
+
+## Consequences
+
+False positives occur (a capitalised phrase replaced as a person); they are visible in the report and reversible. A bare eight-digit number is ambiguous between phone and CVR and is decided by checksum and context. OCR noise and inflected compounds are not handled in v1 (documented limits).
+
+## Fixtures
+
+Synthetic Danish and English documents under `tests/fixtures/`, with `expected.json` naming what must be replaced and what must be detected but kept; a leakage test (no original, nor its digit string, survives) and a round-trip test per fixture. No real data; the CPR and CVR numbers are checksum-valid synthetic values or deliberately invalid ones.

nyms-0.0.2/docs/adr/0004-surrogate-generation.md

@@ -0,0 +1,28 @@
+# ADR 0004: surrogate generation: typed counters by default, keyed readable words as the tool
+
+*2026-11-08 · status: accepted · research: [surrogates-crypto-and-keys](../research/surrogates-crypto-and-keys.md), [pipeline-architecture-and-agent-practice](../research/pipeline-architecture-and-agent-practice.md) §2*
+
+## Context
+
+The user listed the strategies to support: aliases chosen by hand, plain numbering, a cryptographic hash, and "a cryptographic hash mapped to a sequence of letters that is actually readable", asking for the last one to be built. Two research reports disagreed on the default. The surrogate report reasoned that word-shaped tokens survive LLM paraphrase best; the pipeline report found one measured benchmark (bilingual, single vendor, 2026) in which plain bracketed typed tokens such as `[PERSON_1]` were altered least by models (1 in 8,099 occurrences, against 101 in 5,099 for XML-style tags) and restored at 99.99% with a tolerant restorer. Readable surrogates were not measured. Unkeyed hashes of low-entropy identifiers (a CPR number has at most about 10^10 values) are reversible by enumeration and are rejected by the EDPB's guidance.
+
+## Decision
+
+- A **strategy** is a callable `(entity_type, value, *, vault) -> str`, called once per new value by the mapping store; `vault` exposes `__contains__` and `count(entity_type)`.
+- **Default: `Counter`**, `[PERSON_1]`, numbered per type within the project, persisted with the store so the numbering continues across documents. Measured evidence beats reasoning; the counter carries no information about the value.
+- **`KeyedWords`**: HMAC-SHA-256 of `entity_type || NUL || value` under the project key, rendered as words. The same key gives the same surrogate on any machine without sharing a store. The key must be secret and at least 16 bytes; there is no unkeyed option. Collisions with an existing surrogate add a word.
+- **`RandomWords`**: the classic tokenisation vault, random bytes rendered as words.
+- **`Aliases`**: the user's own surrogates for chosen values, any other strategy as the fallback.
+- **Rendering is separate from the source of bits**: `words_from_bytes(data, n, wordlist)` writes the integer in base `len(wordlist)`, most significant word first. The bundled list is the EFF short wordlist 2.0 (1,296 common words, unique three-letter prefixes, CC BY 3.0, attributed in `nyms/data/wordlists/NOTICE.md`): 10.3 bits per word, two words for about 1.7 million values. `syllables_from_bytes` is the list-free fallback. The `readable()` function and the `nyms readable` command expose the renderer stand-alone.
+- **Shape**: word surrogates use a template per entity type, `Alder Brook` (name-shaped) for PERSON and ORG, `[CPR:alder-brook]` for everything else. Numbers never get checksum-valid fake numbers: a surrogate that could pass for a real identifier is a lie the reader cannot detect.
+- **Re-identification is tolerant**: brackets optional, separators loose, case ignored, a genitive after the closing bracket allowed, longest surrogate first, and a word boundary required when the bracket is missing so `PERSON_1` never matches inside `PERSON_11`. Tokens that look like ours but were never issued are reported as `unmatched`, never guessed.
+
+## Alternatives considered
+
+- Format-preserving encryption (FF1): a replacement for numeric identifiers only; NIST's 2025 draft keeps FF1 alone and the maintained Python library implements the withdrawn FF3 variants. Not built.
+- Seeded Faker names: realistic, but a fake that matches a real person is a risk and the dependency is large. A replacement at this seam.
+- XML-style tags: measured worse. Rejected.
+
+## Consequences
+
+Text sent to a model reads as `[PERSON_1] owes [AMOUNT_1]`, which is less fluent than a readable name but round-trips best. A user who wants fluent text switches to `KeyedWords` with one argument and accepts an unmeasured round-trip risk, which `check()` and the `unmatched` list make visible.

nyms-0.0.2/docs/adr/0005-mapping-store-and-key-management.md

@@ -0,0 +1,27 @@
+# ADR 0005: the reversible mapping store and key management
+
+*2026-11-08 · status: accepted · research: [surrogates-crypto-and-keys](../research/surrogates-crypto-and-keys.md) §3 and §4, [pipeline-architecture-and-agent-practice](../research/pipeline-architecture-and-agent-practice.md) §2*
+
+## Context
+
+The mapping (surrogate to original) and the key are the "additional information" of GDPR Art. 4(5): whoever holds them re-identifies everything. Every LLM privacy gateway surveyed keeps such a store and warns that it is sensitive; Presidio keeps none and so cannot be consistent across documents. The in-house storage facade is `dol`; the in-house secret resolution is `config2py`; `cryptography` is installed but is not a dependency of a minimal package.
+
+## Decision
+
+- **Store** (`nyms.mapping.MappingStore`): one record per distinct value (surrogate, original, entity type, normalised form, strategy, time), keyed by surrogate, indexed in memory by `(entity_type, normalised)`. Normalisation decides identity: digits only for numeric identifiers, upper-case alphanumerics for IBAN and case numbers, lower case for e-mail and URL, casefolded and whitespace-collapsed for the rest.
+- **Backing store**: any `MutableMapping[str, dict]`. Default `local_mapping_store(project)`: `dol.JsonFiles` under `~/.local/share/nyms/<project>/mapping/` (XDG-aware), one JSON file per record, directory mode 0700. A `dict` is the in-memory store for tests and one-off runs. The repository never holds a store; the default root is outside any project directory by design.
+- **Encryption at rest is a seam** (`codec=`): `encrypted(key)` gives a Fernet codec (AES-128-CBC with HMAC-SHA-256) from the optional `cryptography` dependency (`pip install "nyms[crypto]"`, `encrypt=True` on the facade). The default is plaintext in the user's data directory because the standard library has no authenticated cipher and a default must not need a new dependency. The trade-off is stated in the README.
+- **Key** (`nyms.keys.resolve_key`): the `NYMS_KEY` environment variable (hex or base64), then a per-project key file under `~/.config/nyms/keys/<project>.key` (32 random bytes, mode 0600, created on first use), then, when a passphrase is given, scrypt derivation with a stored per-project salt. The key file and the mapping directory are never in the same tree. Replacements: `keyring` (OS keychain), `config2py`, `age`, a KMS, each one `key=` argument.
+- **Rotation** is a new project. Keyed-word surrogates change with the key by construction; counter surrogates survive it because they live in the store. Re-keying an encrypted store means decrypting and re-encrypting the records.
+- **Loss**: lose the store and counter surrogates cannot be reversed; lose the key and an encrypted store cannot be read and keyed-word surrogates cannot be regenerated. Both are stated in the README; a backup is the user's job.
+- `MappingStore.to_dict()` and the `nyms mapping` command print originals. They exist because the user owns the data; the docs mark them sensitive.
+
+## Alternatives considered
+
+- Append-only log with compaction: more robust for concurrent writers, more code. One file per record on `dol` gives inspectability and `dol`'s backends for free; deferred.
+- OS keyring as the default key store: the right replacement, but a new dependency and an interactive prompt on first use. Not the default.
+- Per-document keys: destroys cross-document consistency, which is the point of a project. Rejected.
+
+## Consequences
+
+A project is the unit of consistency, of key and of store. Two sessions on one project write to the same directory; `dol.JsonFiles` writes are per record, so interleaving is safe at record granularity but the in-memory counter can issue a duplicate under true concurrency. Not a v1 concern; noted for the store seam.

nyms-0.0.2/docs/research/deidentification-synthesis.md

@@ -0,0 +1,60 @@
+# De-identification toolkit research synthesis: what nyms uses and why
+
+*2026-11-08 · scope: the four theme reports in this folder · requested for: the design of `nyms`, a reversible pseudonymisation toolkit for Danish and English text sent to AI services*
+
+## Summary and recommendation
+
+- **Name the thing correctly**: pseudonymisation (GDPR Art. 4(5)), with the mapping and key as the "additional information" kept outside the pseudonymisation domain. Never claim anonymity; report coverage, not a risk score. The EDPB's 2025 guidelines and the CJEU's 2025 *EDPS v SRB* judgment are the framing; the final post-consultation EDPB text was not confirmed [1][2].
+- **Write the deterministic detectors natively.** No library has Danish recognisers; Presidio, the most mature, has none, and Google DLP has only the CPR type [3][4]. The CPR modulus-11 check cannot be a gate (numbers issued since October 2007 need not pass it) [5]. Study Presidio's span contract, context scoring and overlap rules; do not depend on it.
+- **Default surrogate: `[PERSON_1]`-style bracketed typed counters**, per project, because the only measured benchmark found them altered least by LLMs and restored at 99.99% with a tolerant restorer [6]. Readable keyed-word surrogates are the opt-in strategy and the stand-alone tool the user asked for; the EFF short wordlist (CC BY 3.0) renders them [7]. Unkeyed hashes are rejected [1].
+- **The store is the product's state**: a per-project `dol` store with the key held separately, encryption at rest as an optional codec, propagation of known values to every mention, and a tolerant, non-guessing re-identifier that reports invented tokens. Presidio's statelessness is the gap every LLM gateway had to fill [8][9].
+- **Agent surface**: four verbs (`pseudonymise`, `reidentify`, `check`, `detect`) over one facade, a shipped skill that makes the deterministic pass the first step before any content reaches a model, and a leak check that blocks rather than warns.
+
+## Where the reports agree
+
+All four put a detect / transform / reverse split at the centre, with character-offset spans and a score, and all treat the mapping as sensitive. All say recall beats precision for this task and that the residual risk must be stated in words, not a number. All find Danish support absent from existing tools.
+
+## Where they disagree, and the resolution
+
+| Question | Surrogates report | Pipeline report | Resolution in nyms |
+|---|---|---|---|
+| Default surrogate form | random readable word tokens, name-shaped | bracketed typed token `[PERSON_1]` | **Counter default** (measured), `KeyedWords`/`RandomWords` opt-in (ADR 0004) |
+| Default key store | OS keyring | `config2py` | **key file under the config dir**, env var override, passphrase option; keyring and config2py as replacements (ADR 0005) |
+| Encryption at rest | required default | encrypted file | **optional codec** (no stdlib cipher; a default must not need a new dependency); README states the trade-off (ADR 0005) |
+
+## What the build uses
+
+| Part | From the research | Implementation |
+|---|---|---|
+| Vocabulary and caveat | terminology report §1, §2 | ADR 0001, `nyms.core.CAVEAT` |
+| Taxonomy and policy | terminology report §3, §5 | ADR 0002, `nyms.taxonomy`, `nyms.policy` |
+| Detectors, overlap rule, two passes | detection report §1–§3, pipeline report §2 | ADR 0003, `nyms.detect`, `nyms.spans` |
+| Strategies, word rendering, tolerant restorer | surrogates report §1–§2, pipeline report §2 | ADR 0004, `nyms.surrogates`, `nyms.core.reidentify` |
+| Store, key, rotation, loss | surrogates report §3–§4 | ADR 0005, `nyms.mapping`, `nyms.keys` |
+| Fixtures and tests | detection report §4, pipeline report §5 | `tests/fixtures/`, leakage and round-trip tests; Faker's `da_DK` has no CPR generator, so fixtures carry hand-made values |
+
+## In-house components reused
+
+`dol` (the store facade), `cw` (the CLI from the tools module), the pattern design of `openloops.egress` (credential prefixes, one heuristic rule last). Found but not used in v1: `hm` (Major System; a rendering aid, not a hash encoding), `creel` and `aix` (candidates for the model-based second pass), `config2py` (a key-resolution replacement).
+
+## Open questions carried forward
+
+1. Does the bracketed-token advantage hold for Danish and for current frontier models? Rerun the benchmark method on the Danish fixtures (an issue).
+2. Which Danish NER (DaCy, spaCy, GLiNER PII) is good enough for the second pass, measured by leakage rate on the fixtures?
+3. Is the final EDPB 01/2025 text changed on hashing and key separation?
+4. A Danish name list with a redistributable licence: Danmarks Statistik's terms were not found.
+5. Quasi-identifier actions (date shifting, amount bucketing) and Art. 9 flags: deferred from the policy (ADR 0002).
+
+## REFERENCES
+
+1. [EDPB, Guidelines 01/2025 on Pseudonymisation](https://www.edpb.europa.eu/system/files/2025-01/edpb_guidelines_202501_pseudonymisation_en.pdf)
+2. [EUR-Lex, Case C-413/23 P, EDPS v SRB](https://eur-lex.europa.eu/eli/C/2025/5551/oj/eng)
+3. [Presidio, supported entities](https://presidio.dataprivacystack.org/supported_entities/)
+4. [Google Sensitive Data Protection, infoType reference](https://docs.cloud.google.com/sensitive-data-protection/docs/infotypes-reference)
+5. [LookupTax, Denmark TIN: CPR and CVR number guide](https://www.lookuptax.com/docs/tax-identification-number/denmark-tax-id-guide)
+6. [Beskorovainyi V. Reversible Pseudonymisation of Documents Sent to External LLMs, arXiv:2610.34012 (2026)](https://arxiv.org/html/2610.34012)
+7. [EFF, Deep Dive: EFF's New Wordlists for Random Passphrases](https://www.eff.org/dice)
+8. [Presidio Anonymizer documentation](https://presidio.dataprivacystack.org/anonymizer/)
+9. [LLM Guard Anonymize scanner](https://protectai.github.io/llm-guard/input_scanners/anonymize)
+
+The theme reports: [terminology-and-regulation](terminology-and-regulation.md), [detection-libraries-and-methods](detection-libraries-and-methods.md), [surrogates-crypto-and-keys](surrogates-crypto-and-keys.md), [pipeline-architecture-and-agent-practice](pipeline-architecture-and-agent-practice.md). Prompts used: [prompts/](prompts/).