jev-secrets 1.0.0__tar.gz

jev_secrets-1.0.0/.gitignore

@@ -0,0 +1,11 @@
+/vendor/
+/node_modules/
+/js/dist/
+.env
+.phpunit.result.cache
+__pycache__/
+.pytest_cache/
+.ruff_cache/
+.venv/
+/dist/
+*.egg-info/

jev_secrets-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Fox Islam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

jev_secrets-1.0.0/PKG-INFO

@@ -0,0 +1,342 @@
+Metadata-Version: 2.5
+Name: jev-secrets
+Version: 1.0.0
+Summary: Reversible PII redaction for Jev requests
+Author-email: Fox Islam <foxislam@outlook.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: jev,pii,pseudonymisation,redaction,system-one,typesafe
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Security
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: sdk
+Requires-Dist: typesafe-sdk<1,>=0.7; extra == 'sdk'
+Description-Content-Type: text/markdown
+
+# jev-secrets
+
+Replaces names, keys and other secrets in a [Jev](https://docs.typesafe.ai) request with
+stand-ins before it is sent, and puts the originals back in the answers. For the
+[PHP SDK](https://github.com/Fox-Islam/typesafe-sdk-php) and the official
+[JavaScript](https://www.npmjs.com/package/@typesafe-ai/sdk) and
+[Python](https://pypi.org/project/typesafe-sdk/) SDKs, or any code that builds the request body
+itself.
+
+```
+John Smith <john.smith@acme.com> was charged twice on 2024-03-15. Key sk_live_4eC39HqLyjWD
+Tozd Cpabl <tozd.cpabl@ulco.zuy> was charged twice on 2024-03-15. Key sk_live_3uF21GlNlxHG
+```
+
+The second line is what Jev sees, with `John Smith` passed as a value and the rest found by the
+default patterns. In each replaced word every letter becomes another letter of the same case,
+vowels stay vowels, digits become digits, and punctuation stays, so a name still reads as a name
+and an email address as an email address. The same word gets the same stand-in everywhere in the
+request, including in the questions. The answers come back with the original question ids and
+choice labels.
+
+This is pseudonymisation, not anonymisation: the mapping is held in memory in order to be able
+to reverse the process on the way back to the caller.
+
+**Contents:** [Install](#install) · [Use](#use) · [What gets replaced](#what-gets-replaced) ·
+[What a stand-in looks like](#what-a-stand-in-looks-like) · [The key](#the-key) ·
+[What is rewritten and restored](#what-is-rewritten-and-restored) · [Config](#config) ·
+[When it refuses to send](#when-it-refuses-to-send) · [What it does not do](#what-it-does-not-do) ·
+[Does it change the answers?](#does-it-change-the-answers)
+
+## Install
+
+```sh
+composer require phox/jev-secrets        # PHP 8.3+, with ext-intl and ext-mbstring
+npm install @phox-js/jev-secrets         # Node 20.19+
+pip install jev-secrets                  # Python 3.10+; jev-secrets[sdk] adds the SDK wrapper
+```
+
+| | Tested on | SDK versions the wrapper is tested with |
+| --- | --- | --- |
+| PHP | 8.3, 8.4, 8.5 | `phox/typesafe-sdk-php` 0.3 and 0.4 |
+| JavaScript | Node 20.19, 22, 24 | `@typesafe-ai/sdk` 0.5.7 and 0.6.0 |
+| Python | 3.10 to 3.14 | `typesafe-sdk` 0.7.0 and 0.7.1 |
+
+A request of 1000 strings, 500 KB of JSON, takes 0.4-0.5 s in PHP and JavaScript and 1.1-1.4 s
+in Python; [docs/performance.md](docs/performance.md) has the measurements.
+
+## Use
+
+With the SDK, wrap the client. Each call takes the values to hide in that request, on top of
+whatever the config finds:
+
+```php
+use Phox\JevSecrets\Secrets;
+use Phox\JevSecrets\TypeSafe\SecretClient;
+
+$client = new SecretClient(new \Phox\TypeSafe\Client(), new Secrets(['terms' => ['Acme Ltd']]));
+
+$response = $client->systemOne()
+    ->state(['ticket' => $ticket])
+    ->noul('refund', 'Does John Smith ask for a refund?')
+    ->values(['John Smith'])
+    ->send();
+```
+
+```js
+import { TypeSafeClient } from '@typesafe-ai/sdk';
+import { Secrets, withSecrets } from '@phox-js/jev-secrets';
+
+const client = withSecrets(new TypeSafeClient(), new Secrets({ terms: ['Acme Ltd'] }));
+const { answers } = await client.systemOne(
+    { state: { ticket }, questions: { refund: { type: 'noul', instructions: 'Does John Smith ask for a refund?' } } },
+    { values: ['John Smith'] },
+);
+```
+
+```python
+from typesafe_sdk import TypeSafeClient
+from jev_secrets import Secrets
+from jev_secrets.typesafe import SecretClient
+
+client = SecretClient(TypeSafeClient(), Secrets({'terms': ['Acme Ltd']}))
+result = client.system_one(
+    state={'ticket': ticket},
+    questions={'refund': {'type': 'noul', 'instructions': 'Does John Smith ask for a refund?'}},
+    values=['John Smith'],
+)
+```
+
+`jev_secrets.typesafe.AsyncSecretClient` wraps the async Python client. The wrappers sit above
+the SDK, not in its transport, because the SDKs log request bodies at debug level before the
+transport sees them.
+
+Any config option can be changed at call time. `scope` and `dates` merge entry by entry; any
+other option given replaces the configured one:
+
+```php
+$client->systemOne()->state($ticket)->noul('refund', 'Refund?')
+    ->overrides(['disable' => ['phone'], 'dates' => ['order' => 'mdy']])
+    ->send();
+```
+
+In JavaScript it is `options.overrides`, in Python `overrides=`, and without an SDK the third
+argument of `pseudo()`.
+
+Without an SDK, rewrite the body yourself and restore the decoded response:
+
+```php
+$done = (new Secrets())->pseudo($body, ['John Smith']);
+$answers = $done->restore(json_decode(send($done->request), true));
+```
+
+`pseudo()` is also available as `pseudonymise()`, `pseudonymize()` and `mask()`, and `restore()`
+as `unmask()`.
+
+`$done->replacements` lists what was replaced, as a path into the sent request and the source
+that found it, without the values themselves.
+
+## What gets replaced
+
+The sources, all optional:
+
+| Source | What it matches |
+| --- | --- |
+| Values passed to the call | those strings |
+| Config | `terms` (fixed strings), `fields` (paths whose whole value is replaced), `patterns` (your regexes) |
+| Defaults | the patterns below, and the value under a key such as `password`, `token` or `api_key` |
+
+`secretKeys` controls those key names: `false` for none, a list for exactly those, or
+`{"add": [...], "remove": [...]}` to change the default list. Names compare lowercased,
+ignoring spaces, hyphens and underscores, so `member_number` covers `Member-Number`.
+
+Values and terms match case-insensitively, at word boundaries. Their words may be joined by
+any run of spaces, underscores, hyphens or dots, so `Ellen Park` is also found in
+`is_from_ellen_park` and `ellen.park@example.com`. A value shorter than 2 characters is refused.
+The value found at a field path is also replaced wherever else it appears in the request.
+
+**Names cannot be found by pattern.** Pass them as values, list them as terms, or name the
+fields they're in.
+
+Dates, and numbers the default patterns do not cover, are not replaced by default. Pass one as a
+value, or name its field, and a number is replaced digit by digit, a date or time moved
+([below](#what-a-stand-in-looks-like)).
+With `dates.detect: true` every date and time in the request is moved.
+
+<!-- patterns -->
+| Default pattern | Finds |
+| --- | --- |
+| `email` | an email address |
+| `ipv4` | a dotted-quad IPv4 address |
+| `ipv6` | an IPv6 address, full or compressed, with at least one decimal digit |
+| `card` | a card number of 12 to 19 digits that passes the Luhn check |
+| `iban` | an IBAN that passes the mod-97 check |
+| `phone` | a phone number written with a leading `+` and 8 to 15 digits |
+| `us-ssn` | a US Social Security number written with dashes |
+| `uk-nino` | a UK National Insurance number |
+| `aws-access-key` | an AWS access key id, keeping its prefix, such as `AKIA` |
+| `github-token` | a GitHub token, keeping its prefix, such as `ghp_` |
+| `github-pat` | a fine-grained GitHub token, keeping the `github_pat_` prefix |
+| `stripe-key` | a Stripe secret or restricted key, keeping its prefix, such as `sk_live_` |
+| `slack-token` | a Slack token, keeping its prefix, such as `xoxb-` |
+| `anthropic-key` | an Anthropic API key, keeping its prefix, such as `sk-ant-api03-` |
+| `openai-key` | an OpenAI API key, keeping its prefix, such as `sk-proj-` |
+| `google-api-key` | a Google API key, keeping the `AIza` prefix |
+| `jwt` | a JSON Web Token, keeping the leading `eyJ` |
+| `private-key` | the body of a PEM private key, keeping the BEGIN and END lines |
+| `bearer-token` | a bearer token of 16 or more characters, keeping the word `Bearer` |
+| `credential-assignment` | a value after `password=`, `api_key:`, `token=` and the like in text, keeping the name |
+<!-- /patterns -->
+
+Where two matches overlap they merge, so a value inside an email address takes the rest of the
+address with it. `catalogue/patterns.json` holds the patterns and the key names; the PHP,
+JavaScript and Python implementations read the same file.
+
+## What a stand-in looks like
+
+Recorded outputs of the character generator under a fixed key, from `catalogue/fixtures.json`:
+
+<!-- stand-ins -->
+| Input | Stand-in |
+| --- | --- |
+| `John` / `JOHN` | `Tozd` / `TOZD` |
+| `O'Brien-Smythe` | `I'Ghees-Dmjcbu` |
+| `José Müller` | `Taré Hömsif` |
+| `Дмитрий Иванов` | `Ьхэвлаӗ Ычозюв` |
+| `김민준` | `펇쫹떀` |
+| `2024-12-31 23:59` | `1011-11-21 12:33` |
+| `10.0.0.1` | `22.9.9.7` |
+<!-- /stand-ins -->
+
+- **Letters:** each becomes another from the same alphabet and case. With `letters: "shape"`,
+  the default, vowels stay vowels and consonants stay consonants, so the stand-in is
+  pronounceable. Latin, Greek, Cyrillic, Armenian, Georgian, Hebrew, Arabic, Devanagari, Thai,
+  kana, Hangul and CJK are drawn from their own scripts; a letter from any other script becomes
+  an ASCII letter. Accents stay on.
+- **Digits:** with `digits: "lower"`, the default, 0 and 1 stay and every other digit becomes a
+  lower one of at least 1, so an IP address or a version stays valid. A span made only of 0s
+  and 1s, like `10.0.0.1`, is drawn under `digits: "any"` instead.
+- **Dates and times** are moved, not rewritten: every date in the request by the same number of
+  weeks, every time by the same number of minutes, without crossing midnight. Order, intervals
+  and weekdays survive, and the format stays as written, so `Monday, 16 March 2026 at 3:40 pm`
+  becomes `Monday, 12 October 2026 at 11:46 am`. `dates.order` decides whether `03/04/2026` is
+  3 April (`dmy`, the default) or 4 March (`mdy`); a year written first is always year, month,
+  day. With `dates.mode: "digits"` they get the digit rule instead, which is how the
+  `2024-12-31 23:59` row above was made. [docs/dates.md](docs/dates.md) lists the formats.
+- **Everything else** stays: spaces, punctuation, `@`, `.`, `-`, `_`.
+
+A stand-in is drawn again until it differs from every word already in the request and from
+every other stand-in. Short numbers under the digit rule can run out of stand-ins and share one;
+[docs/generation.md](docs/generation.md) has the exact rules.
+
+## The key
+
+Stand-ins and date offsets are drawn under a key:
+
+| `key` | What each call uses |
+| --- | --- |
+| left out | the `JEV_SECRETS_KEY` environment variable, read on every call; without it, a random key |
+| a string | that string |
+| `false` | a random key, whatever the environment holds |
+
+With a random key the stand-ins change from call to call. With a fixed one the same request
+gives the same stand-ins every time, in every implementation. Nothing is stored between calls,
+so rotating the key costs nothing: calls after the change get different stand-ins. The library
+reads the process environment, not a `.env` file; load that the way your framework does.
+
+## What is rewritten and restored
+
+| Part of the request | Rewritten | Restored in the answers |
+| --- | --- | --- |
+| `state` strings and numbers | yes | - |
+| `state` object keys | with `scope.keys` | - |
+| question `instructions` and `criteria` text | yes | a score's `legend`, exactly as sent |
+| question ids | where they contain a match | the answer keys |
+| choice labels | where they contain a match | `choice` and the keys of `probabilities` |
+| noul criteria keys, `type`, `model`, anything else at the top level | never | - |
+
+A score's legend keys and probability keys are rubric indices and are never restored, nor is
+any number in the response. A string field in an answer that this library does not know gets
+word-by-word replacement of the stand-ins.
+
+## Config
+
+The same object in PHP, JavaScript and Python.
+[spec/config.schema.json](spec/config.schema.json) has every option and its default.
+
+```json
+{
+  "terms": ["Acme Ltd", "Project Nightingale"],
+  "fields": ["state.customer.name", "state.orders.*.email"],
+  "patterns": [{ "id": "order-id", "pattern": "ORD-(?<value>[0-9]{6})" }],
+  "allow": ["support@acme.com"],
+  "disable": ["ipv4"],
+  "scope": { "keys": true },
+  "secretKeys": { "add": ["member_number"] },
+  "dates": { "order": "mdy", "detect": true }
+}
+```
+
+`allow` protects a string from terms, fields, patterns and the defaults; a value passed to a
+call still replaces it.
+
+## When it refuses to send
+
+After rewriting, every original value is looked for again across every string and key under
+`state` and `questions`. If one is still there, the call throws `LeakException` (`LeakError` in
+JavaScript and Python) and nothing is sent. The message names the path and the source, never
+the value, and suggests the fix for the usual causes: a value in an object key while
+`scope.keys` is off, or in the questions while `scope.questions` is off.
+
+In PHP, a request that cannot be searched is refused too: text that is not valid UTF-8, or a
+regex that stops at PCRE's backtrack limit (`MatchException`). Matching cannot say what it
+missed, so nothing is sent. JavaScript and Python strings are always Unicode, and their regexes
+have no such limit.
+
+A config that cannot work is refused when it is made: an unknown option, a pattern that does not
+compile, a term with no letter or digit or shorter than `minLength`. A value passed to a call is
+checked the same way when the call is made.
+
+## What it does not do
+
+- **Find names by itself.** Only values, terms and fields hide names.
+- **Hide shape.** A stand-in has the same length, capitals, punctuation and word count as the
+  original, and the same word always gets the same stand-in, so the number of times a name
+  appears is visible.
+- **Hide every digit.** Under the default digit rule 0 and 1 are kept and 2 can only become 1.
+- **Keep order between numbers.** Each number is lowered on its own, so 19 and 22 can come out
+  as 17 and 11, and two short numbers can share a stand-in when the rule leaves no other in
+  range. Dates and times keep their order unless `dates.mode` is `digits`.
+- **Keep a date's relation to the real world.** A moved date is up to a year away from the
+  original, so "is this overdue?" against today, "last Friday" or an age written in words no
+  longer line up.
+- **Catch everything the defaults target.** The patterns trade recall for precision: a phone
+  number without a leading `+` is missed, a version string like `1.2.3.4` reads as an IPv4
+  address, and about one in ten 12 to 19 digit numbers passes the Luhn check by chance.
+- **Unlink calls under a fixed key.** With a fixed key, from the config or `JEV_SECRETS_KEY`, a
+  word gets the same stand-in in every call, so whoever receives the requests can tell that two
+  calls mention the same person, and anyone holding the key can test a guessed name against a
+  stand-in. Use a random key unless you need repeatable output.
+- **Keep an object keyed `"0"`, `"1"`, ... an object in PHP.** PHP decodes it, and `{}`, as a
+  list, so its keys are not rewritten and it is sent as a JSON array. The PHP SDK has the same
+  limit. For the same reason `secretKeys: {}` means no secret keys in PHP, and no change to the
+  default list in JavaScript and Python.
+- **Keep numeric-keyed requests identical in JavaScript.** JavaScript orders integer-like object
+  keys ascending, and stand-ins that collide are redrawn in walk order.
+
+## Does it change the answers?
+
+Over 67 labelled questions on 50 PII-heavy requests ([docs/evidence.md](docs/evidence.md)):
+
+- The default config got 64 of 67 right in both runs; tags got 58, the plain request 67. The
+  default moved answers least, by 0.05 on average against 0.17 for tags.
+- All 16 questions comparing dates or times kept their answers: every date and time moves by
+  the same offset. Names in Chinese, Arabic, Greek and Japanese script made no difference.
+- What replacing removes is lost: whether an email domain is a company's, or `+44` a UK number.
+- Comparing two plain numbers passed as values fails, since the digit rule keeps neither size
+  nor order. Leave out of `values` any number a question compares.
+
+## Documentation
+
+[docs/generation.md](docs/generation.md) and [docs/dates.md](docs/dates.md) are the byte-level
+contract the implementations share. `catalogue/fixtures.json` holds the cases each must
+reproduce: `php scripts/golden.php` records them from the PHP implementation, and the JavaScript
+and Python suites require the same output.