hyperscale-crypto 0.1.0__tar.gz

hyperscale_crypto-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hyperscale Consulting
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hyperscale_crypto-0.1.0/PKG-INFO

@@ -0,0 +1,390 @@
+Metadata-Version: 2.4
+Name: hyperscale-crypto
+Version: 0.1.0
+Summary: Application-tier encrypted Django fields backed by Tink keysets and AWS KMS
+Keywords: django,encryption,tink,kms,fields
+Author: Andy Caine
+Author-email: Andy Caine <andy@hyperscale.consulting>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 5.2
+Classifier: Framework :: Django :: 6.1
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Security :: Cryptography
+Classifier: Typing :: Typed
+Requires-Dist: django>=5.2
+Requires-Dist: tink[awskms]>=1.16.0
+Requires-Python: >=3.14
+Project-URL: Homepage, https://github.com/hyperscale-consulting/hyperscale-crypto
+Project-URL: Repository, https://github.com/hyperscale-consulting/hyperscale-crypto
+Project-URL: Changelog, https://github.com/hyperscale-consulting/hyperscale-crypto/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/hyperscale-consulting/hyperscale-crypto/issues
+Description-Content-Type: text/markdown
+
+# hyperscale-crypto
+
+Application-tier encrypted Django fields backed by Tink keysets and AWS KMS.
+
+Install the `hyperscale-crypto` package; import it as `hyperscale.crypto`.
+
+Values are encrypted in Python before they reach the database. Two Tink
+keysets hold the data keys: `aead` (AES256-GCM, randomised) and `daead`
+(AES256-SIV, deterministic). Both are stored in your repository, wrapped by a
+KMS key, and unwrapped in memory at startup.
+
+## Install
+
+```bash
+uv add hyperscale-crypto
+```
+
+```python
+INSTALLED_APPS = [
+    # ...
+    "hyperscale.crypto",
+]
+```
+
+Then run `python manage.py migrate`. The app ships migrations for three
+tables: `KeysetState` (the active deterministic key), `KeysetCanary` (the
+startup check) and `KeyRotationEvent` (the rotation log).
+
+## Settings
+
+```python
+import os
+from pathlib import Path
+
+BASE_DIR = Path(__file__).resolve().parent.parent
+
+HYPERSCALE_CRYPTO = {
+    "mode": os.environ.get("CRYPTO_MODE", "kms"),  # "kms" or "cleartext"
+    "kms_key_uri": os.environ.get("CRYPTO_KMS_KEY_URI", ""),
+    "keyset_dir": BASE_DIR / "keysets",  # required
+    "reunwrap_seconds": 900,
+    "revoke_on_reunwrap_failure": False,
+    "retention_days": 35,
+    "git_sha": os.environ.get("GIT_SHA", ""),
+}
+```
+
+| Key | Default | Meaning |
+|---|---|---|
+| `mode` | `"kms"` | `"kms"` reads KMS-wrapped keysets; `"cleartext"` reads the dev files |
+| `kms_key_uri` | `""` | `aws-kms://arn:aws:kms:<region>:<account>:key/<id>`; required in `kms` mode |
+| `keyset_dir` | none (required) | directory holding the keyset files |
+| `reunwrap_seconds` | `900` | how often a running process re-reads and re-unwraps the keysets |
+| `revoke_on_reunwrap_failure` | `False` | drop cached keys when a re-unwrap fails (see [Startup](#startup)) |
+| `retention_days` | `35` | minimum days between retiring a key and destroying it |
+| `git_sha` | `$GIT_SHA` or `""` | recorded on every `KeyRotationEvent` |
+
+`reunwrap_seconds` and `retention_days` must be positive integers and
+`revoke_on_reunwrap_failure` a real `bool` (not the string `"false"`). In
+`kms` mode, `kms_key_uri` must start with `aws-kms://` (or `fake-kms://` for
+tests). `mode="cleartext"` and `fake-kms://` key URIs are only accepted when
+the `ENV` environment variable is `development` or `test`. Any
+misconfiguration (missing or unknown keys, a bad value, cleartext outside
+those environments) raises `ImproperlyConfigured` during Django setup, not on
+first use.
+
+## Keysets
+
+```bash
+python manage.py keyset_init              # both keysets; --keyset aead|daead|all
+```
+
+In `kms` mode this writes `<keyset_dir>/aead.json` and
+`<keyset_dir>/daead.json`, each wrapped by the KMS key. Commit them: the
+wrapped files are safe to store, and every deploy ships the keys it needs.
+Protect the keyset path with branch protection and signed commits, since
+whoever can change these files can change which keys the application trusts.
+
+In `cleartext` mode the files are `<keyset_dir>/dev/aead.json` and
+`<keyset_dir>/dev/daead.json`, unwrapped, for development and tests only.
+
+`keyset_init` refuses to overwrite a keyset that already exists.
+
+## Fields
+
+```python
+from django.db import models
+
+from hyperscale.crypto import fields as ef
+
+
+class Customer(models.Model):
+    owner_id = models.IntegerField()
+    notes = ef.EncryptedTextField(blank=True, default="")
+    token = ef.EncryptedCharField(
+        max_length=64, context=lambda obj: f"owner:{obj.owner_id}"
+    )
+    ni_number = ef.DeterministicEncryptedCharField(max_length=9, unique=True)
+```
+
+| Field | Underlying | Deterministic variant |
+|---|---|---|
+| `EncryptedCharField` | `CharField` | `DeterministicEncryptedCharField` |
+| `EncryptedTextField` | `TextField` | |
+| `EncryptedEmailField` | `EmailField` | `DeterministicEncryptedEmailField` |
+| `EncryptedDateField` | `DateField` | |
+| `EncryptedDecimalField` | `DecimalField` | |
+| `EncryptedIntegerField` | `IntegerField` | `DeterministicEncryptedIntegerField` |
+| `EncryptedBooleanField` | `BooleanField` | |
+
+- **Storage.** Every field is a `text` column holding `hc1:` followed by the
+  URL-safe base64 Tink ciphertext. `None` and `""` are stored as they are.
+  `max_length` and the other validators apply to the plaintext.
+- **Reading.** Values decrypt lazily on first attribute access.
+  `.values()` and `.values_list()` return the stored value wrapped in the
+  `hyperscale.crypto.fields.Ciphertext` marker (a `str` subclass), not the
+  plaintext. To copy a stored value unchanged, assign that `Ciphertext`
+  instance; a plain `str` is always encrypted.
+- **Lookups.** Randomised fields support `isnull` only. Deterministic fields
+  support `exact`, `in` and `isnull`; any other lookup or transform raises
+  `FieldError`. Ordering is unsupported: it would sort by ciphertext.
+  `db_index` is rejected on every encrypted field, and `unique=True` on a
+  randomised field (equal values encrypt differently, so the database cannot
+  enforce it); use the deterministic variant. `auto_now` and `auto_now_add`
+  are rejected on `EncryptedDateField`.
+- **Bulk writes.** `QuerySet.update()`, `bulk_update()` and raw saves (such
+  as `loaddata`) encrypt plaintext values too. A field with a `context`
+  cannot be written that way (there is no instance to compute the context
+  from) and raises `FieldError`; save the instances instead. Query
+  expressions are refused (`FieldError`) because their result would be
+  stored unencrypted or would corrupt the ciphertext; the exceptions are
+  `F()` of the same field, `Value(x, output_field=<the field>)` and the
+  `Case`/`When` tree `bulk_update()` builds from such values.
+- **`context`** (randomised fields only) is a callable from the model instance
+  to a string that is bound into the ciphertext. It must be a stable property
+  of the row, such as an owner or tenant id. If a context input must change,
+  read the field before changing it and save both together. Saving a row
+  whose context changed while the field was never read raises
+  `DecryptionError` (naming the model, field and pk) instead of storing a
+  value that would no longer decrypt. Context fields cannot be read or
+  written from a migration (historical models do not carry the callable);
+  that raises `FieldError`. Use a management command with the live model.
+- **Deterministic fields** require `unique=True` and reject `context`. Every
+  deterministic write (`save()` of a new or loaded row, `QuerySet.update()`,
+  `bulk_update()`) must run inside a transaction, on every backend; outside
+  one it raises `TransactionManagementError`. On PostgreSQL the write also
+  takes a share lock on the `KeysetState` row. Set `ATOMIC_REQUESTS = True`
+  on the database and wrap background writers in `transaction.atomic()`.
+
+## Rotation
+
+Randomised (`aead`) keys rotate online:
+
+```bash
+python manage.py keyset_rotate --keyset aead      # new primary key
+# commit aead.json and deploy everywhere
+python manage.py keyset_reencrypt                 # --batch-size 500 (>0); resumable
+python manage.py keyset_retire aead <old_key_id>  # refuses while rows use it
+# after retention_days:
+python manage.py keyset_destroy aead <old_key_id>
+```
+
+Deterministic (`daead`) keys switch in one step, because lookups must find
+every row under a single key:
+
+```bash
+python manage.py keyset_add_key --keyset daead    # enabled, not active
+# commit daead.json and deploy everywhere
+python manage.py keyset_rotate --keyset daead --to <new_key_id> [--yes] [--lock-timeout 30]
+python manage.py keyset_retire daead <old_key_id>
+# after retention_days:
+python manage.py keyset_destroy daead <old_key_id>
+```
+
+The daead switch prints the row counts, asks you to type `confirm` (unless
+`--yes`), then re-encrypts every deterministic value in one transaction. It
+locks the `KeysetState` row for that transaction (on PostgreSQL, the table in
+`EXCLUSIVE` mode first, so the switch queues fairly behind in-flight writers
+instead of starving), and every deterministic writer waits until it commits.
+Reads are not blocked. `--lock-timeout <seconds>` (PostgreSQL) gives up
+cleanly, changing nothing, if in-flight writers hold the lock longer than
+that. On large tables, run it in a maintenance window.
+
+`keyset_retire` refuses while any row is still encrypted under the key,
+counting the startup canary row as one: after an aead rotation run
+`keyset_reencrypt` (it moves the canary too) even if no model has data yet.
+
+`keyset_destroy` refuses until `retention_days` have passed since the key was
+retired. Set `retention_days` to cover your database backup window: a backup
+that still holds values under a destroyed key cannot be decrypted.
+
+Every command records a `KeyRotationEvent` row (`init`, `add_key`, `rotate`,
+`reencrypt`, `retire`, `destroy`) with the key id, the actor, details such as
+row counts, and `git_sha`. Pass `--actor <name>` to record who ran it; it
+defaults to the OS user.
+
+## Evidence
+
+```bash
+python manage.py keyset_status          # human-readable
+python manage.py keyset_status --json   # machine-readable
+```
+
+The JSON has these top-level keys:
+
+- `keysets`: per keyset, every key id with its `status` (`ENABLED`,
+  `DISABLED`, `DESTROYED`) and whether it is `primary` and (daead) `active`.
+- `rows_by_key`: per keyset, per field (`app.Model.field`), the number of
+  rows under each key id, read from ciphertext prefixes without decrypting.
+  The canary row is listed too, as
+  `hyperscale_crypto.KeysetCanary.aead_value` / `.daead_value`.
+- `last_events`: the time of the most recent event of each action, or `null`.
+- `canary_ok`, and `canary_error` when the canary check failed.
+- `state_matches_file`: whether the database's active daead key is an enabled
+  key in the shipped `daead.json`.
+
+## Startup
+
+When the app is ready, it runs a canary check: it decrypts a known value
+stored in `KeysetCanary` with both keysets (writing the value on first run)
+and raises `ImproperlyConfigured`, naming the keyset, if either keyset
+cannot be loaded (a missing file, or one that does not unwrap) or cannot
+decrypt it. This
+catches a wrong KMS key, a swapped keyset file or a database restored under
+different keys before any request is served. The check is skipped for
+`makemigrations`, `migrate`, `collectstatic`, `check`, `showmigrations` and
+`sqlmigrate`, under pytest, and before the canary table has been migrated.
+
+Startup neither arms the re-unwrap timer nor keeps its database
+connection: under a pre-fork server (gunicorn, uWSGI) it runs in the parent,
+and neither survives a fork. Instead, the first use of the keys in each
+process starts a daemon timer that re-reads and re-unwraps the keysets every
+`reunwrap_seconds`, so a revoked KMS grant takes effect without a restart. A
+forked worker resets the inherited timer state and starts its own. If a
+re-unwrap fails, the default is to keep the cached keys and log the error,
+favouring availability during a KMS outage. With
+`revoke_on_reunwrap_failure = True` the cached keys are dropped and every
+later use must unwrap through KMS again, so revoking KMS access stops
+decryption within one interval, at the cost of failing requests while KMS is
+unreachable.
+
+## Testing your app
+
+Set `ENV=test` and point the settings at throwaway keysets from
+`hyperscale.crypto.testing`:
+
+```python
+# conftest.py
+import os
+from collections.abc import Iterator
+from pathlib import Path
+
+import pytest
+
+from hyperscale.crypto import keysets, testing
+
+os.environ.setdefault("ENV", "test")
+
+
+@pytest.fixture(autouse=True)
+def crypto(tmp_path: Path, settings) -> Iterator[None]:
+    testing.write_dev_keysets(tmp_path)  # cleartext dev/aead.json, dev/daead.json
+    settings.HYPERSCALE_CRYPTO = {"mode": "cleartext", "keyset_dir": tmp_path}
+    keysets.reset()
+    yield
+    keysets.reset()
+```
+
+To exercise the KMS code path without AWS, use `uri = testing.fake_kms_uri()`
+and `testing.write_kms_keysets(tmp_path, uri)`, with
+`{"mode": "kms", "kms_key_uri": uri, "keyset_dir": tmp_path}`.
+`keysets.reset()` drops the cached keysets so each test sees its own.
+
+## Security notes
+
+- **Deterministic encryption leaks equality.** Equal plaintexts in the same
+  column give equal ciphertexts, so anyone with database access can see which
+  rows share a value and how often. On a unique column there is nothing to
+  count, which is why deterministic fields require `unique=True`. Use them
+  only where you must look rows up by value.
+- **Associated data.** Every ciphertext is bound to
+  `<app_label>.<model_name>.<field>` (plus `|<context>` when set). A value
+  copied to another column, model or context fails to decrypt instead of
+  being read in the wrong place.
+- **Only `Ciphertext` values pass through.** A value is stored without
+  encryption only when it is a `hyperscale.crypto.fields.Ciphertext`, the
+  marker that `.values()`, `.values_list()` and unread loaded fields carry.
+  Every plain `str`, including one that starts with `hc1:`, is plaintext and
+  is encrypted. A ciphertext copied from another row and submitted as text is
+  stored encrypted as that text; it never decrypts to the other row's value.
+- **The canary** stops a process from serving with keys that cannot read the
+  data (see [Startup](#startup)).
+
+## Limitations
+
+- **One database.** Keyset state, the canary and the rotation log live in the
+  `default` database, and rotation reads and writes models there; a `using`
+  argument or database router is not honoured.
+- **PostgreSQL or sqlite.** Other backends are untested. The share and table
+  locks that serialise deterministic writes against a daead switch exist
+  only on PostgreSQL; on sqlite the database's own write lock serialises
+  them.
+- **`dumpdata` emits plaintext.** Serialisation reads fields through the
+  model, so fixtures and dumps contain decrypted values. Treat them as
+  sensitive. `loaddata` re-encrypts on the way in, but cannot load a field
+  with a `context` (raw saves have no instance for it).
+- **`keyset_status` / `rotation.status()` write the canary** on first run if
+  it does not exist yet, like startup does.
+- **Where each command runs.** Every command records its `KeyRotationEvent`
+  in the database it is pointed at, so run them against the database whose
+  evidence you want. `keyset_retire` and `keyset_destroy` read row counts and
+  the retire date from the production database and edit the keyset files, so
+  they need both the production database and a checkout of the repository
+  (commit the changed file afterwards). `keyset_rotate` and `keyset_add_key`
+  edit the keyset files too; `keyset_rotate --keyset daead` also rewrites
+  production rows.
+
+## Optional: background tasks
+
+`hyperscale.crypto.tasks.reencrypt_aead_task` runs `keyset_reencrypt` as a
+background task. It uses Django's built-in tasks framework (`django.tasks`,
+Django 6.0 and later) and, on older Django, the
+[django-tasks](https://github.com/RealOrangeOne/django-tasks) backport if it
+is installed. With neither, the module defines nothing. Configure a task
+backend in `TASKS` as usual.
+
+```python
+from hyperscale.crypto.tasks import reencrypt_aead_task
+
+reencrypt_aead_task.enqueue(batch_size=500, actor="ops")
+```
+
+## Development
+
+Requires [uv](https://docs.astral.sh/uv/) and Python 3.14+.
+
+```bash
+uv sync                             # create the venv and install dependencies
+uv run pytest                       # run the tests
+uv run pre-commit run --all-files   # lint, format and lockfile checks
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the full set of checks.
+
+## Releasing
+
+Bump `version` in `pyproject.toml` and `__version__` in
+`src/hyperscale/crypto/__init__.py`, update `CHANGELOG.md`, then publish a
+GitHub release tagged `v<version>`. The `publish` workflow checks the tag
+matches the package version, builds, runs `twine check` and publishes to PyPI
+with trusted publishing.
+
+Before the first release, register the project on PyPI with this repository
+and `publish.yml` as a trusted publisher, and create a `pypi` environment in
+the repository settings.
+
+## Design
+
+Design spec: [`docs/superpowers/specs/2026-09-24-encryption-primitive-design.md`](docs/superpowers/specs/2026-09-24-encryption-primitive-design.md)
+
+## License
+
+MIT. See [LICENSE](LICENSE).