@pikku/kysely 0.13.22 → 0.13.24

package/knowledge/decisions/classification-driven-encryption-happens-in-a-plugin.md

@@ -0,0 +1,71 @@
+---
+type: decision
+title: Classification-driven encryption happens in a Kysely plugin
+description: The existing column classification manifest drives a transparent query plugin, rather than encrypting the whole database file or asking call sites to encrypt
+tags: [security, encryption, kysely, classification]
+---
+
+# Classification-driven encryption happens in a Kysely plugin
+
+Encrypting a local database at rest has two obvious shapes and both were
+rejected. Encrypting the **whole file** on shutdown and decrypting on startup is
+the simplest to describe and the easiest to lose data with: a crash, a battery
+death, or two processes racing the same file leaves a half-written database with
+no recovery story. Encrypting **every column** keeps the file safe but destroys
+the database — you cannot `WHERE`, `ORDER BY`, or join on ciphertext, so the app
+either reads whole tables into memory or stops using SQL.
+
+The middle answer already exists in the repo. `classification.gen.ts` is a
+per-column manifest of `private` / `secret` / `public` with a `form` of `plain`,
+`hashed`, `wrapped` or `sealed`. Only `wrapped` and `sealed` columns are
+encrypted; everything else stays queryable, which is most of the schema. The
+manifest is generated from the hand-authored `db/annotations.ts` by
+`pikku db migrate`, so the decision about a column is made once, in one place,
+and the query layer follows it.
+
+Enforcement belongs in a **Kysely plugin**, not at call sites. A call site that
+forgets to encrypt writes plaintext into a column the manifest says is secret,
+and nothing catches it — the row looks fine. A plugin sits in the query path
+every write goes through, so "encrypted" becomes a property of the column rather
+than a property of the developer's memory.
+
+Transparency turned out to be **asymmetric**, and the implementation says so.
+Reads are genuinely transparent: `transformResult` is async, so ciphertext is
+decrypted on the way out and application code is unchanged. Writes are not.
+`transformQuery` is synchronous — it returns a `RootOperationNode`, not a
+promise — and WebCrypto is async, so the plugin cannot encrypt on the way in.
+What it does instead is **refuse**: plaintext heading for a `wrapped` or
+`sealed` column throws, and values are produced by
+`ClassificationCrypto.encryptColumn()`. The security property is intact — a
+forgotten call site is a loud error rather than a silent plaintext row — but
+"transparent" describes only half of it.
+
+Two alternatives were rejected to get there. Node's synchronous `crypto` would
+fork crypto away from core and break every non-Node runtime. Proxying the query
+builder is the kind of workaround the repo bans. A **`Driver` wrapper**
+encrypting compiled-query parameters *would* deliver full write transparency and
+reuses the manifest walk and resolver verbatim, so it stays open — but it needs
+a parameter-index-to-column mapping recovered after compilation, and that
+mapping goes fragile exactly where inserts get interesting (multi-row,
+`onConflict().doUpdateSet()`). A mapping that is subtly wrong encrypts the wrong
+column, which is worse than the problem it solves. Not worth it until the write
+path is the thing actually hurting.
+
+That plugin lives here, in `@pikku/kysely`, and not in a SQLite-specific
+package. Nothing about wrapping a column value is dialect-specific, and the same
+plugin is wanted against Postgres — fabric has the same classified columns and
+the same reason to protect them. Putting it beside the dialect would mean
+writing it twice.
+
+The `hashed` form must never be encrypted, and the existing `ColumnForm` JSDoc
+already says why: a token hash *is* the lookup key, so encrypting it makes the
+lookup impossible.
+
+The cost was measured on an M-series Mac rather than estimated: envelope
+encryption runs about **38µs per value** to write and **33µs** to read, which
+over 500 rows batched with `Promise.all` is roughly **9ms**. Key derivation is
+the expensive step at **~48ms**, and it happens once, at unlock.
+
+**What this rules out:** SQLCipher or whole-file encryption, encrypting every
+column, and any design where an application function calls the crypto helpers
+directly.

package/knowledge/decisions/index.md

@@ -0,0 +1,15 @@
+---
+type: overview
+title: Decisions
+description: How encrypted columns are wrapped, and which key wraps them
+---
+
+# Decisions
+
+<!-- pikku:knowledge-index -->
+
+- [Classification-driven encryption happens in a Kysely plugin](classification-driven-encryption-happens-in-a-plugin.md) — The existing column classification manifest drives a transparent query plugin, rather than encrypting the whole database file or asking call sites to encrypt
+- [KEKs are scoped by purpose as well as version](keks-are-scoped-by-purpose-as-well-as-version.md) — Rows carry a keyId, and the plugin resolves KEKs through a seam rather than holding one; v1 ships a single key, but scoping never becomes a migration
+- [The plugin is given a resolver, never a key](the-plugin-is-given-a-resolver-never-a-key.md) — Assembly order for classified columns — a plain Kysely handle, then a plugin that asks a `KEKResolver` per operation rather than being handed a key it holds for the life of the process
+
+<!-- /pikku:knowledge-index -->

package/knowledge/decisions/keks-are-scoped-by-purpose-as-well-as-version.md

@@ -0,0 +1,45 @@
+---
+type: decision
+title: KEKs are scoped by purpose as well as version
+description: Rows carry a keyId, and the plugin resolves KEKs through a seam rather than holding one; v1 ships a single key, but scoping never becomes a migration
+tags: [security, encryption, kysely, keys]
+---
+
+# KEKs are scoped by purpose as well as version
+
+`KyselySecretService` already versions its key-encrypting key: each row stores a
+`keyVersion` beside its `ciphertext` and `wrappedDek`, per-version salts live in
+their own table, and `rotateKEK` walks rows through `envelopeRewrap`. Version is
+the *when* of a key. It is not the *what* — one KEK per version means every
+encrypted column in the database is protected by the same secret, so unlocking
+anything unlocks everything.
+
+That is acceptable for the first release and wrong as a permanent shape. A
+desktop app has one user and one passphrase; a multi-tenant deployment does not,
+and "the passphrase that opens my notes also opens my credentials" is a
+distinction people expect to be able to draw. So the design keeps scoping
+possible without paying for it now.
+
+Two things carry that. Encrypted rows store a **`keyId`** alongside `keyVersion`,
+so a row records which key protects it rather than inheriting the answer from
+context — retrofitting that column later would mean rewriting every encrypted
+row in every deployment. And the plugin does not hold a KEK; it calls a
+**resolver** that maps a `keyId` to one. In v1 the resolver returns the same key
+for every id. Scoping by tenant, or by purpose, replaces the resolver.
+
+Separate purposes derive from the same key material rather than requiring
+separate passphrases. `expandKeyMaterial(keyMaterial, info, salt)` is the HKDF
+helper already used for `REMOTE_SESSION_INFO`, and a distinct `info` string per
+purpose yields independent keys from one unlock — which is what makes multiple
+KEKs practical for a single-user app at all.
+
+The envelope survives all of this unchanged. Core's decision that
+[secrets use a per-secret DEK wrapped by a KEK](../../../../core/knowledge/decisions/security/core-secrets-use-a-per-secret-dek-wrapped-by-a-kek.md)
+already rules out collapsing it, and scoping is the argument *for* the envelope
+rather than against it: rescoping or rotating a key rewraps DEKs, touching a
+small constant per row, while a design that encrypted values directly under the
+KEK would have to decrypt and re-encrypt every value.
+
+**What this rules out:** a single process-wide KEK with no row-level record of
+which key applies, a separate passphrase per purpose, and deferring the `keyId`
+column to a later migration.

package/knowledge/decisions/the-plugin-is-given-a-resolver-never-a-key.md

@@ -0,0 +1,45 @@
+---
+type: decision
+title: The plugin is given a resolver, never a key
+description: Assembly order for classified columns — a plain Kysely handle, then a plugin that asks a `KEKResolver` per operation rather than being handed a key it holds for the life of the process
+tags: [security, encryption, kysely, classification]
+---
+
+# The plugin is given a resolver, never a key
+
+The pieces look circular the first time you assemble them: the plugin needs a
+key, and whatever produces that key may itself need to read from the database.
+It resolves because the plugin is never handed a key at all.
+
+```ts
+const db = new Kysely<DB>({ dialect })
+
+const crypto = new ClassificationCrypto({ resolveKEK })
+const classified = db.withPlugin(
+  createClassificationPlugin({ manifest, crypto })
+)
+```
+
+**The plugin is given a resolver, never a key.** `resolveKEK` is asked _per
+operation_. That is the whole reason a key can arrive after everything around it
+has been constructed — neither the services, the Kysely instance nor the plugin
+are rebuilt when it does — and it is why a key source that later becomes
+unavailable makes the next classified query fail rather than use a stale key.
+
+**A resolver that reads from the database reads through the plain handle**, not
+the classified one, and whatever it reads must not itself be classified. A key
+source that needed its own key to be read could never answer.
+
+**A resolver may derive a KEK any way it likes**, as long as the same `keyId`
+resolves to the same key: from a passphrase, from a KMS, from an environment
+secret. `ClassificationCrypto` only requires the key and the version it should
+stamp into the envelope. Deriving on every call is the mistake to avoid — key
+stretching is the expensive step, so a resolver caches per `keyId`.
+
+**What this rules out:** constructing the Kysely instance with the classification
+plugin already attached before a key source exists, giving a database-backed
+resolver the classified handle, and resolving a key once to cache it inside the
+plugin.
+
+Related: [[classification-driven-encryption-happens-in-a-plugin]],
+[[keks-are-scoped-by-purpose-as-well-as-version]].

package/knowledge/index.md

@@ -0,0 +1,18 @@
+---
+type: overview
+title: Knowledge
+description: Why encryption in this package is a query-layer concern rather than a call-site one
+---
+
+# Knowledge
+
+Encrypting data at rest inside a database that still has to answer SQL queries
+is a set of trade-offs, not a feature toggle. What gets encrypted, who does the
+encrypting, and which key protects which rows are three separate questions, and
+answering them independently is what keeps the result usable.
+
+These notes record how this package answers them.
+
+<!-- pikku:knowledge-index -->
+- [decisions](decisions/index.md) — a rule that was chosen, and what it rules out
+<!-- /pikku:knowledge-index -->

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pikku/kysely",
-  "version": "0.13.22",
+  "version": "0.13.24",
   "description": "Kysely-backed Pikku services — workflow state, secrets, sessions, scopes and audit",
   "author": "yasser.fadl@gmail.com",
   "license": "MIT",
@@ -21,7 +21,7 @@
     "prepublishOnly": "yarn build"
   },
   "peerDependencies": {
-    "@pikku/core": "^0.12.93"
+    "@pikku/core": "^0.12.104"
   },
   "dependencies": {
     "kysely": "^0.29.5"

package/src/classification-crypto.test.ts

@@ -0,0 +1,143 @@
+import { describe, test } from 'node:test'
+import assert from 'node:assert/strict'
+import { deriveKEK, generateKEKSalt } from '@pikku/core/crypto-utils'
+import {
+  ClassificationCrypto,
+  DEFAULT_KEY_ID,
+  isColumnEnvelope,
+  parseColumnEnvelope,
+  type KEKResolver,
+} from './classification-crypto.js'
+
+const passphrase = 'a-passphrase-long-enough-to-be-real-key-material'
+
+/**
+ * A resolver that derives a KEK per keyId and remembers it.
+ *
+ * Caching is the part under test as much as the derivation: `resolveKEK` is
+ * called per operation, and a resolver that re-derived every time would pay
+ * key stretching on every row.
+ */
+const testResolver = (): KEKResolver => {
+  const keks = new Map<string, Promise<CryptoKey>>()
+  return async (keyId) => {
+    let kek = keks.get(keyId)
+    if (!kek) {
+      kek = deriveKEK(passphrase, generateKEKSalt())
+      keks.set(keyId, kek)
+    }
+    return { kek: await kek, keyVersion: 1 }
+  }
+}
+
+const staticCrypto = async () =>
+  new ClassificationCrypto({ resolveKEK: testResolver() })
+
+describe('column envelope', () => {
+  test('round-trips a value through the default key', async () => {
+    const crypto = await staticCrypto()
+    const envelope = await crypto.encryptColumn(DEFAULT_KEY_ID, 'hunter2')
+    assert.equal(await crypto.decryptColumn(envelope), 'hunter2')
+  })
+
+  test('the stored envelope names the key that protects it', async () => {
+    const crypto = await staticCrypto()
+    const envelope = await crypto.encryptColumn('credentials', 'shh')
+    const parsed = parseColumnEnvelope(envelope)
+    assert.equal(parsed?.keyId, 'credentials')
+    assert.equal(parsed?.keyVersion, 1)
+  })
+
+  test('a value is not stored in the clear', async () => {
+    const crypto = await staticCrypto()
+    const envelope = await crypto.encryptColumn(DEFAULT_KEY_ID, 'hunter2')
+    assert.ok(!envelope.includes('hunter2'))
+  })
+
+  test('every value gets its own DEK', async () => {
+    const crypto = await staticCrypto()
+    const a = parseColumnEnvelope(
+      await crypto.encryptColumn(DEFAULT_KEY_ID, 'same')
+    )
+    const b = parseColumnEnvelope(
+      await crypto.encryptColumn(DEFAULT_KEY_ID, 'same')
+    )
+    assert.notEqual(a?.wrappedDek, b?.wrappedDek)
+    assert.notEqual(a?.ciphertext, b?.ciphertext)
+  })
+
+  test('round-trips non-string values', async () => {
+    const crypto = await staticCrypto()
+    const value = { card: '4242', meta: [1, 2, 3] }
+    const envelope = await crypto.encryptColumn(DEFAULT_KEY_ID, value)
+    assert.deepEqual(await crypto.decryptColumn(envelope), value)
+  })
+
+  test('plaintext is not mistaken for an envelope', () => {
+    assert.equal(isColumnEnvelope('hunter2'), false)
+    assert.equal(isColumnEnvelope(''), false)
+    assert.equal(parseColumnEnvelope('hunter2'), null)
+  })
+
+  test('a keyId containing the separator survives the round trip', async () => {
+    const crypto = await staticCrypto()
+    const envelope = await crypto.encryptColumn('tenant.42', 'x')
+    assert.equal(parseColumnEnvelope(envelope)?.keyId, 'tenant.42')
+    assert.equal(await crypto.decryptColumn(envelope), 'x')
+  })
+})
+
+describe('KEK scoping', () => {
+  test('different keyIds derive different KEKs from one passphrase', async () => {
+    const resolve = testResolver()
+    const a = await resolve(DEFAULT_KEY_ID)
+    const b = await resolve('credentials')
+    assert.notEqual(a.kek, b.kek)
+  })
+
+  test('a value written under one keyId does not open under another', async () => {
+    const crypto = await staticCrypto()
+    const envelope = await crypto.encryptColumn('credentials', 'shh')
+    const forged = envelope.replace(
+      /^(pikku1\.)[^.]+/,
+      `$1${Buffer.from('notes').toString('base64url')}`
+    )
+    await assert.rejects(() => crypto.decryptColumn(forged))
+  })
+
+  test('the same keyId resolves to the same KEK, so derivation is paid once', async () => {
+    const resolve = testResolver()
+    assert.equal(
+      (await resolve(DEFAULT_KEY_ID)).kek,
+      (await resolve(DEFAULT_KEY_ID)).kek
+    )
+  })
+
+  test('rewrapping moves a value to a new key without touching the ciphertext', async () => {
+    const crypto = await staticCrypto()
+    const envelope = await crypto.encryptColumn(DEFAULT_KEY_ID, 'portable')
+    const before = parseColumnEnvelope(envelope)
+
+    const rewrapped = await crypto.rewrapColumn(envelope, 'credentials')
+    const after = parseColumnEnvelope(rewrapped)
+
+    assert.equal(after?.keyId, 'credentials')
+    assert.equal(
+      after?.ciphertext,
+      before?.ciphertext,
+      'rewrap must re-wrap the DEK, not re-encrypt the value'
+    )
+    assert.notEqual(after?.wrappedDek, before?.wrappedDek)
+    assert.equal(await crypto.decryptColumn(rewrapped), 'portable')
+  })
+
+  test('a resolver may hand back a KEK it derived any way it likes', async () => {
+    const kek = await deriveKEK(passphrase, generateKEKSalt())
+    const crypto = new ClassificationCrypto({
+      resolveKEK: async () => ({ kek, keyVersion: 7 }),
+    })
+    const envelope = await crypto.encryptColumn('anything', 'v')
+    assert.equal(parseColumnEnvelope(envelope)?.keyVersion, 7)
+    assert.equal(await crypto.decryptColumn(envelope), 'v')
+  })
+})

package/src/classification-crypto.ts

@@ -0,0 +1,163 @@
+import {
+  envelopeDecrypt,
+  envelopeEncrypt,
+  envelopeRewrap,
+} from '@pikku/core/crypto-utils'
+import { DEFAULT_KEY_ID } from '@pikku/core/classification'
+import type { WrappedValue } from '@pikku/core/classification'
+
+/**
+ * The key a column is protected by when its classification names none. A
+ * deployment that never scopes its keys stores this id in every row and still
+ * gets the scoping seam for free.
+ *
+ * Re-exported from core rather than declared again: the same constant decides
+ * which record `initialize` mints and which one a stored envelope names, and
+ * two copies of it would only ever be found to disagree by a row that could
+ * not be opened.
+ */
+export { DEFAULT_KEY_ID }
+
+const ENVELOPE_PREFIX = 'pikku1'
+
+/** The parts of a stored envelope, as they came off the column. */
+export interface ColumnEnvelope {
+  keyId: string
+  keyVersion: number
+  wrappedDek: string
+  ciphertext: string
+}
+
+/**
+ * A KEK for one `keyId`, plus the version it was derived at.
+ *
+ * The version is the resolver's to report rather than the caller's to ask for:
+ * on a write it stamps the row, and on a read the row already carries the
+ * version the value was written under.
+ */
+export interface ResolvedKEK {
+  kek: CryptoKey
+  keyVersion: number
+}
+
+/**
+ * Maps a `keyId` — and, when reading, the version the row was written under —
+ * to the KEK that opens it.
+ *
+ * This is the seam the whole design turns on. An unlock gate, a per-tenant
+ * lookup or a KMS call all replace this one function without reshaping anything
+ * that calls it.
+ */
+export type KEKResolver = (
+  keyId: string,
+  keyVersion?: number
+) => Promise<ResolvedKEK>
+
+const encodePart = (value: string): string =>
+  Buffer.from(value, 'utf8').toString('base64url')
+
+const decodePart = (value: string): string =>
+  Buffer.from(value, 'base64url').toString('utf8')
+
+const serializeEnvelope = (envelope: ColumnEnvelope): WrappedValue =>
+  [
+    ENVELOPE_PREFIX,
+    encodePart(envelope.keyId),
+    String(envelope.keyVersion),
+    envelope.wrappedDek,
+    envelope.ciphertext,
+  ].join('.') as WrappedValue
+
+/**
+ * Read a stored envelope, or null when the value is not one.
+ *
+ * Every part is base64url except the version, so `.` cannot occur inside a part
+ * and a fixed split is unambiguous — a `keyId` may contain dots.
+ */
+export const parseColumnEnvelope = (value: unknown): ColumnEnvelope | null => {
+  if (typeof value !== 'string') return null
+  const parts = value.split('.')
+  if (parts.length !== 5 || parts[0] !== ENVELOPE_PREFIX) return null
+
+  const [, encodedKeyId, rawVersion, wrappedDek, ciphertext] = parts
+  const keyVersion = Number(rawVersion)
+  if (!Number.isInteger(keyVersion) || !wrappedDek || !ciphertext) return null
+
+  try {
+    return {
+      keyId: decodePart(encodedKeyId!),
+      keyVersion,
+      wrappedDek,
+      ciphertext,
+    }
+  } catch {
+    return null
+  }
+}
+
+export const isColumnEnvelope = (value: unknown): boolean =>
+  parseColumnEnvelope(value) !== null
+
+export interface ClassificationCryptoOptions {
+  resolveKEK: KEKResolver
+}
+
+/**
+ * Envelope encryption for a single classified column.
+ *
+ * Unlike `KyselySecretService`, which owns its table and can afford a column
+ * per envelope part, a classified column is one column in someone else's table.
+ * So the envelope is self-describing: the `keyId` and `keyVersion` travel in
+ * the value, which is what lets a row record the key that protects it without
+ * a schema change to every table that has a secret in it.
+ */
+export class ClassificationCrypto {
+  private readonly resolveKEK: KEKResolver
+
+  constructor(options: ClassificationCryptoOptions) {
+    this.resolveKEK = options.resolveKEK
+  }
+
+  async encryptColumn(keyId: string, value: unknown): Promise<WrappedValue> {
+    const { kek, keyVersion } = await this.resolveKEK(keyId)
+    const { ciphertext, wrappedDEK } = await envelopeEncrypt(kek, value)
+    return serializeEnvelope({
+      keyId,
+      keyVersion,
+      wrappedDek: wrappedDEK,
+      ciphertext,
+    })
+  }
+
+  async decryptColumn<T = unknown>(value: unknown): Promise<T> {
+    const envelope = parseColumnEnvelope(value)
+    if (!envelope) {
+      throw new Error('Value is not a pikku column envelope')
+    }
+    const { kek } = await this.resolveKEK(envelope.keyId, envelope.keyVersion)
+    return envelopeDecrypt<T>(kek, envelope.ciphertext, envelope.wrappedDek)
+  }
+
+  /**
+   * Move a value to another key by re-wrapping its DEK. The ciphertext is
+   * untouched, so the cost is a constant per row rather than proportional to
+   * the value — the property that makes rescoping and rotation affordable.
+   */
+  async rewrapColumn(value: unknown, newKeyId: string): Promise<WrappedValue> {
+    const envelope = parseColumnEnvelope(value)
+    if (!envelope) {
+      throw new Error('Value is not a pikku column envelope')
+    }
+    const { kek: oldKEK } = await this.resolveKEK(
+      envelope.keyId,
+      envelope.keyVersion
+    )
+    const { kek: newKEK, keyVersion } = await this.resolveKEK(newKeyId)
+    return serializeEnvelope({
+      keyId: newKeyId,
+      keyVersion,
+      wrappedDek: await envelopeRewrap(oldKEK, newKEK, envelope.wrappedDek),
+      ciphertext: envelope.ciphertext,
+    })
+  }
+}