cvx-kit 0.1.13 → 0.1.14

package/docs/querying.md

@@ -0,0 +1,293 @@
+# `cvx-kit/querying` — typed, authorized stream pagination
+
+Declare relationships with `cvx-kit/relations`, configure one query engine, then
+paginate with a small declarative chain. The engine runs streams and relationship
+lookups in the caller's Convex query transaction. Filtering determines eligible
+parents before they fill a page. `include` and native `paginated(dto)` remain
+independent APIs.
+
+## Declare tables and indexes
+
+Keep shapes in `domain/<module>/schema.ts` and native indexes in its `table.ts`.
+These examples put tasks, labels, and their junction in one owning module.
+
+```ts
+// convex/domain/tasks/schema.ts
+import { z } from 'zod'
+import { zodTable } from 'cvx-kit/zod-table'
+
+export const tasks = zodTable(
+	'tasks',
+	(id) => ({
+		title: z.string(),
+		department: z.string(),
+		status: z.string(),
+		priority: z.number(),
+		reference: id('tasks').nullable().optional(),
+	}),
+	{ publicFields: ['title', 'department', 'status', 'priority'] },
+)
+
+export const labels = zodTable(
+	'labels',
+	() => ({
+		name: z.string(),
+	}),
+	{ publicFields: ['name'] },
+)
+
+export const edges = zodTable(
+	'edges',
+	(id) => ({
+		task: id('tasks'),
+		label: id('labels'),
+	}),
+	{ publicFields: [] },
+)
+```
+
+```ts
+// convex/domain/tasks/table.ts
+import { tasks, labels, edges } from './schema'
+
+export const tables = {
+	tasks: tasks.table.index('by_department_status', ['department', 'status']),
+	labels: labels.table,
+	edges: edges.table.index('by_task', ['task']),
+}
+```
+
+Assemble those table definitions in the application's root schema. Declare every
+index used by a relationship before compiling the relation registry.
+
+## Declare relationships once
+
+```ts
+// convex/relations.ts
+import { defineRelations } from 'cvx-kit/relations'
+import './domain/tasks/table'
+import { tasks, labels, edges } from './domain/tasks/schema'
+
+export const relations = defineRelations({ tasks, labels, edges }, (r) => ({
+	tasks: {
+		referenceTask: r.one.tasks({ from: r.tasks.reference, to: r.tasks._id }),
+		labels: r.many.labels({
+			from: r.tasks._id.through(r.edges.task),
+			to: r.labels._id.through(r.edges.label),
+		}),
+	},
+}))
+```
+
+A direct one edge uses `r.one.target({ from, to })`; a direct many edge uses
+`r.many.target({ from, to })`. For example, with a stored `labels.task` foreign
+key and a supporting `by_task` index on labels:
+
+```ts
+r.many.labels({ from: r.tasks._id, to: r.labels.task })
+```
+
+Relationship names distinguish multiple references to the same table. Self
+references and junctions are supported. Tables must be object-shaped `zodTable`
+or `tenantTable` boundaries; variant tables are outside this API. Declarations
+are validated against table ownership, field compatibility, and supporting
+indexes. Convex does not infer relationships; this registry supplies them.
+
+## Configure the engine and authentication
+
+```ts
+// convex/querying.ts
+import { createQueryEngine } from 'cvx-kit/querying'
+import schema from './schema'
+import { relations } from './relations'
+import { cursor } from './functions'
+
+export const querying = createQueryEngine({ schema, relations, cursor })
+```
+
+The application's `functions.ts` facade configures `createAuthFunctions`,
+including its membership resolution and effective read/tenancy policies. Call
+`querying(ctx)` with a query context supplied by that facade's `authQuery`.
+The engine consumes the private managed reader capability installed there and
+rechecks composed authorization for parents, junctions, targets, and continuation
+reads. The example declarations do not establish access policy on their own.
+See [auth.md](./auth.md) and [tenancy.md](./tenancy.md) for policy setup.
+
+Unknown contexts fail closed with `UNMANAGED_QUERY_CONTEXT`. Mutation and action
+contexts are unsupported. A host with another auth system can centrally configure
+`trustedAdapter`; it must attest the query reader, effective policy, and security
+scope. Do not expose adapter selection to endpoint arguments.
+
+`cursor` has `{ version, currentKeyId, keys }`. `keys` maps key IDs to server-only
+64-byte `Uint8Array` secrets; `currentKeyId` names the key used for new tokens.
+Load real secrets through server configuration, never from clients or a constant
+fixture key. Keep prior keys during rotation to preserve existing cursors.
+The default codec uses authenticated AES-SIV encryption; there is no plaintext
+fallback. Identical bound payloads reveal equality.
+
+## Define input and result contracts
+
+```ts
+// Inside convex/domain/tasks/contracts.ts
+const input = relations.tables.tasks.paginationOpts({
+	department: true,
+	status: true,
+	priority: true,
+	referenceTask: { status: true },
+	labels: { name: true },
+})
+const result = relations.tables.tasks.pageDto({
+	referenceTask: true,
+	labels: true,
+})
+```
+
+`paginationOpts(selection)` extends native Convex pagination options with optional
+`filters` and global `order: 'asc' | 'desc'`. The selection permits client filters;
+it does not change public output or grant access. The result selection must
+match `.with(...)`. `.pageDto({})` returns a native page of the table's public
+DTOs including branded `_id`. One relationships return a public DTO or `null`;
+many relationships return their own bounded native pages. Missing and unreadable
+targets have the same public result. Duplicate junction occurrences remain in
+output, even when target reads are memoized.
+
+For scalar-only input, use `tasks.paginationOpts({ status: true })`. Both forms
+are Zod objects and support `.extend(...)` for endpoint-specific arguments.
+Use the contracts through your domain Query registry and keep public auth
+registration in `api/<entity>.ts`; see [domain-contracts.md](./domain-contracts.md).
+
+## Query and filter
+
+```ts
+// Inside an authenticated domain Query handler
+const department = args.filters?.department
+const indexCondition = typeof department === 'string' ? department : undefined
+const page = await querying(ctx)
+	.tasks.where({ status: 'open' })
+	.when(indexCondition, 'by_department_status')
+	.with({ referenceTask: true, labels: true })
+	.paginate(args)
+```
+
+`.where(spec)` adds server-owned constraints, combined with client filters by
+AND. Use it for mandatory business constraints; effective auth policy still
+controls access. `.with(selection)` chooses related output. A relation can be
+used as an eligibility filter without also including it in output. A parent
+relationship predicate does not filter the nested output page: it determines
+whether the parent is eligible. Child-page filters are a separate query input.
+
+Example client args for the contract above:
+
+```ts
+{
+  numItems: 25,
+  cursor: null,
+  order: 'desc',
+  filters: {
+    department: 'engineering',
+    priority: { gte: 2, lt: 5 },
+    labels: { some: { name: 'urgent' } },
+    not: { status: 'blocked' },
+  },
+}
+```
+
+Scalar predicates accept direct equality or operator objects with `eq`, `in`,
+and, for supported range types, `gt`, `gte`, `lt`, and `lte`. Optional fields
+also support `exists`. `and` and `or` contain nonempty arrays of predicates;
+`not` contains one predicate. Use `fields: { ... }` to address scalar fields
+whose names collide with grammar keys such as `and`, `or`, `not`, or `fields`.
+One relationships accept a nested predicate or `null`; many relationships use
+`some` or `none` over authorized targets. Unknown fields and incompatible scalar
+values/operators fail validation. Filter depth is at most four, the whole tree
+has at most 256 nodes, and an `in` list contains at most 100 values.
+
+## Conditional indexes and ordering
+
+`.when(value, 'index_name')` selects the first present condition. `false`, `0`,
+and `''` are present; `null` and `undefined` are absent. The condition chooses an
+index but does not bind its fields: validated filters and `.where(...)` supply
+the bindings. An incompatible explicit index fails instead of silently scanning.
+Relationship traversal retains its mandatory foreign-key prefix.
+
+Without an explicit choice, the planner selects the longest usable equality
+prefix, with stable index-name ordering for ties and creation-time fallback.
+OR branches contribute only guaranteed common bindings. `order: 'desc'` reverses
+the selected native index order; it does not sort by an arbitrary field.
+
+## Continue relationship pages
+
+Return many relationships as pages, then use a separate authenticated endpoint
+for their next page. Validate `parent` with `tasks.schema.shape._id` and use the
+related table's pagination contract:
+
+```ts
+const input = relations.tables.labels
+	.paginationOpts({ name: true })
+	.extend({ parent: tasks.schema.shape._id })
+const result = relations.tables.labels.pageDto({})
+
+// Inside the child Query handler
+const { parent, ...options } = args
+return querying(ctx).tasks.relation(parent, 'labels').paginate(options)
+```
+
+The client sends the parent ID and the nested page's `continueCursor`, preserving
+its original order and output selection. For the parent example above that means
+`order: 'desc'`, selection `{}`, and no child filters on resume; adding a `name`
+filter would change the binding and reject that cursor. A new child traversal
+with `cursor: null` can establish its own filters. Typed
+paths can traverse several declared edges to a many relationship. Each request
+rechecks the parent path and current authorization; a child token cannot be
+replayed under another parent.
+
+Root, child, split, and end cursors bind filters, output selection, index, order,
+ancestry, principal/tenant/security scope, and registry/policy/engine versions.
+Changing page size alone remains valid. Treat tokens as opaque; preserve native
+`isDone`, `continueCursor`, and optional page-status/split metadata. Permission
+changes are rechecked on resume. Native `endCursor` supports bounded partitions.
+
+## Central limits and operational behavior
+
+`createQueryEngine` also accepts `maxPageSize`, `budgets`, `policyVersion`,
+`engineVersion`, and `trustedAdapter`. These belong in central configuration,
+not per-endpoint filter/project callbacks. Page sizes are positive integers up
+to 100; `maxPageSize` can lower that ceiling. Shared request budgets cover read
+operations, index ranges, raw candidates, document bytes, fan-out, traversal
+depth, and memo entries, including managed policy reads. Native request read
+limits may lower ceilings. Output relationship depth is at most four.
+
+If eligibility cannot be established within the budget, execution throws
+`MANAGED_READ_BUDGET`; it does not return a misleading exhausted page or conclude
+that a negative relationship predicate matched. Policy errors propagate rather
+than turning into eligible rows. Memoization is request-local and preserves
+junction multiplicity. Relationships can require many database reads; one public
+query does not imply one physical read. Document accounting is an application
+bound, not a measurement of Convex storage/index/prefetch I/O.
+
+Arbitrary async predicates, custom projections/sorts, mutation/action execution,
+search/vector optimization, and global caching are outside this API. Index
+selection reduces candidates where possible; filtering every supported scalar
+field does not guarantee an efficient index for every filter combination.
+
+## Migration and verification
+
+Use `toPublicDto(row, true)` with `dto(true)` instead of manual ID spreads.
+Replace copied pagination schemas with table/registry helpers. Move eligibility
+predicates before bounded page filling, and replace unbounded relationship
+collection with declared many pages and child continuation. See the diagnostics
+and examples in [oxlint.md](./oxlint.md).
+
+The [packed example](../../test/fixture-packed-querying/convex/) contains complete
+schema, relation, engine, contract, Query-registry, auth-adapter, and child-endpoint
+wiring. `bun run test:package:querying` exercises strict installed and minimum-peer
+consumers, all consumer rules, and multi-entity/filter/order continuation tests.
+The local native smoke runner exercises registered Convex validators and cursors;
+these checks do not establish hosted production performance.
+
+Run `bunx vp test run test/querying-relations-stress.test.ts --silent=false` to
+report per-page p10, p50, and p99 latency for each filter/order/page-size case,
+plus an aggregate for duplicate child continuations. Reports include sample
+counts and use nearest-rank percentiles. A small sample's p99 is its maximum;
+the measurements include `convex-test` and instrumentation overhead. Read
+counters and ordered-result assertions remain separate from latency measurements.

package/docs/zod-table.md

@@ -49,19 +49,21 @@ unregistered foreign ID and does not throw; peer installation prevents the
 supported package graph from creating that ambiguity. Verify critical ID
 validators retain `kind === 'id'` and the expected `tableName` after installation.
 
-| Property                                     | What it is                                           | Where you use it               |
-| -------------------------------------------- | ---------------------------------------------------- | ------------------------------ |
-| `table`                                      | `defineTable(...)` from the storage shape            | `convex/schema.ts`             |
-| `storage`                                    | zod object of the raw row (no `_id`/`_creationTime`) | validating rows                |
-| `schema`                                     | `storage` + `_id` + `_creationTime`                  | full-document validation       |
-| `insertSchema`                               | storage minus timestamps minus `serverFields`        | mutation args                  |
-| `updateSchema`                               | `insertSchema.partial()`                             | patch mutation args            |
-| `commandInput`                               | strict pick of `commandFields`                       | command protocol inputs        |
-| `publicDto`                                  | strict pick of `publicFields`                        | query return validators        |
-| `toPublicDto(row)`                           | projects **and re-parses** a row into the DTO        | query handlers                 |
-| `insert(omit?)` / `update(omit?)`            | schema with per-callsite field omission              | specialized mutations          |
-| `tools.insert` / `tools.update` / `tools.id` | ready-made LLM tool input schemas                    | agent tool definitions         |
-| `tableName`                                  | the literal table name                               | indexes, triggers registration |
+| Property                                     | What it is                                             | Where you use it               |
+| -------------------------------------------- | ------------------------------------------------------ | ------------------------------ |
+| `table`                                      | `defineTable(...)` from the storage shape              | `convex/schema.ts`             |
+| `storage`                                    | zod object of the raw row (no `_id`/`_creationTime`)   | validating rows                |
+| `schema`                                     | `storage` + `_id` + `_creationTime`                    | full-document validation       |
+| `insertSchema`                               | storage minus timestamps minus `serverFields`          | mutation args                  |
+| `updateSchema`                               | `insertSchema.partial()`                               | patch mutation args            |
+| `commandInput`                               | strict pick of `commandFields`                         | command protocol inputs        |
+| `publicDto`                                  | strict pick of `publicFields`                          | query return validators        |
+| `dto(options?)`                              | public return schema with optional ID and extensions   | query return validators        |
+| `toPublicDto(row)`                           | projects **and re-parses** a row into the DTO          | query handlers                 |
+| `paginationOpts(selection)`                  | native pagination options, selected filters, and order | query input validators         |
+| `insert(omit?)` / `update(omit?)`            | schema with per-callsite field omission                | specialized mutations          |
+| `tools.insert` / `tools.update` / `tools.id` | ready-made LLM tool input schemas                      | agent tool definitions         |
+| `tableName`                                  | the literal table name                                 | indexes, triggers registration |
 
 ## Redaction is runtime, not just types
 
@@ -82,6 +84,84 @@ export const get = authQuery({
 })
 ```
 
+## Public DTO parameters
+
+`publicDto` is the strict Zod schema selected by `publicFields`; it takes no
+parameters. Use `dto(...)` to derive a return schema and `toPublicDto(row, ...)`
+to project a row with the same options. These methods are also available on
+`tenantTable` boundaries.
+
+```ts
+documents.toPublicDto(row) // publicFields only
+documents.toPublicDto(row, false) // same default projection
+documents.toPublicDto(row, true) // publicFields plus _id
+documents.toPublicDto(row, { _id: true }) // explicit equivalent
+
+const withoutId = documents.dto() // same boundary as publicDto
+const withId = documents.dto(true) // branded Id<'documents'>
+const explicit = documents.dto({ _id: true })
+```
+
+Including `_id` requires a row containing an ID and validates it through the
+table's ID schema. It does not expose `_creationTime`, tenant metadata, or other
+storage fields. Do not spread an ID onto a parsed DTO manually; use the matching
+`dto(true)` return validator.
+
+Computed fields need both a schema and explicitly supplied values:
+
+```ts
+const extend = { url: z.url() }
+const returnSchema = documents.dto({ _id: true, extend })
+
+const output = documents.toPublicDto(row, {
+	_id: true,
+	extend,
+	values: { url: `https://example.test/documents/${row._id}` },
+})
+returnSchema.parse(output)
+```
+
+`extend` is a map of field names to Zod validators, not an `extendFields` wrapper.
+`values` belongs only to the projector; `dto(...)` describes the result. Extension
+values are parsed as a strict object: missing required fields, unknown values,
+and invalid values fail. Values are never copied implicitly from storage.
+
+Extensions cannot replace public fields or use `_id`, `_creationTime`, `tenant`,
+`createdAt`, `updatedAt`, `archivedAt`, or `__proto__`. Their schemas must validate
+without changing values: transforms, pipes, coercion, preprocess, defaults,
+catches, and overwrites are rejected, including inside nested schemas.
+Refinements must not mutate their input. Existing public-field schemas retain
+their existing parsing behavior.
+
+## Pagination options
+
+Select filterable scalar fields independently of the public DTO mask:
+
+```ts
+const input = documents.paginationOpts({ title: true, ownerId: true })
+const args = input.parse({
+	numItems: 25,
+	cursor: null,
+	order: 'desc',
+	filters: { title: 'Proposal' },
+})
+```
+
+The helper extends Convex's native pagination options with optional `filters`
+and `order: 'asc' | 'desc'`. It returns a Zod object that can be used directly as
+query args or extended with endpoint-specific fields. `numItems` must be a
+positive integer at most 100; engine configuration may lower that ceiling.
+Unknown filter fields fail. Stored scalar fields, including explicitly selected
+server-owned fields and IDs, can be filterable even when absent from the DTO.
+Arrays, objects, and other unsupported non-scalar fields cannot be selected.
+This schema creates no authorization policy by itself.
+
+For relationship filters and paginated public output, use
+`relations.tables.documents.paginationOpts(selection)` and `.pageDto(selection)`.
+See [querying.md](./querying.md) for declarations, conditional indexes, filters,
+authorization, and child-page continuation. Existing native pagination can keep
+using `paginated(dto)`; the querying engine is independent of `include`.
+
 ## Opinionated timestamps
 
 Every `zodTable` bakes in three optional, **server-owned** fields:

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "cvx-kit",
-	"version": "0.1.13",
+	"version": "0.1.14",
 	"description": "Reusable Convex application kit: foundation kernels, approvals component, zod table boundaries, WorkOS-backed auth function constructors, and audited command protocol.",
 	"keywords": [
 		"component",
@@ -30,6 +30,14 @@
 			"types": "./dist/zod-table.d.mts",
 			"import": "./dist/zod-table.mjs"
 		},
+		"./relations": {
+			"types": "./dist/relations-public.d.mts",
+			"import": "./dist/relations-public.mjs"
+		},
+		"./querying": {
+			"types": "./dist/querying.d.mts",
+			"import": "./dist/querying.mjs"
+		},
 		"./effect": {
 			"types": "./dist/effect.d.mts",
 			"import": "./dist/effect.mjs"
@@ -134,6 +142,7 @@
 		"test:package:oxlint": "bun run build && node scripts/test-packed-oxlint.mjs",
 		"test:package:effect": "bun run build && node scripts/test-packed-effect.mjs",
 		"test:package:auth": "bun run build && node scripts/test-packed-auth.mjs",
+		"test:package:querying": "bun run build && node scripts/test-packed-querying.mjs",
 		"test:package:smoke:effect": "bun run build && node scripts/smoke-packed-effect.mjs",
 		"test:package:smoke:bun": "bun scripts/smoke-packed-components.mjs bun",
 		"test:package:smoke:npm": "node scripts/smoke-packed-components.mjs npm",
@@ -145,6 +154,7 @@
 	},
 	"dependencies": {
 		"@convex-dev/workflow": "0.4.10",
+		"@noble/ciphers": "2.4.0",
 		"@oxlint/plugins": "1.87.0",
 		"@standard-schema/spec": "^1.1.0",
 		"convex-audit-log": "^0.2.0",

package/scripts/owned-process.mjs

@@ -0,0 +1,140 @@
+import { spawn } from 'node:child_process'
+import { setTimeout as delay } from 'node:timers/promises'
+
+export function processGroupAlive(pid) {
+	try {
+		process.kill(-pid, 0)
+		return true
+	} catch (error) {
+		if (error.code === 'ESRCH') return false
+		throw error
+	}
+}
+const owned = new Set()
+export async function cleanupOwnedProcess(pid, graceMs = 1000) {
+	if (!owned.has(pid)) throw new Error('Process group is not owned by this runner')
+	const signals = [],
+		errors = []
+	for (const signal of ['SIGINT', 'SIGTERM', 'SIGKILL']) {
+		if (!processGroupAlive(pid)) break
+		try {
+			process.kill(-pid, signal)
+			signals.push(signal)
+		} catch (error) {
+			if (error.code !== 'ESRCH') errors.push({ signal, code: error.code })
+		}
+		const deadline = performance.now() + graceMs
+		while (processGroupAlive(pid) && performance.now() < deadline) await delay(20)
+	}
+	return { pid, groupId: pid, signals, errors, alive: processGroupAlive(pid) }
+}
+
+export async function runOwnedProcess({
+	command,
+	args,
+	cwd,
+	env,
+	timeoutMs = 180000,
+	graceMs = 1000,
+	onSpawn = () => {},
+}) {
+	const started = performance.now()
+	const child = spawn(command, args, { cwd, env, detached: true, stdio: ['pipe', 'pipe', 'pipe'] })
+	let stdout = '',
+		stderr = '',
+		timedOut = false,
+		spawnError,
+		timeoutCleanup
+	child.stdout.on('data', (bytes) => {
+		stdout += bytes
+	})
+	child.stderr.on('data', (bytes) => {
+		stderr += bytes
+	})
+	const closed = new Promise((resolve) => {
+		child.once('error', (error) => {
+			spawnError = { code: error.code, message: error.message }
+			resolve({ code: null, signal: null })
+		})
+		child.once('close', (code, signal) => resolve({ code, signal }))
+	})
+	if (child.pid) {
+		owned.add(child.pid)
+		onSpawn(child.pid)
+	}
+	child.stdin.end()
+	let finishTimeout
+	const timeoutFinished = new Promise((resolve) => {
+		finishTimeout = resolve
+	})
+	const timer = setTimeout(() => {
+		timedOut = true
+		timeoutCleanup = child.pid
+			? cleanupOwnedProcess(child.pid, graceMs).catch((error) => ({
+					pid: child.pid,
+					alive: true,
+					error: { code: error.code, message: error.message },
+				}))
+			: Promise.resolve(null)
+		timeoutCleanup.then(() => finishTimeout({ code: child.exitCode, signal: child.signalCode }))
+	}, timeoutMs)
+	let exit, cleanup
+	try {
+		exit = await Promise.race([closed, timeoutFinished])
+		if (timeoutCleanup) await timeoutCleanup
+	} finally {
+		clearTimeout(timer)
+		if (child.pid) {
+			try {
+				cleanup = await cleanupOwnedProcess(child.pid, graceMs)
+			} catch (error) {
+				cleanup = {
+					pid: child.pid,
+					alive: true,
+					error: { code: error.code, message: error.message },
+				}
+			}
+		}
+	}
+	if (timedOut) {
+		child.stdout.destroy()
+		child.stderr.destroy()
+		child.unref()
+	}
+	return {
+		...exit,
+		spawnError,
+		stdout,
+		stderr,
+		timedOut,
+		pid: child.pid,
+		milliseconds: performance.now() - started,
+		timeoutCleanup: await timeoutCleanup,
+		cleanup,
+	}
+}
+
+export async function collectOwnedCleanup({ pids, ports, checkPort, failure, graceMs = 1000 }) {
+	const groups = []
+	for (const pid of pids) {
+		try {
+			groups.push(await cleanupOwnedProcess(pid, graceMs))
+		} catch (error) {
+			groups.push({ pid, alive: true, error: { code: error.code, message: error.message } })
+		}
+	}
+	const portResults = []
+	for (const port of ports) {
+		try {
+			portResults.push({ port, closed: (await checkPort(port)) === port })
+		} catch (error) {
+			portResults.push({ port, closed: false, error: { code: error.code, message: error.message } })
+		}
+	}
+	return {
+		pids: groups.map(({ pid, alive }) => ({ pid, alive })),
+		failure: failure ? { name: failure.name, message: failure.message } : null,
+		groups,
+		ports: portResults,
+	}
+}