@classytic/repo-core 0.4.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +50 -0
- package/dist/repository/index.d.mts +3 -2
- package/dist/repository/index.mjs +2 -1
- package/dist/repository/purge.d.mts +57 -0
- package/dist/repository/purge.mjs +83 -0
- package/dist/repository/types.d.mts +205 -2
- package/dist/testing/conformance.mjs +144 -0
- package/dist/testing/types.d.mts +7 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,56 @@ All notable changes to `@classytic/repo-core` are documented here.
|
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
5
5
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## [0.5.0] - 2026-05-17
|
|
8
|
+
|
|
9
|
+
### Added — compliance-grade tenant cleanup primitive
|
|
10
|
+
|
|
11
|
+
Cross-kit foundation for "what happens to this data on org-delete?" —
|
|
12
|
+
GDPR right-to-be-forgotten, SOC 2 deletion timelines, HIPAA / PCI
|
|
13
|
+
retention rules. Every kit (mongokit, sqlitekit, future pgkit) gets
|
|
14
|
+
the same surface; arc's `cascadeDeleteForOrganization` runner composes
|
|
15
|
+
on top.
|
|
16
|
+
|
|
17
|
+
- **`StandardRepo.purgeByField?(field, value, strategy, options)`** — new optional method. Processes every row matching `field = value` under a declared strategy, chunked under the hood. Optional because not every store needs the surface; arc's cascade runner checks for the method at boot.
|
|
18
|
+
- **`TenantPurgeStrategy`** discriminated union — four variants:
|
|
19
|
+
- `{ type: 'hard' }` — permanent removal (GDPR right-to-be-forgotten).
|
|
20
|
+
- `{ type: 'soft', deletedField?, deletedAtField? }` — recoverable; pairs with TTL indexes for eventual hard-purge.
|
|
21
|
+
- `{ type: 'anonymize', fields }` — retain rows but overwrite declared fields (HIPAA / PCI / SOX-compatible).
|
|
22
|
+
- `{ type: 'skip', reason }` — explicit opt-out with **mandatory** `reason` (compliance forcing function — silent skips are leaks).
|
|
23
|
+
- **`TenantPurgeOptions`** — `batchSize`, `session`, `onProgress`, `signal`. Chunking is mandatory (10M-row tenants can't run as a single `deleteMany`); abort signal is checked between chunks (never mid-write); aborted runs return `ok: false` with cumulative `processed` count (at-least-once cleanup semantics).
|
|
24
|
+
- **`TenantPurgeResult`** + **`TenantPurgeProgress`** — typed result envelope + per-chunk progress event.
|
|
25
|
+
|
|
26
|
+
### Added — kit-agnostic orchestrator (`runChunkedPurge`)
|
|
27
|
+
|
|
28
|
+
The chunk-loop logic — abort handling, progress emission, error-wrapping into result envelope, natural-exit on non-full batch — is identical across kits. Extracting it here means a single bug fix lands for every kit, and the surface a new kit has to implement shrinks to ~80 lines.
|
|
29
|
+
|
|
30
|
+
- **`runChunkedPurge(strategy, options, port)`** — pure orchestrator (130 lines, no I/O).
|
|
31
|
+
- **`PurgePort`** interface — the driving port. Each kit implements two closures: `selectChunkIds(limit)` + `applyStrategy(ids, strategy)`.
|
|
32
|
+
- **`WritingPurgeStrategy`** — strategy union with `skip` excluded (orchestrator handles `skip` before the port is consulted, so ports only see `hard` / `soft` / `anonymize`).
|
|
33
|
+
|
|
34
|
+
Hexagonal pattern: orchestrator is the use-case, `PurgePort` is the driving port, each kit's port factory is the adapter. Adding a new strategy (e.g. `archive`) = one union member + one case per port. Adding a new kit = one port file + ~10-line method.
|
|
35
|
+
|
|
36
|
+
### Added — 8 cross-kit conformance scenarios
|
|
37
|
+
|
|
38
|
+
In `src/testing/conformance.ts`, gated by the new `ConformanceFeatures.purgeByField?: boolean` flag. When both mongokit and sqlitekit pass the same scenarios, cross-kit byte-stability for tenant cleanup is provable:
|
|
39
|
+
|
|
40
|
+
1. `hard` removes every matching row, leaves others intact
|
|
41
|
+
2. `hard` empty match → `processed: 0`, `ok: true`
|
|
42
|
+
3. `anonymize` overwrites declared fields, keeps the row
|
|
43
|
+
4. `skip` is a no-op, returns reason
|
|
44
|
+
5. Chunking: `batchSize` honored, `onProgress` fires per chunk
|
|
45
|
+
6. Idempotent: re-running on the same tenant is a no-op
|
|
46
|
+
7. Scoping: only matching rows affected (cross-tenant safety)
|
|
47
|
+
8. Abort signal: stops between chunks, returns partial count + `ok: false`
|
|
48
|
+
|
|
49
|
+
The `soft` strategy is intentionally NOT in the conformance suite — it requires writable `deleted` / `deletedAt` fields not present on the shared `ConformanceDoc`; each kit covers `soft` in its own integration tests.
|
|
50
|
+
|
|
51
|
+
### Migration notes
|
|
52
|
+
|
|
53
|
+
- **Existing kits:** the new method is optional — kits don't break. mongokit 3.14.0 and sqlitekit 0.4.0 ship implementations; older kit versions continue to work, they just can't honor a `purgeByField` call.
|
|
54
|
+
- **Existing hosts:** no breaking changes. Hosts using arc's `cascadeDeleteForOrganization` automatically pick up the new strategy surface once arc 2.16.0 lands.
|
|
55
|
+
- **Build sync (workspace dev):** kits need `cp -r dist/* ../mongokit/node_modules/@classytic/repo-core/dist/` etc. after a workspace bump until npm publish.
|
|
56
|
+
|
|
7
57
|
## [0.4.0] - 2026-05-04
|
|
8
58
|
|
|
9
59
|
### Added — kit-shared building blocks (consolidation)
|
|
@@ -4,5 +4,6 @@ import { nestDottedKeys, nestDottedKeysAll } from "./agg-output.mjs";
|
|
|
4
4
|
import { PLUGIN_ORDER_CONSTRAINTS, Plugin, PluginFunction, PluginType, validatePluginOrder } from "./plugin-types.mjs";
|
|
5
5
|
import { RepositoryBase, RepositoryBaseOptions } from "./base.mjs";
|
|
6
6
|
import { STANDARD_REPO_OPTION_KEYS, StandardRepoOptionKey } from "./options.mjs";
|
|
7
|
-
import { AggCacheOptions, AggDateBucket, AggDateBucketInterval, AggDateBucketUnit, AggExecutionHints, AggMeasure, AggPaginationRequest, AggRequest, AggResult, AggRow, AggTopN, AggTopNTies, BulkCreateResult, BulkWriteOperation, BulkWriteResult, ClaimTransition, ClaimVersionTransition, DeleteManyResult, DeleteOptions, DeleteResult, FilterInput, FindOneAndUpdateOptions, InferDoc, KeysetAggPaginationResult, MinimalRepo, PaginationParams, QueryOptions, RepositorySession, StandardRepo, UpdateManyResult, WriteOptions } from "./types.mjs";
|
|
8
|
-
|
|
7
|
+
import { AggCacheOptions, AggDateBucket, AggDateBucketInterval, AggDateBucketUnit, AggExecutionHints, AggMeasure, AggPaginationRequest, AggRequest, AggResult, AggRow, AggTopN, AggTopNTies, BulkCreateResult, BulkWriteOperation, BulkWriteResult, ClaimTransition, ClaimVersionTransition, DeleteManyResult, DeleteOptions, DeleteResult, FilterInput, FindAllOptions, FindOneAndUpdateOptions, InferDoc, KeysetAggPaginationResult, MinimalRepo, PaginationParams, PurgeRetryPolicy, QueryOptions, RepositorySession, StandardRepo, TenantPurgeOptions, TenantPurgeProgress, TenantPurgeResult, TenantPurgeStrategy, UpdateManyResult, WriteOptions } from "./types.mjs";
|
|
8
|
+
import { PurgePort, WritingPurgeStrategy, runChunkedPurge } from "./purge.mjs";
|
|
9
|
+
export { type AggCacheOptions, type AggDateBucket, type AggDateBucketInterval, type AggDateBucketUnit, type AggExecutionHints, type AggMeasure, type AggPaginationRequest, type AggRequest, type AggResult, type AggRow, type AggTopN, type AggTopNTies, type BulkCreateResult, type BulkWriteOperation, type BulkWriteResult, type ClaimTransition, type ClaimVersionTransition, type DeleteManyResult, type DeleteOptions, type DeleteResult, type FilterInput, type FindAllOptions, type FindOneAndUpdateOptions, type InferDoc, type KeysetAggPaginationResult, type LookupPopulateOptions, type LookupPopulateResult, type LookupRow, type LookupSpec, type MinimalRepo, PLUGIN_ORDER_CONSTRAINTS, type PaginationParams, type Plugin, type PluginFunction, type PluginType, type PurgePort, type PurgeRetryPolicy, type QueryOptions, RepositoryBase, type RepositoryBaseOptions, type RepositorySession, STANDARD_REPO_OPTION_KEYS, type StandardRepo, type StandardRepoOptionKey, type TenantPurgeOptions, type TenantPurgeProgress, type TenantPurgeResult, type TenantPurgeStrategy, type UpdateInput, type UpdateManyResult, type WriteOptions, type WritingPurgeStrategy, nestDottedKeys, nestDottedKeysAll, runChunkedPurge, validatePluginOrder };
|
|
@@ -2,4 +2,5 @@ import { nestDottedKeys, nestDottedKeysAll } from "./agg-output.mjs";
|
|
|
2
2
|
import { PLUGIN_ORDER_CONSTRAINTS, validatePluginOrder } from "./plugin-types.mjs";
|
|
3
3
|
import { RepositoryBase } from "./base.mjs";
|
|
4
4
|
import { STANDARD_REPO_OPTION_KEYS } from "./options.mjs";
|
|
5
|
-
|
|
5
|
+
import { runChunkedPurge } from "./purge.mjs";
|
|
6
|
+
export { PLUGIN_ORDER_CONSTRAINTS, RepositoryBase, STANDARD_REPO_OPTION_KEYS, nestDottedKeys, nestDottedKeysAll, runChunkedPurge, validatePluginOrder };
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
import { TenantPurgeOptions, TenantPurgeResult, TenantPurgeStrategy } from "./types.mjs";
|
|
2
|
+
|
|
3
|
+
//#region src/repository/purge.d.ts
|
|
4
|
+
/**
|
|
5
|
+
* Strategies that perform a write. `skip` is handled by the orchestrator
|
|
6
|
+
* before the port is ever consulted, so ports only deal with the three
|
|
7
|
+
* writing variants.
|
|
8
|
+
*/
|
|
9
|
+
type WritingPurgeStrategy = Exclude<TenantPurgeStrategy, {
|
|
10
|
+
type: 'skip';
|
|
11
|
+
}>;
|
|
12
|
+
/**
|
|
13
|
+
* Driver-facing port the orchestrator drives. Each kit implements one
|
|
14
|
+
* closure over its driver primitives + the purge predicate.
|
|
15
|
+
*
|
|
16
|
+
* **Plugin-bypass invariant.** Implementations MUST bypass tenant
|
|
17
|
+
* scoping in plugin hooks — the caller's `field = value` predicate IS
|
|
18
|
+
* the authoritative scope; a tenant-injecting hook would narrow to the
|
|
19
|
+
* wrong tenant. Pass `bypassTenant: true` on inner Repository calls
|
|
20
|
+
* (which keeps audit / cache hooks active but disables tenant injection).
|
|
21
|
+
*
|
|
22
|
+
* **Throughput contract.** Implementations should issue the minimum
|
|
23
|
+
* number of round-trips a chunk requires:
|
|
24
|
+
*
|
|
25
|
+
* - `hard` on SQLite: `DELETE FROM t WHERE field = ? LIMIT n` — 1 RT
|
|
26
|
+
* - `hard` on Mongo: `find(filter, {_id:1}).limit(n)` + `deleteMany`
|
|
27
|
+
* — 2 RTs (Mongo has no DELETE LIMIT)
|
|
28
|
+
* - `soft`: read ids + updateMany with `$set: {deleted, deletedAt}` — 2 RTs
|
|
29
|
+
* - `anonymize` static fields: read ids + updateMany — 2 RTs
|
|
30
|
+
* - `anonymize` with function-form replacers: read docs +
|
|
31
|
+
* `bulkWrite([updateOne, …])` — 2 RTs (vs N+1 with per-doc fan-out)
|
|
32
|
+
*/
|
|
33
|
+
interface PurgePort {
|
|
34
|
+
/**
|
|
35
|
+
* Process one chunk under the given strategy. Returns the row count
|
|
36
|
+
* actually touched (≤ `limit`). The orchestrator loops until this
|
|
37
|
+
* returns less than `limit` (natural exit) or the abort signal fires.
|
|
38
|
+
*
|
|
39
|
+
* Returning `0` signals "no more matching rows"; the orchestrator
|
|
40
|
+
* exits. Returning a partial batch (`< limit`) is also a terminal
|
|
41
|
+
* signal — saves one round-trip on the last chunk.
|
|
42
|
+
*/
|
|
43
|
+
purgeChunk(strategy: WritingPurgeStrategy, limit: number): Promise<number>;
|
|
44
|
+
}
|
|
45
|
+
/**
|
|
46
|
+
* Drive a chunked purge to completion. Returns a `TenantPurgeResult`
|
|
47
|
+
* envelope describing what happened — never throws for in-strategy
|
|
48
|
+
* errors (those wrap into `result.error`); only throws for invalid
|
|
49
|
+
* input (`batchSize < 1`).
|
|
50
|
+
*
|
|
51
|
+
* @param strategy Strategy declaration — `skip` short-circuits.
|
|
52
|
+
* @param options Chunking + signal + progress + optional retry.
|
|
53
|
+
* @param port Kit-specific driver glue (one `purgeChunk` method).
|
|
54
|
+
*/
|
|
55
|
+
declare function runChunkedPurge(strategy: TenantPurgeStrategy, options: TenantPurgeOptions, port: PurgePort): Promise<TenantPurgeResult>;
|
|
56
|
+
//#endregion
|
|
57
|
+
export { PurgePort, WritingPurgeStrategy, runChunkedPurge };
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
//#region src/repository/purge.ts
|
|
2
|
+
/**
|
|
3
|
+
* Drive a chunked purge to completion. Returns a `TenantPurgeResult`
|
|
4
|
+
* envelope describing what happened — never throws for in-strategy
|
|
5
|
+
* errors (those wrap into `result.error`); only throws for invalid
|
|
6
|
+
* input (`batchSize < 1`).
|
|
7
|
+
*
|
|
8
|
+
* @param strategy Strategy declaration — `skip` short-circuits.
|
|
9
|
+
* @param options Chunking + signal + progress + optional retry.
|
|
10
|
+
* @param port Kit-specific driver glue (one `purgeChunk` method).
|
|
11
|
+
*/
|
|
12
|
+
async function runChunkedPurge(strategy, options, port) {
|
|
13
|
+
const start = Date.now();
|
|
14
|
+
if (strategy.type === "skip") return {
|
|
15
|
+
strategy: "skip",
|
|
16
|
+
processed: 0,
|
|
17
|
+
ok: true,
|
|
18
|
+
durationMs: Date.now() - start,
|
|
19
|
+
skipReason: strategy.reason
|
|
20
|
+
};
|
|
21
|
+
const batchSize = options.batchSize ?? 1e3;
|
|
22
|
+
if (!Number.isInteger(batchSize) || batchSize < 1) throw new Error("purgeByField: batchSize must be a positive integer");
|
|
23
|
+
const retry = options.retry;
|
|
24
|
+
let processed = 0;
|
|
25
|
+
try {
|
|
26
|
+
while (true) {
|
|
27
|
+
if (options.signal?.aborted) return {
|
|
28
|
+
strategy: strategy.type,
|
|
29
|
+
processed,
|
|
30
|
+
ok: false,
|
|
31
|
+
durationMs: Date.now() - start
|
|
32
|
+
};
|
|
33
|
+
const chunkSize = await runChunkWithRetry(() => port.purgeChunk(strategy, batchSize), retry);
|
|
34
|
+
if (chunkSize === 0) break;
|
|
35
|
+
processed += chunkSize;
|
|
36
|
+
if (options.onProgress) await options.onProgress({
|
|
37
|
+
processed,
|
|
38
|
+
chunkSize,
|
|
39
|
+
elapsedMs: Date.now() - start
|
|
40
|
+
});
|
|
41
|
+
if (chunkSize < batchSize) break;
|
|
42
|
+
}
|
|
43
|
+
} catch (err) {
|
|
44
|
+
return {
|
|
45
|
+
strategy: strategy.type,
|
|
46
|
+
processed,
|
|
47
|
+
ok: false,
|
|
48
|
+
durationMs: Date.now() - start,
|
|
49
|
+
error: {
|
|
50
|
+
message: err instanceof Error ? err.message : String(err),
|
|
51
|
+
chunkOffset: processed
|
|
52
|
+
}
|
|
53
|
+
};
|
|
54
|
+
}
|
|
55
|
+
return {
|
|
56
|
+
strategy: strategy.type,
|
|
57
|
+
processed,
|
|
58
|
+
ok: true,
|
|
59
|
+
durationMs: Date.now() - start
|
|
60
|
+
};
|
|
61
|
+
}
|
|
62
|
+
/**
|
|
63
|
+
* Run `fn` with exponential backoff when retry is enabled. Falls through
|
|
64
|
+
* to a single attempt when `retry` is undefined (default behavior).
|
|
65
|
+
*/
|
|
66
|
+
async function runChunkWithRetry(fn, retry) {
|
|
67
|
+
if (!retry) return fn();
|
|
68
|
+
const maxAttempts = retry.maxAttempts ?? 3;
|
|
69
|
+
const baseDelayMs = retry.baseDelayMs ?? 100;
|
|
70
|
+
const shouldRetry = retry.shouldRetry ?? (() => true);
|
|
71
|
+
let lastErr;
|
|
72
|
+
for (let attempt = 0; attempt < maxAttempts; attempt++) try {
|
|
73
|
+
return await fn();
|
|
74
|
+
} catch (err) {
|
|
75
|
+
lastErr = err;
|
|
76
|
+
if (attempt === maxAttempts - 1) break;
|
|
77
|
+
if (!shouldRetry(err, attempt + 1)) break;
|
|
78
|
+
await new Promise((r) => setTimeout(r, baseDelayMs * 2 ** attempt));
|
|
79
|
+
}
|
|
80
|
+
throw lastErr instanceof Error ? lastErr : new Error(String(lastErr));
|
|
81
|
+
}
|
|
82
|
+
//#endregion
|
|
83
|
+
export { runChunkedPurge };
|
|
@@ -59,6 +59,26 @@ interface WriteOptions extends QueryOptions {
|
|
|
59
59
|
/** Upsert on update/replace. */
|
|
60
60
|
upsert?: boolean;
|
|
61
61
|
}
|
|
62
|
+
/**
|
|
63
|
+
* Options for the optional `findAll` verb.
|
|
64
|
+
*
|
|
65
|
+
* `sort` is intentionally `Record<string, unknown>` (not a structured
|
|
66
|
+
* `SortSpec`) so kits can keep their own driver-shaped sort representation
|
|
67
|
+
* without forcing repo-core to depend on a particular dialect. Mongokit's
|
|
68
|
+
* `findAll` accepts `SortSpec | string`; sqlitekit accepts the same shape
|
|
69
|
+
* cross-walked into ORDER BY. `limit` is the bounded-but-not-paginated
|
|
70
|
+
* cap that callers reach for when `getAll` would force an unwanted count
|
|
71
|
+
* round-trip and `findAll` without a limit would over-fetch.
|
|
72
|
+
*
|
|
73
|
+
* Both fields are optional; absence means "kit default" (typically:
|
|
74
|
+
* `sort` defaults to natural order, `limit` defaults to "no limit").
|
|
75
|
+
*/
|
|
76
|
+
interface FindAllOptions extends QueryOptions {
|
|
77
|
+
/** Sort disambiguating when natural order isn't acceptable. */
|
|
78
|
+
sort?: Record<string, unknown> | string;
|
|
79
|
+
/** Cap the result set at the driver level. When omitted, returns all matching docs. */
|
|
80
|
+
limit?: number;
|
|
81
|
+
}
|
|
62
82
|
/**
|
|
63
83
|
* Delete-operation options.
|
|
64
84
|
*
|
|
@@ -83,6 +103,139 @@ interface FindOneAndUpdateOptions extends QueryOptions {
|
|
|
83
103
|
/** Insert when no doc matches. Default: false. */
|
|
84
104
|
upsert?: boolean;
|
|
85
105
|
}
|
|
106
|
+
/**
|
|
107
|
+
* Strategy for processing rows matched by a tenant/scope field — the
|
|
108
|
+
* decision every multi-tenant host makes when an organization (or any
|
|
109
|
+
* tenant) is deleted. Each variant maps to a kit-native primitive;
|
|
110
|
+
* arc's `createOrgDeleteCascade` orchestrates the per-resource declarations.
|
|
111
|
+
*
|
|
112
|
+
* **Why a discriminated union (not an enum)**: each strategy carries
|
|
113
|
+
* its own arguments — `fields` for anonymize, `reason` for skip. The
|
|
114
|
+
* union forces callers to supply them at the type level rather than
|
|
115
|
+
* fail at runtime.
|
|
116
|
+
*
|
|
117
|
+
* **Compliance shapes covered**:
|
|
118
|
+
* - `hard` — GDPR right-to-be-forgotten, SOC 2 deletion timelines.
|
|
119
|
+
* - `soft` — recoverable deletes within audit retention windows; pairs
|
|
120
|
+
* with TTL indexes (MongoDB `expireAfterSeconds` on `deletedAt`) for
|
|
121
|
+
* eventual hard-purge.
|
|
122
|
+
* - `anonymize` — records that legally must outlive the tenant (audit
|
|
123
|
+
* ledgers, financial records, medical history) but must lose PII
|
|
124
|
+
* linkage. HIPAA / PCI / SOX-compatible.
|
|
125
|
+
* - `skip` — explicitly opt out, with a mandatory `reason` that surfaces
|
|
126
|
+
* in audit reports.
|
|
127
|
+
*/
|
|
128
|
+
type TenantPurgeStrategy =
|
|
129
|
+
/**
|
|
130
|
+
* Permanently remove every matching row.
|
|
131
|
+
*/
|
|
132
|
+
{
|
|
133
|
+
type: 'hard';
|
|
134
|
+
}
|
|
135
|
+
/**
|
|
136
|
+
* Mark every matching row as deleted via the soft-delete convention.
|
|
137
|
+
* Pair with `softDeletePlugin` or a TTL index for eventual cleanup.
|
|
138
|
+
*/
|
|
139
|
+
| {
|
|
140
|
+
type: 'soft'; /** Boolean flag field set to `true`. Default `'deleted'`. */
|
|
141
|
+
deletedField?: string; /** Timestamp field set to purge-time. Default `'deletedAt'`. */
|
|
142
|
+
deletedAtField?: string;
|
|
143
|
+
}
|
|
144
|
+
/**
|
|
145
|
+
* Retain every matching row but overwrite the declared fields. Field
|
|
146
|
+
* values can be static (preferred — statically inspectable for audit)
|
|
147
|
+
* or per-row functions when deterministic transforms are needed
|
|
148
|
+
* (hashing, derived identifiers).
|
|
149
|
+
*/
|
|
150
|
+
| {
|
|
151
|
+
type: 'anonymize';
|
|
152
|
+
fields: Record<string, unknown | ((doc: Record<string, unknown>) => unknown)>;
|
|
153
|
+
}
|
|
154
|
+
/**
|
|
155
|
+
* Take no action. `reason` is required — an undocumented skip is a
|
|
156
|
+
* silent compliance leak. Surfaces in audit reports + introspection.
|
|
157
|
+
*/
|
|
158
|
+
| {
|
|
159
|
+
type: 'skip';
|
|
160
|
+
reason: string;
|
|
161
|
+
};
|
|
162
|
+
/**
|
|
163
|
+
* Per-call options for `purgeByField`. Chunking is required for
|
|
164
|
+
* correctness on large tenant datasets; the kit implementation MUST
|
|
165
|
+
* honor `batchSize` to avoid OOM / lock contention.
|
|
166
|
+
*/
|
|
167
|
+
interface TenantPurgeOptions {
|
|
168
|
+
/** Rows per batch. Default kit-specific (typically 1000). */
|
|
169
|
+
batchSize?: number;
|
|
170
|
+
/** Driver session for transactional callers. */
|
|
171
|
+
session?: RepositorySession;
|
|
172
|
+
/** Per-chunk progress callback. `processed` is cumulative. */
|
|
173
|
+
onProgress?: (event: TenantPurgeProgress) => void | Promise<void>;
|
|
174
|
+
/**
|
|
175
|
+
* Abort signal. Kits MUST check between chunks and finalize with the
|
|
176
|
+
* cumulative `processed` count when aborted (no rollback — chunks
|
|
177
|
+
* already committed remain committed; this is at-least-once cleanup).
|
|
178
|
+
*/
|
|
179
|
+
signal?: AbortSignal;
|
|
180
|
+
/**
|
|
181
|
+
* Retry transient chunk-level failures (network blips, write
|
|
182
|
+
* conflicts, busy-locks). Default `undefined` → no retry: first
|
|
183
|
+
* chunk error aborts the run. Hosts opt in for robustness:
|
|
184
|
+
*
|
|
185
|
+
* ```ts
|
|
186
|
+
* retry: {
|
|
187
|
+
* maxAttempts: 3, // default 3 when block present
|
|
188
|
+
* baseDelayMs: 100, // exponential: 100ms, 200ms, 400ms
|
|
189
|
+
* shouldRetry: (err) => // optional: narrow retry to transient
|
|
190
|
+
* /WriteConflict|SQLITE_BUSY|ECONNRESET/i.test(String(err)),
|
|
191
|
+
* }
|
|
192
|
+
* ```
|
|
193
|
+
*
|
|
194
|
+
* The retry happens at the CHUNK level — already-committed chunks
|
|
195
|
+
* stay committed. A retry that eventually succeeds reports `ok: true`;
|
|
196
|
+
* one that exhausts `maxAttempts` aborts with the underlying error.
|
|
197
|
+
*/
|
|
198
|
+
retry?: PurgeRetryPolicy;
|
|
199
|
+
}
|
|
200
|
+
/**
|
|
201
|
+
* Retry policy for `purgeByField` chunk-level failures. See
|
|
202
|
+
* `TenantPurgeOptions.retry` for the full contract.
|
|
203
|
+
*/
|
|
204
|
+
interface PurgeRetryPolicy {
|
|
205
|
+
/** Max attempts per chunk (including the first try). Default 3. */
|
|
206
|
+
maxAttempts?: number;
|
|
207
|
+
/** Base delay (ms) for exponential backoff. Default 100ms. */
|
|
208
|
+
baseDelayMs?: number;
|
|
209
|
+
/** Decide whether a given error is transient. Default: retry every error. */
|
|
210
|
+
shouldRetry?: (err: unknown, attempt: number) => boolean;
|
|
211
|
+
}
|
|
212
|
+
/** Chunk-level progress event for `purgeByField`. */
|
|
213
|
+
interface TenantPurgeProgress {
|
|
214
|
+
/** Rows processed so far (cumulative across chunks). */
|
|
215
|
+
processed: number;
|
|
216
|
+
/** Rows in the chunk that just completed. */
|
|
217
|
+
chunkSize: number;
|
|
218
|
+
/** Wall-clock ms elapsed since the call started. */
|
|
219
|
+
elapsedMs: number;
|
|
220
|
+
}
|
|
221
|
+
/** Final result of a `purgeByField` invocation. */
|
|
222
|
+
interface TenantPurgeResult {
|
|
223
|
+
/** Strategy that actually executed (echoes input.type). */
|
|
224
|
+
strategy: TenantPurgeStrategy['type'];
|
|
225
|
+
/** Total rows processed (0 for `skip`). */
|
|
226
|
+
processed: number;
|
|
227
|
+
/** True iff the call completed without abort / error. */
|
|
228
|
+
ok: boolean;
|
|
229
|
+
/** Wall-clock ms. */
|
|
230
|
+
durationMs: number;
|
|
231
|
+
/** First error if `ok: false`. Kits abort the run on a chunk failure. */
|
|
232
|
+
error?: {
|
|
233
|
+
message: string;
|
|
234
|
+
chunkOffset: number;
|
|
235
|
+
};
|
|
236
|
+
/** Echoed for `skip` strategy. Undefined for other strategies. */
|
|
237
|
+
skipReason?: string;
|
|
238
|
+
}
|
|
86
239
|
/**
|
|
87
240
|
* Transition spec for `StandardRepo.claim()` — a CAS state change.
|
|
88
241
|
*
|
|
@@ -1258,7 +1411,7 @@ interface StandardRepo<TDoc> extends MinimalRepo<TDoc> {
|
|
|
1258
1411
|
_id: unknown;
|
|
1259
1412
|
} | null>;
|
|
1260
1413
|
distinct?<T = unknown>(field: string, filter?: FilterInput, options?: QueryOptions): Promise<T[]>;
|
|
1261
|
-
findAll?(filter?: FilterInput, options?:
|
|
1414
|
+
findAll?(filter?: FilterInput, options?: FindAllOptions): Promise<TDoc[]>;
|
|
1262
1415
|
/**
|
|
1263
1416
|
* Atomic "look up by filter, insert `data` if missing, return the doc."
|
|
1264
1417
|
*
|
|
@@ -1308,6 +1461,56 @@ interface StandardRepo<TDoc> extends MinimalRepo<TDoc> {
|
|
|
1308
1461
|
* **Promoted from optional to required in repo-core 0.2.0.**
|
|
1309
1462
|
*/
|
|
1310
1463
|
deleteMany(filter: FilterInput, options?: DeleteOptions): Promise<DeleteManyResult>;
|
|
1464
|
+
/**
|
|
1465
|
+
* Compliance-grade cleanup primitive — processes every row matching
|
|
1466
|
+
* `field = value` under the given strategy. Powers tenant-scoped
|
|
1467
|
+
* data cleanup (org delete → cascade across every multi-tenant
|
|
1468
|
+
* resource) without forcing each consumer to hand-roll `deleteMany`
|
|
1469
|
+
* + chunking + audit + idempotency.
|
|
1470
|
+
*
|
|
1471
|
+
* Strategy → kit-native primitive:
|
|
1472
|
+
* - `hard` → chunked `deleteMany({ [field]: value })`
|
|
1473
|
+
* - `soft` → chunked `updateMany` setting deleted/deletedAt
|
|
1474
|
+
* - `anonymize` → chunked `updateMany` applying the field map per-row
|
|
1475
|
+
* - `skip` → no-op; `result.skipReason` echoes the declared reason
|
|
1476
|
+
*
|
|
1477
|
+
* **Chunking is mandatory.** Implementations MUST honor `batchSize`
|
|
1478
|
+
* — a 10M-row tenant cleanup can't run as a single `deleteMany`
|
|
1479
|
+
* (lock contention, oplog blowup, replication lag). Kits process
|
|
1480
|
+
* rows in chunks of `batchSize` (default ~1000) and emit per-chunk
|
|
1481
|
+
* `onProgress` events.
|
|
1482
|
+
*
|
|
1483
|
+
* **Index requirement — load-bearing for tractability.** The store
|
|
1484
|
+
* MUST have an index leading with `field` (single-field
|
|
1485
|
+
* `{ [field]: 1 }`, or a compound index whose first column is
|
|
1486
|
+
* `field`). Without it, every chunk's selection runs a full
|
|
1487
|
+
* collection / table scan — purge becomes O(n²) on large tenants
|
|
1488
|
+
* and can lock the table for minutes. Verify before shipping:
|
|
1489
|
+
* - mongo: `db.coll.getIndexes()` shows an index keyed on `field`.
|
|
1490
|
+
* - sqlite: `EXPLAIN QUERY PLAN SELECT … WHERE field = ?` shows
|
|
1491
|
+
* `SEARCH … USING INDEX`, never `SCAN`.
|
|
1492
|
+
*
|
|
1493
|
+
* **Idempotent.** Re-running with the same arguments is safe — rows
|
|
1494
|
+
* already deleted/anonymized simply don't match the next pass.
|
|
1495
|
+
* Crucial for at-least-once cascade workers that may retry after
|
|
1496
|
+
* partial failure.
|
|
1497
|
+
*
|
|
1498
|
+
* **Plugin composition.** Kits route the underlying chunked ops
|
|
1499
|
+
* through their standard `before:deleteMany` / `before:updateMany`
|
|
1500
|
+
* hooks so audit / cache-invalidation / observability plugins fire
|
|
1501
|
+
* naturally — no separate `before:purgeByField` hook is required.
|
|
1502
|
+
*
|
|
1503
|
+
* **Optional method.** Kits without bulk-cleanup needs leave this
|
|
1504
|
+
* undefined. Arc's `createOrgDeleteCascade` checks for the method
|
|
1505
|
+
* at boot and emits a clear error naming the offending resource if
|
|
1506
|
+
* a tenant-flagged resource's repo lacks it.
|
|
1507
|
+
*
|
|
1508
|
+
* @param field Document field to match against (e.g. `'organizationId'`).
|
|
1509
|
+
* @param value Value the field must equal (e.g. the deleted org id).
|
|
1510
|
+
* @param strategy Strategy declaration — see {@link TenantPurgeStrategy}.
|
|
1511
|
+
* @param options Chunking, session, progress, abort signal.
|
|
1512
|
+
*/
|
|
1513
|
+
purgeByField?(field: string, value: unknown, strategy: TenantPurgeStrategy, options?: TenantPurgeOptions): Promise<TenantPurgeResult>;
|
|
1311
1514
|
/**
|
|
1312
1515
|
* Heterogeneous bulk write. Stays optional — kits dispatch each op
|
|
1313
1516
|
* against the appropriate driver primitive inside a single transaction;
|
|
@@ -1407,4 +1610,4 @@ interface StandardRepo<TDoc> extends MinimalRepo<TDoc> {
|
|
|
1407
1610
|
withTransaction?<T>(fn: (txRepo: StandardRepo<TDoc>) => Promise<T>, options?: Record<string, unknown>): Promise<T>;
|
|
1408
1611
|
}
|
|
1409
1612
|
//#endregion
|
|
1410
|
-
export { AggCacheOptions, AggDateBucket, AggDateBucketInterval, AggDateBucketUnit, AggExecutionHints, AggMeasure, AggPaginationRequest, AggRequest, AggResult, AggRow, AggTopN, AggTopNTies, BulkCreateResult, BulkWriteOperation, BulkWriteResult, ClaimTransition, ClaimVersionTransition, DeleteManyResult, DeleteOptions, DeleteResult, FilterInput, FindOneAndUpdateOptions, InferDoc, KeysetAggPaginationResult, MinimalRepo, PaginationParams, QueryOptions, RepositorySession, StandardRepo, UpdateManyResult, WriteOptions };
|
|
1613
|
+
export { AggCacheOptions, AggDateBucket, AggDateBucketInterval, AggDateBucketUnit, AggExecutionHints, AggMeasure, AggPaginationRequest, AggRequest, AggResult, AggRow, AggTopN, AggTopNTies, BulkCreateResult, BulkWriteOperation, BulkWriteResult, ClaimTransition, ClaimVersionTransition, DeleteManyResult, DeleteOptions, DeleteResult, FilterInput, FindAllOptions, FindOneAndUpdateOptions, InferDoc, KeysetAggPaginationResult, MinimalRepo, PaginationParams, PurgeRetryPolicy, QueryOptions, RepositorySession, StandardRepo, TenantPurgeOptions, TenantPurgeProgress, TenantPurgeResult, TenantPurgeStrategy, UpdateManyResult, WriteOptions };
|
|
@@ -34,6 +34,7 @@ function runStandardRepoConformance(harness) {
|
|
|
34
34
|
const skipNoSubMinuteBuckets = !aggGate || !ops?.dateBucketSubMinute;
|
|
35
35
|
const skipNoStddev = !aggGate || !ops?.stddev;
|
|
36
36
|
const skipNoCache = !aggGate || !ops?.cache;
|
|
37
|
+
const skipNoPurge = !harness.features.purgeByField;
|
|
37
38
|
describe(`[conformance] ${harness.name}`, () => {
|
|
38
39
|
let ctx;
|
|
39
40
|
beforeEach(async () => {
|
|
@@ -1124,6 +1125,149 @@ function runStandardRepoConformance(harness) {
|
|
|
1124
1125
|
});
|
|
1125
1126
|
});
|
|
1126
1127
|
});
|
|
1128
|
+
describe("purgeByField (tenant cleanup)", () => {
|
|
1129
|
+
const seedTwoTenants = async () => {
|
|
1130
|
+
await ctx.repo.create(harness.makeDoc({
|
|
1131
|
+
name: "a-1",
|
|
1132
|
+
category: "org-a"
|
|
1133
|
+
}));
|
|
1134
|
+
await ctx.repo.create(harness.makeDoc({
|
|
1135
|
+
name: "a-2",
|
|
1136
|
+
category: "org-a"
|
|
1137
|
+
}));
|
|
1138
|
+
await ctx.repo.create(harness.makeDoc({
|
|
1139
|
+
name: "a-3",
|
|
1140
|
+
category: "org-a"
|
|
1141
|
+
}));
|
|
1142
|
+
await ctx.repo.create(harness.makeDoc({
|
|
1143
|
+
name: "b-1",
|
|
1144
|
+
category: "org-b"
|
|
1145
|
+
}));
|
|
1146
|
+
await ctx.repo.create(harness.makeDoc({
|
|
1147
|
+
name: "b-2",
|
|
1148
|
+
category: "org-b"
|
|
1149
|
+
}));
|
|
1150
|
+
};
|
|
1151
|
+
it.skipIf(skipNoPurge)("hard: removes every matching row, leaves others intact", async () => {
|
|
1152
|
+
await seedTwoTenants();
|
|
1153
|
+
const result = await ctx.repo.purgeByField("category", "org-a", { type: "hard" });
|
|
1154
|
+
expect(result.ok).toBe(true);
|
|
1155
|
+
expect(result.strategy).toBe("hard");
|
|
1156
|
+
expect(result.processed).toBe(3);
|
|
1157
|
+
expect(typeof result.durationMs).toBe("number");
|
|
1158
|
+
expect(await ctx.repo.count({ category: "org-a" })).toBe(0);
|
|
1159
|
+
expect(await ctx.repo.count({ category: "org-b" })).toBe(2);
|
|
1160
|
+
});
|
|
1161
|
+
it.skipIf(skipNoPurge)("hard: empty match completes ok with processed: 0", async () => {
|
|
1162
|
+
const result = await ctx.repo.purgeByField("category", "nonexistent", { type: "hard" });
|
|
1163
|
+
expect(result.ok).toBe(true);
|
|
1164
|
+
expect(result.processed).toBe(0);
|
|
1165
|
+
});
|
|
1166
|
+
it.skipIf(skipNoPurge)("anonymize: overwrites declared fields, keeps the row", async () => {
|
|
1167
|
+
await seedTwoTenants();
|
|
1168
|
+
const result = await ctx.repo.purgeByField("category", "org-a", {
|
|
1169
|
+
type: "anonymize",
|
|
1170
|
+
fields: {
|
|
1171
|
+
name: "[REDACTED]",
|
|
1172
|
+
notes: null
|
|
1173
|
+
}
|
|
1174
|
+
});
|
|
1175
|
+
expect(result.ok).toBe(true);
|
|
1176
|
+
expect(result.strategy).toBe("anonymize");
|
|
1177
|
+
expect(result.processed).toBe(3);
|
|
1178
|
+
expect(await ctx.repo.count({ category: "org-a" })).toBe(3);
|
|
1179
|
+
const redacted = await ctx.repo.findAll({ category: "org-a" });
|
|
1180
|
+
for (const row of redacted) {
|
|
1181
|
+
expect(row.name).toBe("[REDACTED]");
|
|
1182
|
+
expect(row.notes).toBeNull();
|
|
1183
|
+
}
|
|
1184
|
+
const others = await ctx.repo.findAll({ category: "org-b" });
|
|
1185
|
+
for (const row of others) expect(row.name).not.toBe("[REDACTED]");
|
|
1186
|
+
});
|
|
1187
|
+
it.skipIf(skipNoPurge)("skip: no-op, returns reason and processed: 0", async () => {
|
|
1188
|
+
await seedTwoTenants();
|
|
1189
|
+
const result = await ctx.repo.purgeByField("category", "org-a", {
|
|
1190
|
+
type: "skip",
|
|
1191
|
+
reason: "audit-retained-per-SOX"
|
|
1192
|
+
});
|
|
1193
|
+
expect(result.ok).toBe(true);
|
|
1194
|
+
expect(result.strategy).toBe("skip");
|
|
1195
|
+
expect(result.processed).toBe(0);
|
|
1196
|
+
expect(result.skipReason).toBe("audit-retained-per-SOX");
|
|
1197
|
+
expect(await ctx.repo.count({ category: "org-a" })).toBe(3);
|
|
1198
|
+
expect(await ctx.repo.count({ category: "org-b" })).toBe(2);
|
|
1199
|
+
});
|
|
1200
|
+
it.skipIf(skipNoPurge)("chunking: batchSize honored, onProgress fires per chunk", async () => {
|
|
1201
|
+
for (let i = 0; i < 25; i++) await ctx.repo.create(harness.makeDoc({
|
|
1202
|
+
name: `chunk-${i}`,
|
|
1203
|
+
category: "org-chunk"
|
|
1204
|
+
}));
|
|
1205
|
+
const progressEvents = [];
|
|
1206
|
+
expect((await ctx.repo.purgeByField("category", "org-chunk", { type: "hard" }, {
|
|
1207
|
+
batchSize: 10,
|
|
1208
|
+
onProgress: (event) => {
|
|
1209
|
+
progressEvents.push({
|
|
1210
|
+
processed: event.processed,
|
|
1211
|
+
chunkSize: event.chunkSize
|
|
1212
|
+
});
|
|
1213
|
+
}
|
|
1214
|
+
})).processed).toBe(25);
|
|
1215
|
+
expect(progressEvents.length).toBe(3);
|
|
1216
|
+
expect(progressEvents[0]).toEqual({
|
|
1217
|
+
processed: 10,
|
|
1218
|
+
chunkSize: 10
|
|
1219
|
+
});
|
|
1220
|
+
expect(progressEvents[1]).toEqual({
|
|
1221
|
+
processed: 20,
|
|
1222
|
+
chunkSize: 10
|
|
1223
|
+
});
|
|
1224
|
+
expect(progressEvents[2]).toEqual({
|
|
1225
|
+
processed: 25,
|
|
1226
|
+
chunkSize: 5
|
|
1227
|
+
});
|
|
1228
|
+
expect(await ctx.repo.count({ category: "org-chunk" })).toBe(0);
|
|
1229
|
+
});
|
|
1230
|
+
it.skipIf(skipNoPurge)("idempotent: re-running on the same tenant is a no-op", async () => {
|
|
1231
|
+
await seedTwoTenants();
|
|
1232
|
+
expect((await ctx.repo.purgeByField("category", "org-a", { type: "hard" })).processed).toBe(3);
|
|
1233
|
+
const second = await ctx.repo.purgeByField("category", "org-a", { type: "hard" });
|
|
1234
|
+
expect(second.ok).toBe(true);
|
|
1235
|
+
expect(second.processed).toBe(0);
|
|
1236
|
+
});
|
|
1237
|
+
it.skipIf(skipNoPurge)("scoping: only rows matching field=value are affected", async () => {
|
|
1238
|
+
await seedTwoTenants();
|
|
1239
|
+
const totalBefore = await ctx.repo.count({});
|
|
1240
|
+
await ctx.repo.purgeByField("category", "org-a", { type: "hard" });
|
|
1241
|
+
expect(await ctx.repo.count({})).toBe(totalBefore - 3);
|
|
1242
|
+
expect(await ctx.repo.count({ category: "org-b" })).toBe(2);
|
|
1243
|
+
});
|
|
1244
|
+
it.skipIf(skipNoPurge)("abort signal: stops between chunks, returns partial count", async () => {
|
|
1245
|
+
for (let i = 0; i < 25; i++) await ctx.repo.create(harness.makeDoc({
|
|
1246
|
+
name: `abort-${i}`,
|
|
1247
|
+
category: "org-abort"
|
|
1248
|
+
}));
|
|
1249
|
+
const controller = new AbortController();
|
|
1250
|
+
const result = await ctx.repo.purgeByField("category", "org-abort", { type: "hard" }, {
|
|
1251
|
+
batchSize: 10,
|
|
1252
|
+
signal: controller.signal,
|
|
1253
|
+
onProgress: (event) => {
|
|
1254
|
+
if (event.processed === 10) controller.abort();
|
|
1255
|
+
}
|
|
1256
|
+
});
|
|
1257
|
+
expect(result.ok).toBe(false);
|
|
1258
|
+
expect(result.processed).toBe(10);
|
|
1259
|
+
expect(await ctx.repo.count({ category: "org-abort" })).toBe(15);
|
|
1260
|
+
});
|
|
1261
|
+
it.skipIf(skipNoPurge)("retry policy is plumbed through (default no retry, opt-in works)", async () => {
|
|
1262
|
+
await seedTwoTenants();
|
|
1263
|
+
const result = await ctx.repo.purgeByField("category", "org-a", { type: "hard" }, { retry: {
|
|
1264
|
+
maxAttempts: 1,
|
|
1265
|
+
baseDelayMs: 10
|
|
1266
|
+
} });
|
|
1267
|
+
expect(result.ok).toBe(true);
|
|
1268
|
+
expect(result.processed).toBe(3);
|
|
1269
|
+
});
|
|
1270
|
+
});
|
|
1127
1271
|
});
|
|
1128
1272
|
}
|
|
1129
1273
|
//#endregion
|
package/dist/testing/types.d.mts
CHANGED
|
@@ -132,6 +132,13 @@ interface ConformanceFeatures {
|
|
|
132
132
|
getOrCreate: boolean;
|
|
133
133
|
/** `count(filter)` and `exists(filter)`. */
|
|
134
134
|
countAndExists: boolean;
|
|
135
|
+
/**
|
|
136
|
+
* `purgeByField(field, value, strategy, options)` — compliance-grade
|
|
137
|
+
* tenant cleanup primitive. Both mongokit and sqlitekit ship this as
|
|
138
|
+
* of repo-core 0.x. Future kits without it leave the flag absent
|
|
139
|
+
* (defaults to false) and skip the cleanup scenarios.
|
|
140
|
+
*/
|
|
141
|
+
purgeByField?: boolean;
|
|
135
142
|
}
|
|
136
143
|
/**
|
|
137
144
|
* One-shot context produced by `harness.setup()`. Scenarios receive a
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@classytic/repo-core",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0",
|
|
4
4
|
"description": "Driver-agnostic repository primitives: hooks, Filter IR, operations, pagination, cache contract. Foundation for mongokit, sqlitekit, pgkit, and prismakit. Lean by design — no plugins ship here; each kit owns its own.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"sideEffects": false,
|