@classytic/repo-core 0.13.0 → 0.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +146 -0
- package/README.md +2 -0
- package/dist/cleanup/index.d.mts +2 -0
- package/dist/cleanup/index.mjs +0 -0
- package/dist/cleanup/types.d.mts +167 -0
- package/dist/filter/index.d.mts +2 -1
- package/dist/filter/index.mjs +2 -1
- package/dist/filter/match-record.d.mts +19 -0
- package/dist/filter/match-record.mjs +187 -0
- package/dist/filter/match.mjs +103 -32
- package/dist/repository/capabilities.d.mts +9 -0
- package/dist/repository/purge.d.mts +40 -4
- package/dist/repository/types.d.mts +42 -4
- package/dist/testing/index.d.mts +2 -1
- package/dist/testing/index.mjs +2 -1
- package/dist/testing/purge-conformance.d.mts +33 -0
- package/dist/testing/purge-conformance.mjs +160 -0
- package/package.json +5 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,152 @@ All notable changes to `@classytic/repo-core` are documented here.
|
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
5
5
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## [0.17.0] - 2026-07-25
|
|
8
|
+
|
|
9
|
+
### Added — `./cleanup`: framework-free cleanup provider step contract
|
|
10
|
+
|
|
11
|
+
- **`@classytic/repo-core/cleanup`** — new subpath (pure types, no runtime).
|
|
12
|
+
Domain kernels (`@classytic/flow`, `@classytic/order`, `@classytic/facts`,
|
|
13
|
+
`@classytic/ledger`, …) export `CleanupStep[]` for the data they own; a host
|
|
14
|
+
framework (`@classytic/arc/cleanup`) folds them into a Cleanup Center recipe.
|
|
15
|
+
The contract lives in repo-core so kernels stay framework-free: kernels already
|
|
16
|
+
depend on repo-core (the chunked-purge mechanics) and MUST NOT depend on arc.
|
|
17
|
+
- **`CleanupStep`** — the core provider interface: `id`, `resource`, `destructive`,
|
|
18
|
+
optional `rebuildActions`; and three lifecycle methods: `estimate(ctx)` (preview
|
|
19
|
+
without mutating), `execute(ctx)` (chunked, idempotent, cancellation-aware),
|
|
20
|
+
and optional `verify(ctx)` (post-checks — a delete count alone is never success).
|
|
21
|
+
- **Supporting types**: `CleanupStepContext` (injected `now`, `signal`, `ambient`,
|
|
22
|
+
`parameters`, `logger`), `CleanupStepExecuteContext` (adds `onProgress` +
|
|
23
|
+
`throwIfCancelled`), `CleanupStepEstimate` (row count, `retained`, `blockers`,
|
|
24
|
+
`warnings`), `CleanupStepOutcome` (`processed`, `ok`, `error`, `cursor`),
|
|
25
|
+
`CleanupStepCheck` (`name`, `ok`, `detail`), `CleanupStepProgress`, `CleanupStepLogger`.
|
|
26
|
+
|
|
27
|
+
Purely additive — zero runtime code, type-only subpath export.
|
|
28
|
+
|
|
29
|
+
## [0.16.0] - 2026-07-24
|
|
30
|
+
|
|
31
|
+
### Added — `runPurgeConformance`: cross-kit chunked-purge contract suite
|
|
32
|
+
|
|
33
|
+
- **`runPurgeConformance(harness)`** (from `@classytic/repo-core/testing`) —
|
|
34
|
+
proves a kit's purge port makes **stable progress for every strategy** when
|
|
35
|
+
the match set exceeds `batchSize`. Every scenario seeds more rows than one
|
|
36
|
+
batch; a chunk budget converts a non-progressing port's infinite loop into a
|
|
37
|
+
crisp assertion failure. Scenarios: hard drain, soft multi-batch WITHOUT
|
|
38
|
+
caller-supplied exclusion predicates, anonymize static + function-form,
|
|
39
|
+
exact-batch boundary, skip, empty scope, abort-between-chunks.
|
|
40
|
+
- **`PurgePort` progression contract (documented, mandatory)** — successive
|
|
41
|
+
`purgeChunk` calls MUST advance through the match set for every strategy via
|
|
42
|
+
stable keyset progression (`pk > lastSeen`, advanced only after the chunk's
|
|
43
|
+
write succeeds). Re-running the bare predicate only self-advances for
|
|
44
|
+
`hard`; `soft`/`anonymize` re-select the same first chunk forever. Offsets
|
|
45
|
+
are explicitly ruled out. A port instance is single-run state.
|
|
46
|
+
|
|
47
|
+
### Fixed
|
|
48
|
+
|
|
49
|
+
- `purgeByField` docstring no longer claims soft/anonymized rows "simply don't
|
|
50
|
+
match the next pass" — they generally DO still match; idempotency holds by
|
|
51
|
+
outcome convergence, and within-run progression is the port's keyset
|
|
52
|
+
responsibility.
|
|
53
|
+
|
|
54
|
+
## [0.15.0] - 2026-07-24
|
|
55
|
+
|
|
56
|
+
### Added — `purgeByFilter`: range/filter-scoped purge + anonymize
|
|
57
|
+
|
|
58
|
+
- **`StandardRepo.purgeByFilter?(filter, strategy, options)`** — the
|
|
59
|
+
range/filter-scoped sibling of `purgeByField`. Where `purgeByField` matches
|
|
60
|
+
a single `field = value` equality, this optional method takes the full
|
|
61
|
+
portable `FilterInput` (Filter IR or a plain kit-native record) and runs any
|
|
62
|
+
`TenantPurgeStrategy` (`hard` / `soft` / `anonymize` / `skip`) over the
|
|
63
|
+
matched slice. THE compliance primitive for "purge/anonymize a dimension
|
|
64
|
+
across a RANGE while RETAINING measures" — redact a PII column across a
|
|
65
|
+
`civilDate` window, hard-delete rows past a retention cutoff, soft-delete a
|
|
66
|
+
compound cohort. Returns the same `TenantPurgeResult` envelope; chunking,
|
|
67
|
+
index requirement, idempotency, plugin composition, and narrowed-write
|
|
68
|
+
re-assertion are identical to `purgeByField`. Gate on the new
|
|
69
|
+
`capabilities.purgeByFilter`.
|
|
70
|
+
- **`RepoCapabilities.purgeByFilter?: boolean`** — feature-detection flag,
|
|
71
|
+
mirroring `purgeByField`.
|
|
72
|
+
- **`PurgePort` doc** clarifies the two bound-predicate forms (equality-bound
|
|
73
|
+
vs filter-bound) both satisfy the single port interface, so `runChunkedPurge`
|
|
74
|
+
drives both unchanged.
|
|
75
|
+
|
|
76
|
+
Strictly additive: `purgeByField`, `TenantPurgeStrategy`, and `runChunkedPurge`
|
|
77
|
+
are unchanged; both new members are optional.
|
|
78
|
+
|
|
79
|
+
## [0.14.0] - 2026-07-16
|
|
80
|
+
|
|
81
|
+
### Added — canonical `matchesRecordFilter` (the `DataAdapter.matchesFilter` home)
|
|
82
|
+
|
|
83
|
+
- **`matchesRecordFilter(item, record)`** + **`policyRecordToFilter(record)`**
|
|
84
|
+
(`@classytic/repo-core/filter`) — THE single, shared implementation of the
|
|
85
|
+
`DataAdapter.matchesFilter` seam. Evaluates arc's Mongo-record `_policyFilters`
|
|
86
|
+
(`{ ownerId }`, `{ organizationId }`, `{ $or: [{ ownerId }, { _id: { $in } }] }`)
|
|
87
|
+
against an already-fetched document IN PROCESS, by converting to the portable
|
|
88
|
+
`Filter` IR and delegating to `matchFilter` — the SAME IR kits compile to
|
|
89
|
+
SQL/Mongo, so in-memory and DB enforcement agree by construction. Every kit's
|
|
90
|
+
adapter now delegates here; **no per-kit matcher, one contract, one IR.**
|
|
91
|
+
Operator scope: `$or`/`$and`/`$nor`/`$not`, `$eq`/`$ne`/`$gt`/`$gte`/`$lt`/`$lte`,
|
|
92
|
+
`$in`/`$nin`, `$exists`, implicit-eq; fails LOUD on anything else. (Distinct from
|
|
93
|
+
`recordToFilter`, the bare-operator query normalizer that does not accept
|
|
94
|
+
`$`-prefixed or logical operators.)
|
|
95
|
+
- **`matchFilter` is now fully array + id + date aware** (additive superset;
|
|
96
|
+
primitive/Date-vs-Date behavior unchanged — strict cases still short-circuit):
|
|
97
|
+
- **id coercion** — an id-like object with a meaningful `toString` (Mongo
|
|
98
|
+
`ObjectId`, `Buffer`, `Decimal128`) matches its string form, so one shared
|
|
99
|
+
matcher serves Mongo (`ObjectId` `_id`) and SQL (primitive ids) alike.
|
|
100
|
+
- **array semantics** — dot-paths fan out over subdocument arrays
|
|
101
|
+
(`items.sku` on `[{sku},{sku}]`), and a scalar condition on a leaf array
|
|
102
|
+
field matches when ANY element satisfies it (`{ tags: 'x' }`,
|
|
103
|
+
`{ scores: { $gt: 5 } }`, regex on array elements). All array unwrapping is
|
|
104
|
+
concentrated in one helper; the comparators stay pure scalar.
|
|
105
|
+
- **Date⇄ISO-string range** — `compare` coerces the string side to a date
|
|
106
|
+
instant when the other side is a genuine `Date` (mirrors `equals`), and
|
|
107
|
+
never claims ordering across a number/string type boundary (fails closed
|
|
108
|
+
instead of matching spuriously).
|
|
109
|
+
- **prototype-pollution-safe reads** — path resolution uses `Object.hasOwn`,
|
|
110
|
+
so a crafted `{ '__proto__.x': … }` / `{ constructor.name: … }` filter can
|
|
111
|
+
never traverse the prototype chain.
|
|
112
|
+
- **Ability parity across kits**: mongokit's earlier standalone matcher
|
|
113
|
+
(subdocument-array fan-out, array-contains, `$regex`, ObjectId coercion) is now
|
|
114
|
+
fully covered by the shared engine — nothing was lost in consolidation.
|
|
115
|
+
- **MongoDB-parity hardening** (validated against the MongoDB manual + sift.js +
|
|
116
|
+
mingo — the two industry-standard in-memory matchers). The authorization-critical
|
|
117
|
+
rule "an absent field participates in comparisons as null/undefined" is now
|
|
118
|
+
fully honored:
|
|
119
|
+
- `{ field: null }`, `$ne`, `$nin`, and a `null` MEMBER of `$in`/`$nin` all
|
|
120
|
+
match a MISSING field (a policy filter `{ status: { $ne: 'archived' } }`
|
|
121
|
+
correctly returns docs that lack the field, exactly as MongoDB does — the top
|
|
122
|
+
silent-authorization-divergence trap). `$ne: null` remains the exception
|
|
123
|
+
(requires present + non-null).
|
|
124
|
+
- `$in` accepts RegExp-literal members (`{ name: { $in: [/^a/] } }`).
|
|
125
|
+
- Comparison ops are type-bracketed (no cross-type ordering; `$gt: null` matches
|
|
126
|
+
nothing); NaN equals NaN for `$eq` (via `Object.is`, not `===`).
|
|
127
|
+
- Numeric dot-path segments resolve as positional array indices (`items.0.sku`).
|
|
128
|
+
- Documented DELIBERATE divergences: `$exists` = present-and-non-null (matches
|
|
129
|
+
the IR `exists` op + SQL `IS NOT NULL` + sift; Mongo/mingo count present-null
|
|
130
|
+
as existing); `Date`⇄ISO-string range leniency; array-literal operands are
|
|
131
|
+
element-matched not exact-matched.
|
|
132
|
+
- New: 41-case matcher suite incl. the 11 researched MongoDB gotchas + adversarial
|
|
133
|
+
(prototype pollution, NaN/Infinity, boolean/zero/empty-string, empty
|
|
134
|
+
`$in`/`$or`/`$and`/`$nor`, fail-loud on unsupported operators) + a 100k-doc
|
|
135
|
+
performance smoke (linear, cached regex).
|
|
136
|
+
- **Security hardening** (from a review against sift.js/mingo CVEs + the OWASP
|
|
137
|
+
NoSQL-injection / ReDoS / prototype-pollution literature):
|
|
138
|
+
- **Prototype-key denylist** on path segments — `__proto__` / `constructor` /
|
|
139
|
+
`prototype` resolve to nothing (fail-closed), string-normalized (the
|
|
140
|
+
object-path CVE-2021-23434 lesson: an array-typed segment bypassed a `===`
|
|
141
|
+
check). Closes the "match an inherited member → wrong auth answer" case and
|
|
142
|
+
the `JSON.parse('{"__proto__":…}')` own-property vector, on top of the
|
|
143
|
+
already-`Object.hasOwn` reads.
|
|
144
|
+
- **`$regex` input-length cap** (64 KiB) — ReDoS is `pattern × input`; a field
|
|
145
|
+
value longer than the cap is treated as no-match (fail-closed) so one slow
|
|
146
|
+
match can't stall the event loop and amplify across a realtime fan-out
|
|
147
|
+
(matcher runs once per subscriber per record). Pattern-side ReDoS is a
|
|
148
|
+
non-issue for framework-supplied (trusted) patterns.
|
|
149
|
+
- Reaffirmed **fail-closed** posture: comparisons never coerce across a
|
|
150
|
+
number/string type boundary (return no-match rather than JS-coerced nonsense
|
|
151
|
+
— the classic over-visibility leak), and unsupported operators throw.
|
|
152
|
+
|
|
7
153
|
## [0.13.0] - 2026-07-15
|
|
8
154
|
|
|
9
155
|
### Added — `StandardRepo.applyTransition?()` contract (state-machine CAS with history)
|
package/README.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# @classytic/repo-core
|
|
2
2
|
|
|
3
|
+
[](https://github.com/sponsors/classytic)
|
|
4
|
+
|
|
3
5
|
**Driver-agnostic repository primitives.** Hooks, Filter IR, operations registry, pagination, URL query parsing, cache contract — the shared foundation for `@classytic/mongokit`, `@classytic/sqlitekit`, and future `@classytic/pgkit` / `@classytic/prismakit`.
|
|
4
6
|
|
|
5
7
|
Repo-core is **infrastructure for kit authors.** End-users install a kit (mongokit / sqlitekit) and import their full API from that one namespace. Repo-core is what each kit's runtime is built on — you typically won't import it directly in application code.
|
|
@@ -0,0 +1,2 @@
|
|
|
1
|
+
import { CleanupStep, CleanupStepCheck, CleanupStepContext, CleanupStepEstimate, CleanupStepExecuteContext, CleanupStepLogger, CleanupStepOutcome, CleanupStepProgress } from "./types.mjs";
|
|
2
|
+
export type { CleanupStep, CleanupStepCheck, CleanupStepContext, CleanupStepEstimate, CleanupStepExecuteContext, CleanupStepLogger, CleanupStepOutcome, CleanupStepProgress };
|
|
File without changes
|
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
//#region src/cleanup/types.d.ts
|
|
2
|
+
/**
|
|
3
|
+
* `@classytic/repo-core/cleanup` — the framework-free **cleanup provider step**
|
|
4
|
+
* contract (data-cleanup / retention design §6.3, §6.6).
|
|
5
|
+
*
|
|
6
|
+
* A domain kernel (`@classytic/flow`, `@classytic/order`, `@classytic/facts`,
|
|
7
|
+
* `@classytic/ledger`, …) exports one or more `CleanupStep`s for the data it
|
|
8
|
+
* owns — the unit retention.md §6.6 calls "cleanup recipe steps." A host
|
|
9
|
+
* (`be-prod`) composes ordered steps into a full Cleanup Center recipe.
|
|
10
|
+
*
|
|
11
|
+
* This contract lives in repo-core — NOT in `@classytic/arc` — on purpose:
|
|
12
|
+
*
|
|
13
|
+
* - Kernels already depend on repo-core (it owns the chunked purge mechanics,
|
|
14
|
+
* §6.3) and MUST stay free of any host-framework dependency. A kernel that
|
|
15
|
+
* imported `@classytic/arc/cleanup` would invert the layering.
|
|
16
|
+
* - Arc's Cleanup Center framework (`@classytic/arc/cleanup`) imports THIS
|
|
17
|
+
* contract and folds `CleanupStep[]` into an arc `CleanupRecipe` via its
|
|
18
|
+
* `recipeFromSteps()` composer. One shape, two consumers, zero cycle.
|
|
19
|
+
*
|
|
20
|
+
* A step is a PURE PROVIDER: it knows only how to estimate / execute / verify a
|
|
21
|
+
* slice of its own data. It never authorizes a superadmin, reads go-live state,
|
|
22
|
+
* speaks HTTP, or opens a Mongo transaction the host didn't hand it. It reuses
|
|
23
|
+
* repo-core's own chunked-purge envelope (`TenantPurgeResult` / progress) under
|
|
24
|
+
* the hood; this contract is the composable wrapper around that envelope.
|
|
25
|
+
*/
|
|
26
|
+
/**
|
|
27
|
+
* Framework-free ambient a cleanup step reads. A structural SUBSET of any host
|
|
28
|
+
* cleanup context (e.g. arc's `CleanupContext`), so passing the host context
|
|
29
|
+
* straight through type-checks — no adapter object required.
|
|
30
|
+
*/
|
|
31
|
+
interface CleanupStepContext {
|
|
32
|
+
/** Injected clock — steps never call `new Date()` directly (testability). */
|
|
33
|
+
readonly now: Date;
|
|
34
|
+
/** Cooperative cancellation — observed between chunks. */
|
|
35
|
+
readonly signal?: AbortSignal | undefined;
|
|
36
|
+
/**
|
|
37
|
+
* Opaque host-provided ambient scope (resolved company/branch, feature
|
|
38
|
+
* gates, …). MUST be JSON-serializable — a host may persist it on a durable
|
|
39
|
+
* run so a worker in another process rebuilds the exact operation context.
|
|
40
|
+
* A step reads only what it declared it needs; nobody else inspects it.
|
|
41
|
+
*/
|
|
42
|
+
readonly ambient?: Readonly<Record<string, unknown>> | undefined;
|
|
43
|
+
/**
|
|
44
|
+
* Operator-supplied recipe parameters (branch id, created-before date, module
|
|
45
|
+
* set, …). Opaque + host-validated at the edge; a step reads/parses the keys
|
|
46
|
+
* it declares. Recipes are boot-time singletons, so parameters arrive HERE at
|
|
47
|
+
* plan/execute time rather than being closed over at construction. The host
|
|
48
|
+
* composer threads the plan's sealed parameters through unchanged, so a
|
|
49
|
+
* worker replays the exact same op.
|
|
50
|
+
*/
|
|
51
|
+
readonly parameters?: Readonly<Record<string, unknown>> | undefined;
|
|
52
|
+
/** Optional structured logger. */
|
|
53
|
+
readonly logger?: CleanupStepLogger | undefined;
|
|
54
|
+
}
|
|
55
|
+
interface CleanupStepLogger {
|
|
56
|
+
info(msg: string, meta?: Record<string, unknown>): void;
|
|
57
|
+
warn(msg: string, meta?: Record<string, unknown>): void;
|
|
58
|
+
error(msg: string, meta?: Record<string, unknown>): void;
|
|
59
|
+
}
|
|
60
|
+
/** One committed chunk's progress — folded into the host run's bounded summary. */
|
|
61
|
+
interface CleanupStepProgress {
|
|
62
|
+
readonly resource: string;
|
|
63
|
+
/** Cumulative rows processed by THIS step so far. */
|
|
64
|
+
readonly processed: number;
|
|
65
|
+
/** Opaque resume cursor (keyset position) for observability. */
|
|
66
|
+
readonly cursor?: string | undefined;
|
|
67
|
+
}
|
|
68
|
+
/**
|
|
69
|
+
* Execution-time context — adds progress reporting + a durable-cancellation
|
|
70
|
+
* probe. Both are optional so a step is trivially runnable in a unit test with
|
|
71
|
+
* a bare `CleanupStepContext`.
|
|
72
|
+
*/
|
|
73
|
+
interface CleanupStepExecuteContext extends CleanupStepContext {
|
|
74
|
+
/** Report one committed chunk. Call AFTER the chunk's write commits. */
|
|
75
|
+
onProgress?(update: CleanupStepProgress): void | Promise<void>;
|
|
76
|
+
/**
|
|
77
|
+
* Throw the host's cancellation error if a cancel was requested for this run.
|
|
78
|
+
* Cheap to call between chunks; backed by the host's durable `cancelRequested`
|
|
79
|
+
* flag (source of truth) plus the in-process `signal`. Absent in a bare test
|
|
80
|
+
* context — treat as "never cancelled."
|
|
81
|
+
*/
|
|
82
|
+
throwIfCancelled?(): void | Promise<void>;
|
|
83
|
+
}
|
|
84
|
+
/**
|
|
85
|
+
* One preview line — maps 1:1 onto a host plan item. A business record class
|
|
86
|
+
* (`'sales facts'`, `'journal entries'`), never a collection name.
|
|
87
|
+
*/
|
|
88
|
+
interface CleanupStepEstimate {
|
|
89
|
+
readonly resource: string;
|
|
90
|
+
/** Estimated records this step would affect. */
|
|
91
|
+
readonly estimated: number;
|
|
92
|
+
/** What this step RETAINS (e.g. `'measures kept, PII redacted'`). */
|
|
93
|
+
readonly retained?: string | undefined;
|
|
94
|
+
/**
|
|
95
|
+
* Domain blockers preventing this step (e.g. `'POSTED_BOOKS_IMMUTABLE'`,
|
|
96
|
+
* `'OPEN_TRANSFER'`). A non-empty list is a HARD STOP — the host refuses to
|
|
97
|
+
* execute until the operator resolves it. Blockers are DOMAIN facts, never a
|
|
98
|
+
* permission decision (that is the host's job).
|
|
99
|
+
*/
|
|
100
|
+
readonly blockers?: readonly string[] | undefined;
|
|
101
|
+
/**
|
|
102
|
+
* Non-blocking warnings surfaced to the operator (e.g. "rebuild may take
|
|
103
|
+
* several minutes on this generation").
|
|
104
|
+
*/
|
|
105
|
+
readonly warnings?: readonly string[] | undefined;
|
|
106
|
+
}
|
|
107
|
+
/** One executed step's outcome — maps onto a host step result. */
|
|
108
|
+
interface CleanupStepOutcome {
|
|
109
|
+
readonly resource: string;
|
|
110
|
+
/** Rows this step actually processed (deleted / redacted / rebuilt). */
|
|
111
|
+
readonly processed: number;
|
|
112
|
+
/**
|
|
113
|
+
* `false` iff the step failed. The host composer STOPS the recipe on the
|
|
114
|
+
* first `ok: false` (retention §8: never return success on a provider
|
|
115
|
+
* failure) and marks the run `failed`/`partial`. A step MUST NOT swallow a
|
|
116
|
+
* failure and report `ok: true`.
|
|
117
|
+
*/
|
|
118
|
+
readonly ok: boolean;
|
|
119
|
+
/** Failure message when `ok: false`. */
|
|
120
|
+
readonly error?: string | undefined;
|
|
121
|
+
/** Opaque resume cursor for observability. */
|
|
122
|
+
readonly cursor?: string | undefined;
|
|
123
|
+
}
|
|
124
|
+
/** One post-check — maps onto a host verification check (§9). */
|
|
125
|
+
interface CleanupStepCheck {
|
|
126
|
+
readonly name: string;
|
|
127
|
+
readonly ok: boolean;
|
|
128
|
+
readonly detail?: string | undefined;
|
|
129
|
+
}
|
|
130
|
+
/**
|
|
131
|
+
* A cleanup PROVIDER STEP — the unit a domain kernel exports for data it owns
|
|
132
|
+
* (retention design §6.6). Composed by the host into an ordered recipe.
|
|
133
|
+
*
|
|
134
|
+
* Idempotency + chunking are the step's responsibility: `execute` must be safe
|
|
135
|
+
* to re-run after a crash/retry (keyset progression, dedupe-by-occurrence, or a
|
|
136
|
+
* naturally-idempotent rebuild), and must observe `signal` / `throwIfCancelled`
|
|
137
|
+
* between chunks so a cancel lands between committed batches, never mid-write.
|
|
138
|
+
*/
|
|
139
|
+
interface CleanupStep {
|
|
140
|
+
/** Stable machine id, unique within a recipe. */
|
|
141
|
+
readonly id: string;
|
|
142
|
+
/** Logical resource label for the preview (e.g. `'sales facts'`). */
|
|
143
|
+
readonly resource: string;
|
|
144
|
+
/**
|
|
145
|
+
* `true` if the step deletes/redacts data; `false` for a pure rebuild
|
|
146
|
+
* (§4.3 — rebuilding a projection is not destructive to source data). A
|
|
147
|
+
* recipe is destructive iff ANY of its steps is.
|
|
148
|
+
*/
|
|
149
|
+
readonly destructive: boolean;
|
|
150
|
+
/**
|
|
151
|
+
* Projection / scaffolding rebuilds this step performs AFTER its cleanup —
|
|
152
|
+
* surfaced in the preview's `rebuildActions` (e.g. `'rebuild sales rollup'`).
|
|
153
|
+
*/
|
|
154
|
+
readonly rebuildActions?: readonly string[] | undefined;
|
|
155
|
+
/** Preview WITHOUT mutating. Idempotent + side-effect-free. */
|
|
156
|
+
estimate(ctx: CleanupStepContext): Promise<CleanupStepEstimate>;
|
|
157
|
+
/** Do the work. Chunked, idempotent, cancellation-aware. */
|
|
158
|
+
execute(ctx: CleanupStepExecuteContext): Promise<CleanupStepOutcome>;
|
|
159
|
+
/**
|
|
160
|
+
* Post-checks — the step owns what "clean" means for its data (§9: a delete
|
|
161
|
+
* count alone is never success). Optional: a pure-rebuild step may skip it,
|
|
162
|
+
* though verifying the watermark/counts is strongly encouraged.
|
|
163
|
+
*/
|
|
164
|
+
verify?(ctx: CleanupStepContext): Promise<readonly CleanupStepCheck[]>;
|
|
165
|
+
}
|
|
166
|
+
//#endregion
|
|
167
|
+
export { CleanupStep, CleanupStepCheck, CleanupStepContext, CleanupStepEstimate, CleanupStepExecuteContext, CleanupStepLogger, CleanupStepOutcome, CleanupStepProgress };
|
package/dist/filter/index.d.mts
CHANGED
|
@@ -3,6 +3,7 @@ import { FALSE, TRUE, and, anyOf as in_, between, contains, endsWith, eq, exists
|
|
|
3
3
|
import { recordToFilter } from "./from-record.mjs";
|
|
4
4
|
import { isFilter } from "./guard.mjs";
|
|
5
5
|
import { asPredicate, matchFilter } from "./match.mjs";
|
|
6
|
+
import { matchesRecordFilter, policyRecordToFilter } from "./match-record.mjs";
|
|
6
7
|
import { SCOPE_ANY, buildTenantScope, mergeScope } from "./scope.mjs";
|
|
7
8
|
import { collectFields, mapFilter, walkFilter } from "./walk.mjs";
|
|
8
|
-
export { FALSE, type Filter, type FilterAnd, type FilterEq, type FilterExists, type FilterFalse, type FilterGt, type FilterGte, type FilterIn, type FilterLike, type FilterLt, type FilterLte, type FilterNe, type FilterNin, type FilterNot, type FilterOp, type FilterOr, type FilterRaw, type FilterRegex, type FilterTrue, SCOPE_ANY, TRUE, and, in_ as anyOf, asPredicate, between, buildTenantScope, collectFields, contains, endsWith, eq, exists, gt, gte, iEq, in_, not as invert, isFilter, isNotNull, isNull, like, lt, lte, mapFilter, matchFilter, mergeScope, ne, nin, nin as noneOf, not, or, raw, recordToFilter, regex, startsWith, walkFilter };
|
|
9
|
+
export { FALSE, type Filter, type FilterAnd, type FilterEq, type FilterExists, type FilterFalse, type FilterGt, type FilterGte, type FilterIn, type FilterLike, type FilterLt, type FilterLte, type FilterNe, type FilterNin, type FilterNot, type FilterOp, type FilterOr, type FilterRaw, type FilterRegex, type FilterTrue, SCOPE_ANY, TRUE, and, in_ as anyOf, asPredicate, between, buildTenantScope, collectFields, contains, endsWith, eq, exists, gt, gte, iEq, in_, not as invert, isFilter, isNotNull, isNull, like, lt, lte, mapFilter, matchFilter, matchesRecordFilter, mergeScope, ne, nin, nin as noneOf, not, or, policyRecordToFilter, raw, recordToFilter, regex, startsWith, walkFilter };
|
package/dist/filter/index.mjs
CHANGED
|
@@ -3,5 +3,6 @@ import { collectFields, mapFilter, walkFilter } from "./walk.mjs";
|
|
|
3
3
|
import { FALSE, TRUE, and, anyOf as in_, between, contains, endsWith, eq, exists, gt, gte, iEq, invert as not, isNotNull, isNull, like, lt, lte, ne, nin, or, raw, regex, startsWith } from "./builders.mjs";
|
|
4
4
|
import { recordToFilter } from "./from-record.mjs";
|
|
5
5
|
import { asPredicate, matchFilter } from "./match.mjs";
|
|
6
|
+
import { matchesRecordFilter, policyRecordToFilter } from "./match-record.mjs";
|
|
6
7
|
import { SCOPE_ANY, buildTenantScope, mergeScope } from "./scope.mjs";
|
|
7
|
-
export { FALSE, SCOPE_ANY, TRUE, and, in_ as anyOf, asPredicate, between, buildTenantScope, collectFields, contains, endsWith, eq, exists, gt, gte, iEq, in_, not as invert, isFilter, isNotNull, isNull, like, lt, lte, mapFilter, matchFilter, mergeScope, ne, nin, nin as noneOf, not, or, raw, recordToFilter, regex, startsWith, walkFilter };
|
|
8
|
+
export { FALSE, SCOPE_ANY, TRUE, and, in_ as anyOf, asPredicate, between, buildTenantScope, collectFields, contains, endsWith, eq, exists, gt, gte, iEq, in_, not as invert, isFilter, isNotNull, isNull, like, lt, lte, mapFilter, matchFilter, matchesRecordFilter, mergeScope, ne, nin, nin as noneOf, not, or, policyRecordToFilter, raw, recordToFilter, regex, startsWith, walkFilter };
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
import { Filter } from "./types.mjs";
|
|
2
|
+
//#region src/filter/match-record.d.ts
|
|
3
|
+
/**
|
|
4
|
+
* Convert an arc Mongo-record `_policyFilters` object into Filter IR.
|
|
5
|
+
* `{}` → `TRUE`. Throws on unsupported top-level operators.
|
|
6
|
+
*/
|
|
7
|
+
declare function policyRecordToFilter(record: Record<string, unknown>): Filter;
|
|
8
|
+
/**
|
|
9
|
+
* Evaluate an arc Mongo-record `_policyFilters` object against a document —
|
|
10
|
+
* converts to Filter IR, then delegates to the shared {@link matchFilter}
|
|
11
|
+
* engine. THE canonical `DataAdapter.matchesFilter` implementation; every
|
|
12
|
+
* kit's adapter delegates here.
|
|
13
|
+
*
|
|
14
|
+
* @param item The already-fetched document / row.
|
|
15
|
+
* @param filters Arc's `_policyFilters` in Mongo record syntax.
|
|
16
|
+
*/
|
|
17
|
+
declare function matchesRecordFilter(item: unknown, filters: Record<string, unknown>): boolean;
|
|
18
|
+
//#endregion
|
|
19
|
+
export { matchesRecordFilter, policyRecordToFilter };
|
|
@@ -0,0 +1,187 @@
|
|
|
1
|
+
import { FALSE, TRUE, and, anyOf as in_, eq, gt, gte, invert as not, isNotNull, isNull, lt, lte, ne, or, regex } from "./builders.mjs";
|
|
2
|
+
import { matchFilter } from "./match.mjs";
|
|
3
|
+
//#region src/filter/match-record.ts
|
|
4
|
+
/**
|
|
5
|
+
* Mongo-record `_policyFilters` → Filter IR conversion + in-memory match.
|
|
6
|
+
*
|
|
7
|
+
* The CANONICAL, single home for evaluating arc's row-level policy filters
|
|
8
|
+
* against an already-fetched document IN PROCESS (no DB round-trip). Every
|
|
9
|
+
* kit's `DataAdapter.matchesFilter` delegates here — one contract, one IR,
|
|
10
|
+
* no per-kit matcher.
|
|
11
|
+
*
|
|
12
|
+
* Arc's permission helpers emit policy filters in Mongo record syntax,
|
|
13
|
+
* kit-agnostic (`requireOwnership` → `{ ownerId }`, multiTenant →
|
|
14
|
+
* `{ organizationId }`, `requireGrant` list resolutions →
|
|
15
|
+
* `{ $or: [{ ownerId }, { _id: { $in } }] }`). This module converts that
|
|
16
|
+
* record into the portable {@link Filter} IR and evaluates it with the
|
|
17
|
+
* shared {@link matchFilter} engine — the SAME IR kits compile to SQL /
|
|
18
|
+
* Mongo, so in-memory and DB-level enforcement agree by construction.
|
|
19
|
+
*
|
|
20
|
+
* `matchFilter` is id-coercion aware (Mongo `ObjectId` `_id` matches its
|
|
21
|
+
* string form — no kit-specific coercion) and array-aware (dot-paths fan
|
|
22
|
+
* out over subdocument arrays; scalar conditions on array fields match any
|
|
23
|
+
* element). See `match.ts`.
|
|
24
|
+
*
|
|
25
|
+
* SCOPE — the operators arc's policy filters emit. Fails LOUD on anything
|
|
26
|
+
* else so a silent mismatch never masquerades as a denial:
|
|
27
|
+
*
|
|
28
|
+
* logical: $or, $and, $nor, $not
|
|
29
|
+
* comparison: implicit-eq, $eq, $ne, $gt, $gte, $lt, $lte
|
|
30
|
+
* membership: $in, $nin
|
|
31
|
+
* existence: $exists (see divergence note below)
|
|
32
|
+
* pattern: $regex (+ $options; RegExp literal accepted)
|
|
33
|
+
*
|
|
34
|
+
* MongoDB parity (validated against the MongoDB manual + sift/mingo):
|
|
35
|
+
* - Missing field ≡ null for `{field: null}`, `$ne`/`$nin`, and a `null`
|
|
36
|
+
* member of `$in`/`$nin` — the authorization-critical rule (a policy
|
|
37
|
+
* filter `{ status: { $ne: 'archived' } }` MUST return docs lacking the
|
|
38
|
+
* field, exactly as MongoDB does).
|
|
39
|
+
* - `$in` accepts RegExp-literal members (Mongo allows `/re/` in `$in`).
|
|
40
|
+
* - Comparison ops are TYPE-BRACKETED: no cross-type ordering
|
|
41
|
+
* (`{ n: { $gt: 5 } }` never matches a string `n`); `$gt: null` matches
|
|
42
|
+
* nothing. NaN equals NaN for `$eq`.
|
|
43
|
+
* - Dot-paths fan out over arrays AND resolve numeric segments as
|
|
44
|
+
* positional indices (`items.0.sku`).
|
|
45
|
+
*
|
|
46
|
+
* DELIBERATE divergences (documented, not bugs):
|
|
47
|
+
* - `$exists` = present-AND-non-null (a null value reads as absent),
|
|
48
|
+
* matching the shared IR `exists` op + SQL `IS NOT NULL` + sift.js.
|
|
49
|
+
* MongoDB/mingo treat present-null as existing; that would require a
|
|
50
|
+
* separate key-presence IR op threaded through every kit's SQL/Mongo
|
|
51
|
+
* compiler. Arc's built-in policy helpers never emit `$exists`; a
|
|
52
|
+
* custom filter that needs Mongo key-presence should use
|
|
53
|
+
* `{ field: { $ne: null } }` (present + non-null) or `{ field: null }`
|
|
54
|
+
* (null OR missing) instead.
|
|
55
|
+
* - `$gt`/`$lt` allow ONE cross-type leniency: a `Date` field compares
|
|
56
|
+
* against an ISO-string operand (JSON policy filters carry dates as
|
|
57
|
+
* strings). Consistent with `$eq`'s Date⇄string coercion.
|
|
58
|
+
* - An array-literal operand (`{ tags: ['a','b'] }`) is element-matched,
|
|
59
|
+
* not exact-array-matched — policy filters never assert whole-array
|
|
60
|
+
* equality.
|
|
61
|
+
*
|
|
62
|
+
* Distinct from {@link recordToFilter}, which is the ergonomic
|
|
63
|
+
* record→IR normalizer for BARE-operator query shorthand (`{ price:
|
|
64
|
+
* { gte } }`) and deliberately does NOT accept `$`-prefixed operators or
|
|
65
|
+
* logical `$or`/`$and`. This function is the arc-policy-filter dialect
|
|
66
|
+
* (`$`-prefixed, with logical operators).
|
|
67
|
+
*/
|
|
68
|
+
/** Field operators understood inside a `{ field: { ... } }` condition. */
|
|
69
|
+
const FIELD_OPS = [
|
|
70
|
+
"$eq",
|
|
71
|
+
"$ne",
|
|
72
|
+
"$gt",
|
|
73
|
+
"$gte",
|
|
74
|
+
"$lt",
|
|
75
|
+
"$lte",
|
|
76
|
+
"$in",
|
|
77
|
+
"$nin",
|
|
78
|
+
"$exists",
|
|
79
|
+
"$regex"
|
|
80
|
+
];
|
|
81
|
+
function isOperatorObject(value) {
|
|
82
|
+
if (value === null || typeof value !== "object") return false;
|
|
83
|
+
if (Array.isArray(value) || value instanceof Date) return false;
|
|
84
|
+
const keys = Object.keys(value);
|
|
85
|
+
return keys.length > 0 && keys.every((k) => k.startsWith("$"));
|
|
86
|
+
}
|
|
87
|
+
/**
|
|
88
|
+
* `$in` with MongoDB parity: a `null` member also matches a MISSING field
|
|
89
|
+
* (inherits `{field: null}` semantics), and RegExp-literal members match by
|
|
90
|
+
* pattern (Mongo allows `/re/` inside `$in`). Split members into
|
|
91
|
+
* null / regex / scalar and OR the branches.
|
|
92
|
+
*/
|
|
93
|
+
function buildIn(field, members) {
|
|
94
|
+
const branches = [];
|
|
95
|
+
const scalars = [];
|
|
96
|
+
let hasNull = false;
|
|
97
|
+
for (const m of members) if (m === null || m === void 0) hasNull = true;
|
|
98
|
+
else if (m instanceof RegExp) branches.push(regex(field, m.source, m.flags));
|
|
99
|
+
else scalars.push(m);
|
|
100
|
+
if (hasNull) branches.push(isNull(field));
|
|
101
|
+
if (scalars.length > 0) branches.push(in_(field, scalars));
|
|
102
|
+
if (branches.length === 0) return FALSE;
|
|
103
|
+
return branches.length === 1 ? branches[0] : or(...branches);
|
|
104
|
+
}
|
|
105
|
+
/** `$nin` is the negation of `$in` — none of the members may match. */
|
|
106
|
+
function buildNin(field, members) {
|
|
107
|
+
const inFilter = buildIn(field, members);
|
|
108
|
+
return inFilter.op === "false" ? TRUE : not(inFilter);
|
|
109
|
+
}
|
|
110
|
+
/** Convert a single `{ field: condition }` entry into a Filter IR node. */
|
|
111
|
+
function fieldFilter(field, condition) {
|
|
112
|
+
if (!isOperatorObject(condition)) return condition === null ? isNull(field) : eq(field, condition);
|
|
113
|
+
const parts = [];
|
|
114
|
+
const options = typeof condition["$options"] === "string" ? condition["$options"] : void 0;
|
|
115
|
+
for (const [op, operand] of Object.entries(condition)) switch (op) {
|
|
116
|
+
case "$options": break;
|
|
117
|
+
case "$eq":
|
|
118
|
+
parts.push(operand === null ? isNull(field) : eq(field, operand));
|
|
119
|
+
break;
|
|
120
|
+
case "$ne":
|
|
121
|
+
parts.push(operand === null ? isNotNull(field) : ne(field, operand));
|
|
122
|
+
break;
|
|
123
|
+
case "$gt":
|
|
124
|
+
parts.push(gt(field, operand));
|
|
125
|
+
break;
|
|
126
|
+
case "$gte":
|
|
127
|
+
parts.push(gte(field, operand));
|
|
128
|
+
break;
|
|
129
|
+
case "$lt":
|
|
130
|
+
parts.push(lt(field, operand));
|
|
131
|
+
break;
|
|
132
|
+
case "$lte":
|
|
133
|
+
parts.push(lte(field, operand));
|
|
134
|
+
break;
|
|
135
|
+
case "$in":
|
|
136
|
+
parts.push(buildIn(field, operand ?? []));
|
|
137
|
+
break;
|
|
138
|
+
case "$nin":
|
|
139
|
+
parts.push(buildNin(field, operand ?? []));
|
|
140
|
+
break;
|
|
141
|
+
case "$exists":
|
|
142
|
+
parts.push(operand ? isNotNull(field) : isNull(field));
|
|
143
|
+
break;
|
|
144
|
+
case "$regex": {
|
|
145
|
+
const pattern = operand instanceof RegExp ? operand.source : String(operand);
|
|
146
|
+
const flags = operand instanceof RegExp ? operand.flags : options;
|
|
147
|
+
parts.push(flags ? regex(field, pattern, flags) : regex(field, pattern));
|
|
148
|
+
break;
|
|
149
|
+
}
|
|
150
|
+
default: throw new Error(`[repo-core] matchesRecordFilter: unsupported field operator '${op}'. Supported: ${FIELD_OPS.join(", ")}.`);
|
|
151
|
+
}
|
|
152
|
+
return parts.length === 1 ? parts[0] : and(...parts);
|
|
153
|
+
}
|
|
154
|
+
/**
|
|
155
|
+
* Convert an arc Mongo-record `_policyFilters` object into Filter IR.
|
|
156
|
+
* `{}` → `TRUE`. Throws on unsupported top-level operators.
|
|
157
|
+
*/
|
|
158
|
+
function policyRecordToFilter(record) {
|
|
159
|
+
const parts = [];
|
|
160
|
+
for (const [key, value] of Object.entries(record)) if (key === "$or") parts.push(or(...asFilterArray(value)));
|
|
161
|
+
else if (key === "$and") parts.push(and(...asFilterArray(value)));
|
|
162
|
+
else if (key === "$nor") parts.push(not(or(...asFilterArray(value))));
|
|
163
|
+
else if (key === "$not") parts.push(not(policyRecordToFilter(value)));
|
|
164
|
+
else if (key.startsWith("$")) throw new Error(`[repo-core] matchesRecordFilter: unsupported top-level operator '${key}'. Supported: $and, $or, $nor, $not.`);
|
|
165
|
+
else parts.push(fieldFilter(key, value));
|
|
166
|
+
if (parts.length === 0) return TRUE;
|
|
167
|
+
return parts.length === 1 ? parts[0] : and(...parts);
|
|
168
|
+
}
|
|
169
|
+
function asFilterArray(value) {
|
|
170
|
+
if (!Array.isArray(value)) throw new Error("[repo-core] matchesRecordFilter: $or/$and/$nor operand must be an array");
|
|
171
|
+
return value.map((entry) => policyRecordToFilter(entry));
|
|
172
|
+
}
|
|
173
|
+
/**
|
|
174
|
+
* Evaluate an arc Mongo-record `_policyFilters` object against a document —
|
|
175
|
+
* converts to Filter IR, then delegates to the shared {@link matchFilter}
|
|
176
|
+
* engine. THE canonical `DataAdapter.matchesFilter` implementation; every
|
|
177
|
+
* kit's adapter delegates here.
|
|
178
|
+
*
|
|
179
|
+
* @param item The already-fetched document / row.
|
|
180
|
+
* @param filters Arc's `_policyFilters` in Mongo record syntax.
|
|
181
|
+
*/
|
|
182
|
+
function matchesRecordFilter(item, filters) {
|
|
183
|
+
if (item === null || typeof item !== "object") return false;
|
|
184
|
+
return matchFilter(item, policyRecordToFilter(filters));
|
|
185
|
+
}
|
|
186
|
+
//#endregion
|
|
187
|
+
export { matchesRecordFilter, policyRecordToFilter };
|
package/dist/filter/match.mjs
CHANGED
|
@@ -7,35 +7,26 @@ function matchFilter(doc, filter) {
|
|
|
7
7
|
case "and": return filter.children.every((child) => matchFilter(doc, child));
|
|
8
8
|
case "or": return filter.children.some((child) => matchFilter(doc, child));
|
|
9
9
|
case "not": return !matchFilter(doc, filter.child);
|
|
10
|
-
case "eq": return
|
|
11
|
-
case "ne": return !
|
|
12
|
-
case "gt": return
|
|
13
|
-
case "gte": return
|
|
14
|
-
case "lt": return
|
|
15
|
-
case "lte": return
|
|
16
|
-
case "in":
|
|
17
|
-
|
|
18
|
-
return filter.values.some((candidate) => equals(v, candidate));
|
|
19
|
-
}
|
|
20
|
-
case "nin": {
|
|
21
|
-
const v = getField(doc, filter.field);
|
|
22
|
-
return !filter.values.some((candidate) => equals(v, candidate));
|
|
23
|
-
}
|
|
10
|
+
case "eq": return someValue(resolve(doc, filter.field), (v) => equals(v, filter.value));
|
|
11
|
+
case "ne": return !someValue(resolve(doc, filter.field), (v) => equals(v, filter.value));
|
|
12
|
+
case "gt": return someValue(resolve(doc, filter.field), (v) => compare(v, filter.value) > 0);
|
|
13
|
+
case "gte": return someValue(resolve(doc, filter.field), (v) => compare(v, filter.value) >= 0);
|
|
14
|
+
case "lt": return someValue(resolve(doc, filter.field), (v) => compare(v, filter.value) < 0);
|
|
15
|
+
case "lte": return someValue(resolve(doc, filter.field), (v) => compare(v, filter.value) <= 0);
|
|
16
|
+
case "in": return someValue(resolve(doc, filter.field), (v) => filter.values.some((candidate) => equals(v, candidate)));
|
|
17
|
+
case "nin": return !someValue(resolve(doc, filter.field), (v) => filter.values.some((candidate) => equals(v, candidate)));
|
|
24
18
|
case "exists": {
|
|
25
|
-
const
|
|
26
|
-
const present = v !== void 0 && v !== null;
|
|
19
|
+
const present = fieldPresent(doc, filter.field);
|
|
27
20
|
return filter.exists ? present : !present;
|
|
28
21
|
}
|
|
29
22
|
case "like": {
|
|
30
|
-
const v = getField(doc, filter.field);
|
|
31
|
-
if (typeof v !== "string") return false;
|
|
32
23
|
const flags = filter.caseSensitivity === "sensitive" ? "" : "i";
|
|
33
|
-
|
|
24
|
+
const re = getOrCompileLike(filter.pattern, flags);
|
|
25
|
+
return someValue(resolve(doc, filter.field), (v) => regexTest(re, v));
|
|
34
26
|
}
|
|
35
27
|
case "regex": {
|
|
36
|
-
const
|
|
37
|
-
|
|
38
|
-
return getOrCompileRegex(filter.pattern, filter.flags).test(v);
|
|
28
|
+
const re = getOrCompileRegex(filter.pattern, filter.flags);
|
|
29
|
+
return someValue(resolve(doc, filter.field), (v) => regexTest(re, v));
|
|
39
30
|
}
|
|
40
31
|
case "raw": return false;
|
|
41
32
|
}
|
|
@@ -47,27 +38,92 @@ function matchFilter(doc, filter) {
|
|
|
47
38
|
function asPredicate(filter) {
|
|
48
39
|
return (doc) => matchFilter(doc, filter);
|
|
49
40
|
}
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
41
|
+
/**
|
|
42
|
+
* Resolve a dot-path to the SET of values it reaches, fanning out over
|
|
43
|
+
* arrays on intermediate segments (Mongo/JSON-path array semantics):
|
|
44
|
+
* `items.sku` on `{ items: [{ sku: 1 }, { sku: 2 }] }` → `[1, 2]`. A path
|
|
45
|
+
* with no array yields a single-element list, so scalar leaf ops behave
|
|
46
|
+
* exactly as before. A leaf array field (`tags`) is returned as one value
|
|
47
|
+
* (the array) so `equals`'s array-contains handles it.
|
|
48
|
+
*/
|
|
49
|
+
/**
|
|
50
|
+
* Path segments that must never be resolved — reading them can surface an
|
|
51
|
+
* inherited member (or, for a JSON-parsed doc where `JSON.parse('{"__proto__":
|
|
52
|
+
* …}')` created a real OWN `__proto__`, a crafted value) and produce a WRONG
|
|
53
|
+
* authorization answer. Denied string-normalized (the object-path CVE-2021-23434
|
|
54
|
+
* lesson: an array-typed segment bypassed a `===` check). Fail closed: any
|
|
55
|
+
* path touching one of these resolves to no values → no match.
|
|
56
|
+
*/
|
|
57
|
+
const DANGEROUS_SEGMENTS = /* @__PURE__ */ new Set([
|
|
58
|
+
"__proto__",
|
|
59
|
+
"constructor",
|
|
60
|
+
"prototype"
|
|
61
|
+
]);
|
|
62
|
+
function resolve(doc, path) {
|
|
63
|
+
let frontier = [doc];
|
|
64
|
+
for (const segment of path.split(".")) {
|
|
65
|
+
if (DANGEROUS_SEGMENTS.has(String(segment))) return [];
|
|
66
|
+
const next = [];
|
|
67
|
+
for (const node of frontier) {
|
|
68
|
+
if (node === null || node === void 0 || typeof node !== "object") continue;
|
|
69
|
+
if (Array.isArray(node)) {
|
|
70
|
+
const idx = Number(segment);
|
|
71
|
+
if (Number.isInteger(idx) && idx >= 0 && idx < node.length) next.push(node[idx]);
|
|
72
|
+
for (const el of node) if (el && typeof el === "object" && Object.hasOwn(el, segment)) next.push(el[segment]);
|
|
73
|
+
} else if (Object.hasOwn(node, segment)) next.push(node[segment]);
|
|
74
|
+
}
|
|
75
|
+
if (next.length === 0) return [];
|
|
76
|
+
frontier = next;
|
|
58
77
|
}
|
|
59
|
-
return
|
|
78
|
+
return frontier;
|
|
79
|
+
}
|
|
80
|
+
/** Is any value reachable at `path` present (defined + non-null)? */
|
|
81
|
+
function fieldPresent(doc, path) {
|
|
82
|
+
return resolve(doc, path).some((v) => v !== void 0 && v !== null);
|
|
83
|
+
}
|
|
84
|
+
/**
|
|
85
|
+
* Apply a scalar predicate to a resolved value SET, unwrapping leaf array
|
|
86
|
+
* values so a scalar condition on an array field (`tags`, `scores`)
|
|
87
|
+
* matches when ANY element satisfies it — Mongo + SQL array semantics,
|
|
88
|
+
* concentrated in ONE place so `equals`/`compare`/regex stay pure scalar.
|
|
89
|
+
*/
|
|
90
|
+
function someValue(values, pred) {
|
|
91
|
+
for (const v of values) if (Array.isArray(v)) {
|
|
92
|
+
if (v.some(pred)) return true;
|
|
93
|
+
} else if (pred(v)) return true;
|
|
94
|
+
return false;
|
|
60
95
|
}
|
|
61
96
|
function equals(a, b) {
|
|
97
|
+
if (a === b) return true;
|
|
98
|
+
if (typeof a === "number" && typeof b === "number") return Number.isNaN(a) && Number.isNaN(b);
|
|
62
99
|
if (a instanceof Date && b instanceof Date) return a.getTime() === b.getTime();
|
|
63
100
|
if (a instanceof Date && typeof b === "string") return a.toISOString() === b;
|
|
64
101
|
if (b instanceof Date && typeof a === "string") return b.toISOString() === a;
|
|
65
|
-
|
|
102
|
+
const as = idString(a);
|
|
103
|
+
const bs = idString(b);
|
|
104
|
+
if (as !== void 0 && bs !== void 0) return as === bs;
|
|
105
|
+
return false;
|
|
106
|
+
}
|
|
107
|
+
/** String form of an id-like value for coercing comparison; else undefined. */
|
|
108
|
+
function idString(value) {
|
|
109
|
+
if (typeof value === "string") return value;
|
|
110
|
+
if (value === null || value === void 0 || typeof value !== "object") return void 0;
|
|
111
|
+
if (Array.isArray(value)) return void 0;
|
|
112
|
+
const s = String(value);
|
|
113
|
+
return s === "[object Object]" ? void 0 : s;
|
|
66
114
|
}
|
|
67
115
|
function compare(a, b) {
|
|
116
|
+
if (a instanceof Date && typeof b === "string") {
|
|
117
|
+
const t = Date.parse(b);
|
|
118
|
+
if (!Number.isNaN(t)) b = new Date(t);
|
|
119
|
+
} else if (b instanceof Date && typeof a === "string") {
|
|
120
|
+
const t = Date.parse(a);
|
|
121
|
+
if (!Number.isNaN(t)) a = new Date(t);
|
|
122
|
+
}
|
|
68
123
|
const aNum = toComparable(a);
|
|
69
124
|
const bNum = toComparable(b);
|
|
70
125
|
if (aNum === void 0 || bNum === void 0) return NaN;
|
|
126
|
+
if (typeof aNum !== typeof bNum) return NaN;
|
|
71
127
|
if (aNum < bNum) return -1;
|
|
72
128
|
if (aNum > bNum) return 1;
|
|
73
129
|
return 0;
|
|
@@ -114,6 +170,21 @@ function getOrCompileRegex(pattern, flags) {
|
|
|
114
170
|
regexCache.set(key, re);
|
|
115
171
|
return re;
|
|
116
172
|
}
|
|
173
|
+
/**
|
|
174
|
+
* Max string length fed to a regex `.test()`. ReDoS is `pattern × input`;
|
|
175
|
+
* even a benign developer-written pattern can go quadratic on a pathological
|
|
176
|
+
* INPUT string — and in a realtime fan-out the matcher runs once per
|
|
177
|
+
* subscriber per record, so one slow match blocks the event loop and
|
|
178
|
+
* amplifies across the whole subscriber set. A field value longer than this
|
|
179
|
+
* is treated as NO MATCH (fail closed) rather than risking a stall; policy
|
|
180
|
+
* filters never regex-test megabyte fields. (Trusted-source patterns make
|
|
181
|
+
* pattern-side ReDoS a non-issue; this caps the input side.)
|
|
182
|
+
*/
|
|
183
|
+
const MAX_REGEX_INPUT = 64 * 1024;
|
|
184
|
+
/** Guarded regex test: string-only, input-length-capped (see MAX_REGEX_INPUT). */
|
|
185
|
+
function regexTest(re, v) {
|
|
186
|
+
return typeof v === "string" && v.length <= MAX_REGEX_INPUT && re.test(v);
|
|
187
|
+
}
|
|
117
188
|
/** SQL `LIKE` pattern → JS regex body. Escapes regex metachars; `%` → `.*`, `_` → `.`. */
|
|
118
189
|
function likeToRegex(pattern) {
|
|
119
190
|
let out = "";
|
|
@@ -126,6 +126,15 @@ interface RepoCapabilities {
|
|
|
126
126
|
* tenant cleanup primitive.
|
|
127
127
|
*/
|
|
128
128
|
purgeByField?: boolean;
|
|
129
|
+
/**
|
|
130
|
+
* `purgeByFilter(filter, strategy, options)` — range/filter-scoped
|
|
131
|
+
* variant of `purgeByField`. Processes rows matching an arbitrary
|
|
132
|
+
* compiled filter (a `civilDate` window, a retention cutoff, a compound
|
|
133
|
+
* cohort) rather than a single `field = value` equality — the GDPR /
|
|
134
|
+
* retention "anonymize a slice across a RANGE while retaining measures"
|
|
135
|
+
* primitive.
|
|
136
|
+
*/
|
|
137
|
+
purgeByFilter?: boolean;
|
|
129
138
|
/**
|
|
130
139
|
* `archiveByFilter(filter, sink, options)` — chunked cold-storage
|
|
131
140
|
* extraction (write-before-delete, at-least-once). The data-lifecycle
|
|
@@ -12,11 +12,30 @@ type WritingPurgeStrategy = Exclude<TenantPurgeStrategy, {
|
|
|
12
12
|
* Driver-facing port the orchestrator drives. Each kit implements one
|
|
13
13
|
* closure over its driver primitives + the purge predicate.
|
|
14
14
|
*
|
|
15
|
+
* **Bound-predicate forms.** The port binds its selection predicate
|
|
16
|
+
* internally — the orchestrator never sees it. Two factory shapes both
|
|
17
|
+
* satisfy this single interface:
|
|
18
|
+
*
|
|
19
|
+
* - **Equality-bound** (`purgeByField`): the predicate is
|
|
20
|
+
* `{ [field]: value }` — the classic tenant-scoped cleanup where an
|
|
21
|
+
* organization id equals the deleted tenant.
|
|
22
|
+
* - **Filter-bound** (`purgeByFilter`): the predicate is a compiled
|
|
23
|
+
* range/compound filter (e.g. a `civilDate` window `gte/lte`,
|
|
24
|
+
* `{ status: 'archived', createdAt: { $lt: cutoff } }`). This is the
|
|
25
|
+
* GDPR/retention "anonymize a PII dimension across a range while
|
|
26
|
+
* RETAINING measures" op — no native equality analogue. Mirrors how
|
|
27
|
+
* the archive port's factory (`createMongoArchivePort`) binds a
|
|
28
|
+
* pre-compiled Filter rather than a `field/value` pair.
|
|
29
|
+
*
|
|
30
|
+
* Because `runChunkedPurge` drives ANY `PurgePort`, both factories reuse
|
|
31
|
+
* the same orchestrator unchanged — only the bound base predicate differs.
|
|
32
|
+
*
|
|
15
33
|
* **Plugin-bypass invariant.** Implementations MUST bypass tenant
|
|
16
|
-
* scoping in plugin hooks — the caller's `field = value`
|
|
17
|
-
* the authoritative scope; a tenant-injecting
|
|
18
|
-
* wrong tenant. Pass `bypassTenant: true` on
|
|
19
|
-
* (which keeps audit / cache hooks active but
|
|
34
|
+
* scoping in plugin hooks — the caller's bound predicate (`field = value`
|
|
35
|
+
* OR the compiled filter) IS the authoritative scope; a tenant-injecting
|
|
36
|
+
* hook would narrow to the wrong tenant. Pass `bypassTenant: true` on
|
|
37
|
+
* inner Repository calls (which keeps audit / cache hooks active but
|
|
38
|
+
* disables tenant injection).
|
|
20
39
|
*
|
|
21
40
|
* **Throughput contract.** Implementations should issue the minimum
|
|
22
41
|
* number of round-trips a chunk requires:
|
|
@@ -38,6 +57,23 @@ interface PurgePort {
|
|
|
38
57
|
* Returning `0` signals "no more matching rows"; the orchestrator
|
|
39
58
|
* exits. Returning a partial batch (`< limit`) is also a terminal
|
|
40
59
|
* signal — saves one round-trip on the last chunk.
|
|
60
|
+
*
|
|
61
|
+
* **PROGRESSION CONTRACT (mandatory).** Successive calls MUST advance
|
|
62
|
+
* through the match set for EVERY strategy. `hard` advances naturally
|
|
63
|
+
* (deleted rows leave the predicate's match set) — but `soft` and
|
|
64
|
+
* `anonymize` mutate rows that usually STILL satisfy the base
|
|
65
|
+
* predicate, so a port that re-runs `find(filter).limit(n)` re-selects
|
|
66
|
+
* the same first chunk forever and the orchestrator never terminates.
|
|
67
|
+
* Implementations must use **stable keyset progression**: order by the
|
|
68
|
+
* primary key and keep an internal `pk > lastSeen` cursor across calls
|
|
69
|
+
* (advance it only after the chunk's write succeeds, so a retried
|
|
70
|
+
* chunk re-selects the same rows). Offsets (`skip`) are not acceptable
|
|
71
|
+
* — they shift under concurrent writes and re-scan the head.
|
|
72
|
+
*
|
|
73
|
+
* A port instance is single-run state: build a fresh port per
|
|
74
|
+
* `runChunkedPurge` invocation, never share one across runs.
|
|
75
|
+
* `runPurgeConformance` (from `@classytic/repo-core/testing`) proves a
|
|
76
|
+
* kit satisfies this contract with match sets larger than `batchSize`.
|
|
41
77
|
*/
|
|
42
78
|
purgeChunk(strategy: WritingPurgeStrategy, limit: number): Promise<number>;
|
|
43
79
|
}
|
|
@@ -1598,10 +1598,13 @@ interface StandardRepo<TDoc> extends MinimalRepo<TDoc> {
|
|
|
1598
1598
|
* - sqlite: `EXPLAIN QUERY PLAN SELECT … WHERE field = ?` shows
|
|
1599
1599
|
* `SEARCH … USING INDEX`, never `SCAN`.
|
|
1600
1600
|
*
|
|
1601
|
-
* **Idempotent.** Re-running with the same arguments is
|
|
1602
|
-
*
|
|
1603
|
-
*
|
|
1604
|
-
*
|
|
1601
|
+
* **Idempotent by outcome.** Re-running with the same arguments is
|
|
1602
|
+
* safe: `hard` rows are gone, and `soft`/`anonymize` rows converge to
|
|
1603
|
+
* the same terminal field values on a second pass (the rows generally
|
|
1604
|
+
* DO still match the predicate — which is exactly why ports must use
|
|
1605
|
+
* keyset progression, not re-selection, to advance WITHIN a run; see
|
|
1606
|
+
* the {@link PurgePort} progression contract). Crucial for
|
|
1607
|
+
* at-least-once cascade workers that may retry after partial failure.
|
|
1605
1608
|
*
|
|
1606
1609
|
* **Plugin composition.** Kits route the underlying chunked ops
|
|
1607
1610
|
* through their standard `before:deleteMany` / `before:updateMany`
|
|
@@ -1619,6 +1622,41 @@ interface StandardRepo<TDoc> extends MinimalRepo<TDoc> {
|
|
|
1619
1622
|
* @param options Chunking, session, progress, abort signal.
|
|
1620
1623
|
*/
|
|
1621
1624
|
purgeByField?(field: string, value: unknown, strategy: TenantPurgeStrategy, options?: TenantPurgeOptions): Promise<TenantPurgeResult>;
|
|
1625
|
+
/**
|
|
1626
|
+
* Range/filter-scoped variant of {@link purgeByField} — processes every
|
|
1627
|
+
* row matching an arbitrary `filter` (not just `field = value`) under the
|
|
1628
|
+
* given {@link TenantPurgeStrategy}. The compliance primitive for
|
|
1629
|
+
* "purge/anonymize a slice across a RANGE while retaining measures":
|
|
1630
|
+
* redact a PII dimension across a `civilDate` window, hard-delete rows
|
|
1631
|
+
* older than a retention cutoff, soft-delete a compound-predicate cohort.
|
|
1632
|
+
*
|
|
1633
|
+
* `purgeByField(field, value, ...)` is the equality special case
|
|
1634
|
+
* (`{ [field]: value }`); this method takes the full portable
|
|
1635
|
+
* {@link FilterInput} (Filter IR or a plain kit-native record), compiled
|
|
1636
|
+
* once by the kit before the chunk loop — the same dual-dialect rule
|
|
1637
|
+
* every other verb follows.
|
|
1638
|
+
*
|
|
1639
|
+
* Everything else is identical to `purgeByField`:
|
|
1640
|
+
* - Strategy → kit-native primitive (`hard`/`soft`/`anonymize`/`skip`).
|
|
1641
|
+
* - **Chunking mandatory** — implementations MUST honor `batchSize`.
|
|
1642
|
+
* - **Index requirement** — the filter's leading field(s) MUST be indexed
|
|
1643
|
+
* or every chunk re-scans the collection / table.
|
|
1644
|
+
* - **Idempotent** — re-running with the same arguments is safe; already
|
|
1645
|
+
* purged rows simply don't match the next pass.
|
|
1646
|
+
* - **Plugin composition** — the chunked ops route through the kit's
|
|
1647
|
+
* `before:deleteMany` / `before:updateMany` hooks so audit /
|
|
1648
|
+
* cache-invalidation / observability plugins fire naturally.
|
|
1649
|
+
* - **Narrowed-write re-assertion** — kits re-assert the base filter on
|
|
1650
|
+
* the `{ _id: { $in: ids }, ...filter }` write, defending against a row
|
|
1651
|
+
* that left the matching set between the id select and the write.
|
|
1652
|
+
*
|
|
1653
|
+
* Optional method; gate on `capabilities.purgeByFilter`.
|
|
1654
|
+
*
|
|
1655
|
+
* @param filter Predicate selecting rows to purge (Filter IR or record).
|
|
1656
|
+
* @param strategy Strategy declaration — see {@link TenantPurgeStrategy}.
|
|
1657
|
+
* @param options Chunking, session, progress, abort signal.
|
|
1658
|
+
*/
|
|
1659
|
+
purgeByFilter?(filter: FilterInput, strategy: TenantPurgeStrategy, options?: TenantPurgeOptions): Promise<TenantPurgeResult>;
|
|
1622
1660
|
/**
|
|
1623
1661
|
* Chunked cold-storage extraction — move every row matching `filter`
|
|
1624
1662
|
* into a host-provided {@link ArchiveSink}, then remove it from the hot
|
package/dist/testing/index.d.mts
CHANGED
|
@@ -3,4 +3,5 @@ import { ConformanceContext, ConformanceDoc, ConformanceFeatures, ConformanceHar
|
|
|
3
3
|
import { runStandardRepoConformance } from "./conformance.mjs";
|
|
4
4
|
import { LockConformanceHarness, runLockAdapterConformance } from "./lock-conformance.mjs";
|
|
5
5
|
import { UsageConformanceHarness, runUsageStoreContract } from "./usage-conformance.mjs";
|
|
6
|
-
|
|
6
|
+
import { PurgeConformanceContext, PurgeConformanceHarness, runPurgeConformance } from "./purge-conformance.mjs";
|
|
7
|
+
export { type AggregateOpsSupport, type ConformanceContext, type ConformanceDoc, type ConformanceFeatures, type ConformanceHarness, type LockConformanceHarness, type PurgeConformanceContext, type PurgeConformanceHarness, type UsageConformanceHarness, runLockAdapterConformance, runPurgeConformance, runStandardRepoConformance, runUsageStoreContract };
|
package/dist/testing/index.mjs
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { runStandardRepoConformance } from "./conformance.mjs";
|
|
2
2
|
import { runLockAdapterConformance } from "./lock-conformance.mjs";
|
|
3
3
|
import { runUsageStoreContract } from "./usage-conformance.mjs";
|
|
4
|
-
|
|
4
|
+
import { runPurgeConformance } from "./purge-conformance.mjs";
|
|
5
|
+
export { runLockAdapterConformance, runPurgeConformance, runStandardRepoConformance, runUsageStoreContract };
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
import { TenantPurgeOptions, TenantPurgeResult, TenantPurgeStrategy } from "../repository/types.mjs";
|
|
2
|
+
//#region src/testing/purge-conformance.d.ts
|
|
3
|
+
/** Everything the shared scenarios need from a kit. */
|
|
4
|
+
interface PurgeConformanceContext {
|
|
5
|
+
/**
|
|
6
|
+
* Seed `inScope` docs matching the purge scope and `outOfScope` docs
|
|
7
|
+
* outside it. Every in-scope doc must carry:
|
|
8
|
+
* - a string field `email` set to `'user-<i>@test.local'` (anonymize target);
|
|
9
|
+
* - a numeric field `amount` (a measure that must SURVIVE soft/anonymize).
|
|
10
|
+
*/
|
|
11
|
+
seed(inScope: number, outOfScope: number): Promise<void>;
|
|
12
|
+
/** Run the kit's chunked purge over the bound scope. */
|
|
13
|
+
purge(strategy: TenantPurgeStrategy, options?: TenantPurgeOptions): Promise<TenantPurgeResult>;
|
|
14
|
+
/** RAW physical count of in-scope rows — MUST bypass soft-delete query filters. */
|
|
15
|
+
countRaw(): Promise<number>;
|
|
16
|
+
/** RAW count of in-scope rows carrying the soft-deleted flag. */
|
|
17
|
+
countSoftFlagged(): Promise<number>;
|
|
18
|
+
/** RAW count of in-scope rows whose `email` equals `value`. */
|
|
19
|
+
countEmail(value: string): Promise<number>;
|
|
20
|
+
/** RAW sum of `amount` across in-scope rows (proves measures retained). */
|
|
21
|
+
sumAmount(): Promise<number>;
|
|
22
|
+
/** RAW physical count of OUT-of-scope rows — must never change. */
|
|
23
|
+
countOutOfScope(): Promise<number>;
|
|
24
|
+
}
|
|
25
|
+
interface PurgeConformanceHarness {
|
|
26
|
+
/** Kit name — the top-level describe() label. */
|
|
27
|
+
name: string;
|
|
28
|
+
/** Fresh isolated context per test (own collection/table). */
|
|
29
|
+
setup(): Promise<PurgeConformanceContext>;
|
|
30
|
+
}
|
|
31
|
+
declare function runPurgeConformance(harness: PurgeConformanceHarness): void;
|
|
32
|
+
//#endregion
|
|
33
|
+
export { PurgeConformanceContext, PurgeConformanceHarness, runPurgeConformance };
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
import { beforeEach, describe, expect, it } from "vitest";
|
|
2
|
+
//#region src/testing/purge-conformance.ts
|
|
3
|
+
/**
|
|
4
|
+
* `runPurgeConformance` — cross-kit chunked-purge contract suite.
|
|
5
|
+
*
|
|
6
|
+
* Proves a kit's `purgeByField`/`purgeByFilter` port makes **stable
|
|
7
|
+
* progress for EVERY strategy** when the match set exceeds `batchSize`
|
|
8
|
+
* — the exact property the naive "re-query the same predicate" port
|
|
9
|
+
* shape violates: `hard` self-advances (deleted rows leave the match
|
|
10
|
+
* set) but `soft`/`anonymize` re-select the same first chunk forever
|
|
11
|
+
* because the mutated rows still satisfy the base predicate.
|
|
12
|
+
*
|
|
13
|
+
* Every scenario seeds MORE rows than `batchSize`, so a port without
|
|
14
|
+
* keyset progression fails here instead of hanging production. A chunk
|
|
15
|
+
* budget (via `onProgress` + abort) converts the would-be infinite loop
|
|
16
|
+
* into a crisp assertion failure.
|
|
17
|
+
*
|
|
18
|
+
* ## Usage from a kit
|
|
19
|
+
*
|
|
20
|
+
* import { runPurgeConformance } from '@classytic/repo-core/testing';
|
|
21
|
+
*
|
|
22
|
+
* describe('mongokit purge conformance', () => {
|
|
23
|
+
* runPurgeConformance({
|
|
24
|
+
* name: 'mongokit',
|
|
25
|
+
* async setup() { …return a PurgeConformanceContext… },
|
|
26
|
+
* });
|
|
27
|
+
* });
|
|
28
|
+
*/
|
|
29
|
+
const IN_SCOPE = 25;
|
|
30
|
+
const OUT_SCOPE = 5;
|
|
31
|
+
const BATCH = 10;
|
|
32
|
+
const AMOUNT_EACH = 7;
|
|
33
|
+
/**
|
|
34
|
+
* Wrap TenantPurgeOptions with a chunk budget: when the port stops
|
|
35
|
+
* progressing, the loop would otherwise run forever — the budget aborts
|
|
36
|
+
* it after `limit` chunks so the suite fails with a readable assertion
|
|
37
|
+
* instead of a vitest timeout.
|
|
38
|
+
*/
|
|
39
|
+
function budgeted(limit, chunks = []) {
|
|
40
|
+
const controller = new AbortController();
|
|
41
|
+
return {
|
|
42
|
+
batchSize: BATCH,
|
|
43
|
+
signal: controller.signal,
|
|
44
|
+
onProgress({ chunkSize }) {
|
|
45
|
+
chunks.push(chunkSize);
|
|
46
|
+
if (chunks.length >= limit) controller.abort();
|
|
47
|
+
}
|
|
48
|
+
};
|
|
49
|
+
}
|
|
50
|
+
/** ceil(25/10) + 1 slack — a progressing port never needs more. */
|
|
51
|
+
const CHUNK_BUDGET = Math.ceil(IN_SCOPE / BATCH) + 1;
|
|
52
|
+
function runPurgeConformance(harness) {
|
|
53
|
+
describe(`${harness.name} — chunked purge conformance`, () => {
|
|
54
|
+
let ctx;
|
|
55
|
+
beforeEach(async () => {
|
|
56
|
+
ctx = await harness.setup();
|
|
57
|
+
});
|
|
58
|
+
it("hard: drains a multi-batch scope exactly once per row", async () => {
|
|
59
|
+
await ctx.seed(IN_SCOPE, OUT_SCOPE);
|
|
60
|
+
const chunks = [];
|
|
61
|
+
const res = await ctx.purge({ type: "hard" }, budgeted(CHUNK_BUDGET, chunks));
|
|
62
|
+
expect(res.ok).toBe(true);
|
|
63
|
+
expect(res.processed).toBe(IN_SCOPE);
|
|
64
|
+
expect(chunks).toEqual([
|
|
65
|
+
10,
|
|
66
|
+
10,
|
|
67
|
+
5
|
|
68
|
+
]);
|
|
69
|
+
expect(await ctx.countRaw()).toBe(0);
|
|
70
|
+
expect(await ctx.countOutOfScope()).toBe(OUT_SCOPE);
|
|
71
|
+
});
|
|
72
|
+
it("soft: progresses across batches WITHOUT caller-supplied exclusion predicates", async () => {
|
|
73
|
+
await ctx.seed(IN_SCOPE, OUT_SCOPE);
|
|
74
|
+
const chunks = [];
|
|
75
|
+
const res = await ctx.purge({ type: "soft" }, budgeted(CHUNK_BUDGET, chunks));
|
|
76
|
+
expect(res.ok).toBe(true);
|
|
77
|
+
expect(res.processed).toBe(IN_SCOPE);
|
|
78
|
+
expect(chunks).toEqual([
|
|
79
|
+
10,
|
|
80
|
+
10,
|
|
81
|
+
5
|
|
82
|
+
]);
|
|
83
|
+
expect(await ctx.countRaw()).toBe(IN_SCOPE);
|
|
84
|
+
expect(await ctx.countSoftFlagged()).toBe(IN_SCOPE);
|
|
85
|
+
expect(await ctx.sumAmount()).toBe(IN_SCOPE * AMOUNT_EACH);
|
|
86
|
+
expect(await ctx.countOutOfScope()).toBe(OUT_SCOPE);
|
|
87
|
+
});
|
|
88
|
+
it("anonymize (static): progresses across batches and retains measures", async () => {
|
|
89
|
+
await ctx.seed(IN_SCOPE, OUT_SCOPE);
|
|
90
|
+
const chunks = [];
|
|
91
|
+
const res = await ctx.purge({
|
|
92
|
+
type: "anonymize",
|
|
93
|
+
fields: { email: "redacted@example.invalid" }
|
|
94
|
+
}, budgeted(CHUNK_BUDGET, chunks));
|
|
95
|
+
expect(res.ok).toBe(true);
|
|
96
|
+
expect(res.processed).toBe(IN_SCOPE);
|
|
97
|
+
expect(chunks).toEqual([
|
|
98
|
+
10,
|
|
99
|
+
10,
|
|
100
|
+
5
|
|
101
|
+
]);
|
|
102
|
+
expect(await ctx.countEmail("redacted@example.invalid")).toBe(IN_SCOPE);
|
|
103
|
+
expect(await ctx.sumAmount()).toBe(IN_SCOPE * AMOUNT_EACH);
|
|
104
|
+
expect(await ctx.countRaw()).toBe(IN_SCOPE);
|
|
105
|
+
expect(await ctx.countOutOfScope()).toBe(OUT_SCOPE);
|
|
106
|
+
});
|
|
107
|
+
it("anonymize (function-form): progresses across batches", async () => {
|
|
108
|
+
await ctx.seed(IN_SCOPE, OUT_SCOPE);
|
|
109
|
+
const res = await ctx.purge({
|
|
110
|
+
type: "anonymize",
|
|
111
|
+
fields: { email: () => "fn-redacted@example.invalid" }
|
|
112
|
+
}, budgeted(CHUNK_BUDGET));
|
|
113
|
+
expect(res.ok).toBe(true);
|
|
114
|
+
expect(res.processed).toBe(IN_SCOPE);
|
|
115
|
+
expect(await ctx.countEmail("fn-redacted@example.invalid")).toBe(IN_SCOPE);
|
|
116
|
+
expect(await ctx.sumAmount()).toBe(IN_SCOPE * AMOUNT_EACH);
|
|
117
|
+
});
|
|
118
|
+
it("exact-batch boundary: inScope === batchSize processes each row once", async () => {
|
|
119
|
+
await ctx.seed(BATCH, OUT_SCOPE);
|
|
120
|
+
const res = await ctx.purge({ type: "soft" }, budgeted(CHUNK_BUDGET));
|
|
121
|
+
expect(res.ok).toBe(true);
|
|
122
|
+
expect(res.processed).toBe(BATCH);
|
|
123
|
+
expect(await ctx.countSoftFlagged()).toBe(BATCH);
|
|
124
|
+
});
|
|
125
|
+
it("skip: declared no-op reaches no rows", async () => {
|
|
126
|
+
await ctx.seed(3, 0);
|
|
127
|
+
const res = await ctx.purge({
|
|
128
|
+
type: "skip",
|
|
129
|
+
reason: "retention-owned"
|
|
130
|
+
});
|
|
131
|
+
expect(res.ok).toBe(true);
|
|
132
|
+
expect(res.processed).toBe(0);
|
|
133
|
+
expect(res.skipReason).toBe("retention-owned");
|
|
134
|
+
expect(await ctx.countRaw()).toBe(3);
|
|
135
|
+
});
|
|
136
|
+
it("empty scope: terminates immediately with zero processed", async () => {
|
|
137
|
+
await ctx.seed(0, OUT_SCOPE);
|
|
138
|
+
const res = await ctx.purge({ type: "hard" }, budgeted(CHUNK_BUDGET));
|
|
139
|
+
expect(res.ok).toBe(true);
|
|
140
|
+
expect(res.processed).toBe(0);
|
|
141
|
+
expect(await ctx.countOutOfScope()).toBe(OUT_SCOPE);
|
|
142
|
+
});
|
|
143
|
+
it("abort between chunks: committed chunks stay, result is ok:false", async () => {
|
|
144
|
+
await ctx.seed(IN_SCOPE, OUT_SCOPE);
|
|
145
|
+
const controller = new AbortController();
|
|
146
|
+
const res = await ctx.purge({ type: "hard" }, {
|
|
147
|
+
batchSize: BATCH,
|
|
148
|
+
signal: controller.signal,
|
|
149
|
+
onProgress() {
|
|
150
|
+
controller.abort();
|
|
151
|
+
}
|
|
152
|
+
});
|
|
153
|
+
expect(res.ok).toBe(false);
|
|
154
|
+
expect(res.processed).toBe(BATCH);
|
|
155
|
+
expect(await ctx.countRaw()).toBe(IN_SCOPE - BATCH);
|
|
156
|
+
});
|
|
157
|
+
});
|
|
158
|
+
}
|
|
159
|
+
//#endregion
|
|
160
|
+
export { runPurgeConformance };
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@classytic/repo-core",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.17.0",
|
|
4
4
|
"description": "Driver-agnostic repository primitives: hooks, Filter IR, operations, pagination, cache contract. Foundation for mongokit, sqlitekit, pgkit, and prismakit. Lean by design — no plugins ship here; each kit owns its own.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"sideEffects": false,
|
|
@@ -102,6 +102,10 @@
|
|
|
102
102
|
"./sync": {
|
|
103
103
|
"types": "./dist/sync/index.d.mts",
|
|
104
104
|
"default": "./dist/sync/index.mjs"
|
|
105
|
+
},
|
|
106
|
+
"./cleanup": {
|
|
107
|
+
"types": "./dist/cleanup/index.d.mts",
|
|
108
|
+
"default": "./dist/cleanup/index.mjs"
|
|
105
109
|
}
|
|
106
110
|
},
|
|
107
111
|
"keywords": [
|