@classytic/repo-core 0.27.1 → 0.28.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +84 -0
- package/dist/cache/plugin/context.mjs +3 -1
- package/dist/pagination/keyset.mjs +0 -2
- package/dist/pagination/types.d.mts +12 -0
- package/dist/repository/index.d.mts +2 -2
- package/dist/repository/index.mjs +2 -2
- package/dist/repository/resilience.d.mts +25 -20
- package/dist/repository/resilience.mjs +66 -2
- package/dist/repository/types.d.mts +99 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,90 @@ All notable changes to `@classytic/repo-core` are documented here.
|
|
|
4
4
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
5
5
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
6
|
|
|
7
|
+
## [0.28.0] - 2026-09-14
|
|
8
|
+
|
|
9
|
+
### Changed — BREAKING: `RetryPolicy.shouldRetry` no longer defaults to retrying everything
|
|
10
|
+
|
|
11
|
+
`shouldRetry` defaulted to `() => true`, with a comment conceding it was unsafe
|
|
12
|
+
and asking callers to pass their own. That is a documented hazard, not a fence:
|
|
13
|
+
a policy added for network blips also re-ran the write on a duplicate key, a
|
|
14
|
+
validation failure and a permission denial — three times, with backoff. Against
|
|
15
|
+
a slow-but-succeeding server and a non-idempotent write, that is a double write.
|
|
16
|
+
|
|
17
|
+
The default is now **`isTransientError`** (exported from `./repository`): retry
|
|
18
|
+
a failure that MAY not have happened — failover, write conflict, lock timeout,
|
|
19
|
+
socket error — never one the server answered deterministically. A 4xx-shaped
|
|
20
|
+
`statusCode` is treated as final, since `HttpError` carries one and every kit
|
|
21
|
+
maps validation and permission failures onto it.
|
|
22
|
+
|
|
23
|
+
**Migration:** a caller relying on retry-everything passes `shouldRetry: () => true`
|
|
24
|
+
explicitly. The point is that it is now a decision rather than an inheritance.
|
|
25
|
+
Tests exercising retry MECHANICS should pass a predicate too, so they stop
|
|
26
|
+
depending on how a generic `Error` happens to be classified.
|
|
27
|
+
|
|
28
|
+
### Added
|
|
29
|
+
|
|
30
|
+
- **`resumeToken` on `ChangeEvent`, `startAfter` on `WatchOptions`.** `watch()`
|
|
31
|
+
already accepted a resume token but a consumer had no way to obtain one, so
|
|
32
|
+
the documented at-least-once-across-restarts contract was unfulfillable.
|
|
33
|
+
Persist the last event's `resumeToken` and hand it back as `resumeAfter` (or
|
|
34
|
+
`startAfter`, which also resumes past an invalidate event) to continue from
|
|
35
|
+
that point. Additive — a kit whose feed is not resumable leaves the field
|
|
36
|
+
absent.
|
|
37
|
+
|
|
38
|
+
- **`AggRequest.countDistinctStrategy`** — `'accumulator'` (default) or
|
|
39
|
+
`'grouped'`. The default holds every distinct value of a group in that
|
|
40
|
+
group's accumulator, so memory grows with the data and MongoDB caps it at
|
|
41
|
+
100MB; `'grouped'` pre-groups by the distinct field so each value becomes a
|
|
42
|
+
row instead, and nothing accumulates. Results are identical — including that
|
|
43
|
+
a missing field is not a distinct value while an explicit `null` is. Kits
|
|
44
|
+
refuse the measure combinations they cannot recombine (`avg`, `percentile`,
|
|
45
|
+
`stddev`, a second `countDistinct`) rather than approximating them.
|
|
46
|
+
|
|
47
|
+
- **`prev` / `hasPrev` on `KeysetPaginationResultCore`** — the backward half of
|
|
48
|
+
cursor pagination, for kits that implement a `before` cursor (mongokit does,
|
|
49
|
+
>= 3.39.0). Both are OPTIONAL, so a kit that only pages forward is unchanged.
|
|
50
|
+
|
|
51
|
+
`prev === undefined` means *"this kit does not answer the question"*, NOT
|
|
52
|
+
*"there is no previous page"*. A UI must read `hasPrev` to decide whether to
|
|
53
|
+
offer the control and treat absent as unknown — reading `undefined` as `false`
|
|
54
|
+
hides a control that should be there.
|
|
55
|
+
|
|
56
|
+
### Changed
|
|
57
|
+
|
|
58
|
+
- **`validateKeysetSort` accepts mixed sort directions.** `{ priority: 1,
|
|
59
|
+
createdAt: -1 }` — the ESR-shaped compound index — used to throw. The keyset
|
|
60
|
+
predicate is a tuple comparison whose operator is chosen per position, so the
|
|
61
|
+
directions never needed to agree; rejecting them pushed callers onto offset
|
|
62
|
+
`skip(n)`, the cost keyset exists to avoid. `_id` now keeps the direction the
|
|
63
|
+
caller gave it, and still follows the primary field when absent.
|
|
64
|
+
|
|
65
|
+
**A kit's filter builder must read the direction PER FIELD** to consume this
|
|
66
|
+
safely. mongokit's `buildKeysetFilter` does (>= 3.39.0). A kit that derives
|
|
67
|
+
one operator from the primary field will now silently mis-paginate a
|
|
68
|
+
mixed-direction sort instead of being protected by the throw — audit yours
|
|
69
|
+
before upgrading.
|
|
70
|
+
|
|
71
|
+
### Fixed
|
|
72
|
+
|
|
73
|
+
- **A cached `getAll` page was served to a caller that asked for a different
|
|
74
|
+
count strategy.** `countStrategy` / `countLimit` were missing from
|
|
75
|
+
`DEFAULT_SHAPE_KEYS_BY_OP.getAll`, so every strategy shared one cache key —
|
|
76
|
+
and the strategies disagree about `total` BY DESIGN: `'none'` reports `0`,
|
|
77
|
+
`'capped'` reports a flagged ceiling, `'exact'` reports the truth. A list
|
|
78
|
+
asking for an exact count could therefore render `0 results` over a full page
|
|
79
|
+
of rows, with nothing reporting an error. Both keys are now on the allowlist.
|
|
80
|
+
|
|
81
|
+
`mode` remains off it deliberately — it selects which ENVELOPE comes back, and
|
|
82
|
+
offset vs keyset are already separated by `page` vs `after`.
|
|
83
|
+
|
|
84
|
+
**On deploy this invalidates existing `getAll` cache entries once** (the key
|
|
85
|
+
shape changed); the next read repopulates them.
|
|
86
|
+
|
|
87
|
+
Pinned by mongokit's `plugin-composition-security.test.ts`, whose
|
|
88
|
+
`countStrategy` case had been `it.skip`ped with a comment documenting the hole
|
|
89
|
+
rather than closing it.
|
|
90
|
+
|
|
7
91
|
## [0.27.1] - 2026-09-11
|
|
8
92
|
|
|
9
93
|
### Fixed
|
|
@@ -38,8 +38,6 @@ function validateKeysetSort(sort, allowedPrimaryFields) {
|
|
|
38
38
|
if (allowedPrimaryFields && allowedPrimaryFields.length > 0) {
|
|
39
39
|
for (const key of nonIdKeys) if (!allowedPrimaryFields.includes(key)) throw new Error(`Keyset sort field "${key}" is not in the strictKeysetSortFields allowlist. Allowed: ${allowedPrimaryFields.join(", ")}. (Protects against lossy null/non-null keyset boundaries.)`);
|
|
40
40
|
}
|
|
41
|
-
for (const key of nonIdKeys) if (sort[key] !== primaryDirection) throw new Error("All sort fields must share the same direction for keyset pagination");
|
|
42
|
-
if (keys.includes("_id") && sort["_id"] !== primaryDirection) throw new Error("_id direction must match primary field direction");
|
|
43
41
|
if (!keys.includes("_id")) return normalizeSort({
|
|
44
42
|
...sort,
|
|
45
43
|
_id: primaryDirection
|
|
@@ -124,6 +124,18 @@ interface KeysetPaginationResultCore<TDoc> {
|
|
|
124
124
|
hasMore: boolean;
|
|
125
125
|
/** Cursor token for the next page, or `null` when there is none. */
|
|
126
126
|
next: string | null;
|
|
127
|
+
/**
|
|
128
|
+
* Cursor token for the PREVIOUS page, or `null` when this is the first.
|
|
129
|
+
* Feed it back as `before` to walk backwards.
|
|
130
|
+
*
|
|
131
|
+
* Optional so a kit that only pages forward is unchanged — but `undefined`
|
|
132
|
+
* means "this kit does not answer the question", NOT "there is no previous
|
|
133
|
+
* page". A UI must read {@link hasPrev} to decide whether to offer the
|
|
134
|
+
* control, and treat absent as unknown rather than as `false`.
|
|
135
|
+
*/
|
|
136
|
+
prev?: string | null;
|
|
137
|
+
/** Whether a previous page exists. Absent when the kit does not track it. */
|
|
138
|
+
hasPrev?: boolean;
|
|
127
139
|
}
|
|
128
140
|
/**
|
|
129
141
|
* Keyset-paginated result envelope.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { LookupPopulateOptions, LookupPopulateResult, LookupRow, LookupSpec } from "../lookup/types.mjs";
|
|
2
2
|
import { UpdateInput } from "../update/types.mjs";
|
|
3
3
|
import { nestDottedKeys, nestDottedKeysAll } from "./agg-output.mjs";
|
|
4
|
-
import { RetryPolicy, throwIfAborted, withRetry } from "./resilience.mjs";
|
|
4
|
+
import { RetryPolicy, isTransientError, throwIfAborted, withRetry } from "./resilience.mjs";
|
|
5
5
|
import { ArchiveOptions, ArchivePort, ArchiveProgress, ArchiveResult, ArchiveSink, runChunkedArchive } from "./archive.mjs";
|
|
6
6
|
import { PLUGIN_ORDER_CONSTRAINTS, Plugin, PluginFunction, PluginType, validatePluginOrder } from "./plugin-types.mjs";
|
|
7
7
|
import { RepositoryBase, RepositoryBaseOptions } from "./base.mjs";
|
|
@@ -12,4 +12,4 @@ import { STANDARD_REPO_OPTION_KEYS, StandardRepoOptionKey } from "./options.mjs"
|
|
|
12
12
|
import { PurgePort, WritingPurgeStrategy, runChunkedPurge } from "./purge.mjs";
|
|
13
13
|
import { ReadOnlyRepoOptions, ReadOnlyRepositoryError, asReadOnlyRepo, isReadOnlyRepo } from "./read-only.mjs";
|
|
14
14
|
import { RetryingTransactionOptions, retryingTransaction } from "./retrying-transaction.mjs";
|
|
15
|
-
export { type AggCacheOptions, type AggDateBucket, type AggDateBucketInterval, type AggDateBucketUnit, type AggExecutionHints, type AggMeasure, type AggPaginationRequest, type AggRequest, type AggResult, type AggRow, type AggTopN, type AggTopNTies, type AggregateOpsSupport, type ArchiveOptions, type ArchivePort, type ArchiveProgress, type ArchiveResult, type ArchiveSink, type BulkCreateResult, type BulkWriteOperation, type BulkWriteResult, type ChangeEvent, type ClaimTransition, type ClaimVersionTransition, type CursorOptions, type DeleteManyResult, type DeleteOptions, type DeleteResult, type DistributionConfig, type DistributionMissHandler, type FilterInput, type FindAllOptions, type FindOneAndUpdateOptions, type InferDoc, type KeysetAggPaginationResult, type LookupPopulateOptions, type LookupPopulateResult, type LookupRow, type LookupSpec, type MinimalRepo, PLUGIN_ORDER_CONSTRAINTS, type PaginationParams, type Plugin, type PluginFunction, type PluginType, type PurgePort, type QueryOptions, type ReadOnlyRepoOptions, ReadOnlyRepositoryError, type RepoCapabilities, RepositoryBase, type RepositoryBaseOptions, type RepositorySession, type RetryPolicy, type RetryingTransactionOptions, STANDARD_REPO_OPTION_KEYS, type StandardRepo, type StandardRepoOptionKey, type TenantPurgeOptions, type TenantPurgeProgress, type TenantPurgeResult, type TenantPurgeStrategy, type TransactionHandle, type TransitionArgs, type TransitionMachine, type UpdateInput, type UpdateManyResult, type WatchOptions, type WriteOptions, type WritingPurgeStrategy, asReadOnlyRepo, createDistributionGuard, filterReferencesKey, isReadOnlyRepo, nestDottedKeys, nestDottedKeysAll, retryingTransaction, runChunkedArchive, runChunkedPurge, throwIfAborted, validatePluginOrder, withRetry };
|
|
15
|
+
export { type AggCacheOptions, type AggDateBucket, type AggDateBucketInterval, type AggDateBucketUnit, type AggExecutionHints, type AggMeasure, type AggPaginationRequest, type AggRequest, type AggResult, type AggRow, type AggTopN, type AggTopNTies, type AggregateOpsSupport, type ArchiveOptions, type ArchivePort, type ArchiveProgress, type ArchiveResult, type ArchiveSink, type BulkCreateResult, type BulkWriteOperation, type BulkWriteResult, type ChangeEvent, type ClaimTransition, type ClaimVersionTransition, type CursorOptions, type DeleteManyResult, type DeleteOptions, type DeleteResult, type DistributionConfig, type DistributionMissHandler, type FilterInput, type FindAllOptions, type FindOneAndUpdateOptions, type InferDoc, type KeysetAggPaginationResult, type LookupPopulateOptions, type LookupPopulateResult, type LookupRow, type LookupSpec, type MinimalRepo, PLUGIN_ORDER_CONSTRAINTS, type PaginationParams, type Plugin, type PluginFunction, type PluginType, type PurgePort, type QueryOptions, type ReadOnlyRepoOptions, ReadOnlyRepositoryError, type RepoCapabilities, RepositoryBase, type RepositoryBaseOptions, type RepositorySession, type RetryPolicy, type RetryingTransactionOptions, STANDARD_REPO_OPTION_KEYS, type StandardRepo, type StandardRepoOptionKey, type TenantPurgeOptions, type TenantPurgeProgress, type TenantPurgeResult, type TenantPurgeStrategy, type TransactionHandle, type TransitionArgs, type TransitionMachine, type UpdateInput, type UpdateManyResult, type WatchOptions, type WriteOptions, type WritingPurgeStrategy, asReadOnlyRepo, createDistributionGuard, filterReferencesKey, isReadOnlyRepo, isTransientError, nestDottedKeys, nestDottedKeysAll, retryingTransaction, runChunkedArchive, runChunkedPurge, throwIfAborted, validatePluginOrder, withRetry };
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { nestDottedKeys, nestDottedKeysAll } from "./agg-output.mjs";
|
|
2
|
-
import { throwIfAborted, withRetry } from "./resilience.mjs";
|
|
2
|
+
import { isTransientError, throwIfAborted, withRetry } from "./resilience.mjs";
|
|
3
3
|
import { runChunkedArchive } from "./archive.mjs";
|
|
4
4
|
import { PLUGIN_ORDER_CONSTRAINTS, validatePluginOrder } from "./plugin-types.mjs";
|
|
5
5
|
import { RepositoryBase } from "./base.mjs";
|
|
@@ -8,4 +8,4 @@ import { STANDARD_REPO_OPTION_KEYS } from "./options.mjs";
|
|
|
8
8
|
import { runChunkedPurge } from "./purge.mjs";
|
|
9
9
|
import { ReadOnlyRepositoryError, asReadOnlyRepo, isReadOnlyRepo } from "./read-only.mjs";
|
|
10
10
|
import { retryingTransaction } from "./retrying-transaction.mjs";
|
|
11
|
-
export { PLUGIN_ORDER_CONSTRAINTS, ReadOnlyRepositoryError, RepositoryBase, STANDARD_REPO_OPTION_KEYS, asReadOnlyRepo, createDistributionGuard, filterReferencesKey, isReadOnlyRepo, nestDottedKeys, nestDottedKeysAll, retryingTransaction, runChunkedArchive, runChunkedPurge, throwIfAborted, validatePluginOrder, withRetry };
|
|
11
|
+
export { PLUGIN_ORDER_CONSTRAINTS, ReadOnlyRepositoryError, RepositoryBase, STANDARD_REPO_OPTION_KEYS, asReadOnlyRepo, createDistributionGuard, filterReferencesKey, isReadOnlyRepo, isTransientError, nestDottedKeys, nestDottedKeysAll, retryingTransaction, runChunkedArchive, runChunkedPurge, throwIfAborted, validatePluginOrder, withRetry };
|
|
@@ -9,24 +9,26 @@
|
|
|
9
9
|
* never drifts between call sites.
|
|
10
10
|
*/
|
|
11
11
|
/**
|
|
12
|
-
*
|
|
13
|
-
*
|
|
12
|
+
* The default `shouldRetry`: retry a failure that may not have happened, never
|
|
13
|
+
* one that definitely did.
|
|
14
14
|
*
|
|
15
|
-
*
|
|
16
|
-
*
|
|
17
|
-
*
|
|
18
|
-
*
|
|
19
|
-
*
|
|
15
|
+
* This used to be `() => true`, with a comment conceding it was unsafe and
|
|
16
|
+
* asking callers to pass their own. That is a documented hazard rather than a
|
|
17
|
+
* fence — a policy set for network blips also re-ran the write on a duplicate
|
|
18
|
+
* key, a validation failure and a permission denial, three times, with backoff.
|
|
19
|
+
* For a non-idempotent write that is not a slow failure, it is a double write.
|
|
20
20
|
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
*
|
|
24
|
-
*
|
|
25
|
-
*
|
|
26
|
-
*
|
|
27
|
-
*
|
|
28
|
-
*
|
|
21
|
+
* Retried: replica-set failover, write conflicts, lock timeouts, socket errors
|
|
22
|
+
* — states where the outcome is genuinely unknown.
|
|
23
|
+
*
|
|
24
|
+
* NOT retried: anything the server answered deterministically. A 4xx-shaped
|
|
25
|
+
* `statusCode` is treated as final, since repo-core's own `HttpError` carries
|
|
26
|
+
* one and every kit maps validation/permission failures onto it.
|
|
27
|
+
*
|
|
28
|
+
* Pass `shouldRetry: () => true` to restore the old behaviour explicitly — the
|
|
29
|
+
* point is that it is now a choice someone made, not a default they inherited.
|
|
29
30
|
*/
|
|
31
|
+
declare function isTransientError(err: unknown): boolean;
|
|
30
32
|
interface RetryPolicy {
|
|
31
33
|
/** Max attempts (including the first try). Default 3 when a policy is present. */
|
|
32
34
|
maxAttempts?: number;
|
|
@@ -42,10 +44,13 @@ interface RetryPolicy {
|
|
|
42
44
|
*/
|
|
43
45
|
jitter?: boolean;
|
|
44
46
|
/**
|
|
45
|
-
* Decide whether a given error is transient.
|
|
46
|
-
*
|
|
47
|
-
*
|
|
48
|
-
*
|
|
47
|
+
* Decide whether a given error is transient.
|
|
48
|
+
*
|
|
49
|
+
* Defaults to {@link isTransientError}: retry a failure that MAY not have
|
|
50
|
+
* happened (failover, write conflict, socket error), never one that
|
|
51
|
+
* definitely did (duplicate key, validation, permission). Pass
|
|
52
|
+
* `() => true` to retry everything — allowed, but as a decision rather
|
|
53
|
+
* than an inheritance.
|
|
49
54
|
*/
|
|
50
55
|
shouldRetry?: (err: unknown, attempt: number) => boolean;
|
|
51
56
|
}
|
|
@@ -66,4 +71,4 @@ declare function withRetry<T>(fn: () => Promise<T>, policy: RetryPolicy | undefi
|
|
|
66
71
|
*/
|
|
67
72
|
declare function throwIfAborted(signal: AbortSignal | undefined): void;
|
|
68
73
|
//#endregion
|
|
69
|
-
export { RetryPolicy, throwIfAborted, withRetry };
|
|
74
|
+
export { RetryPolicy, isTransientError, throwIfAborted, withRetry };
|
|
@@ -1,5 +1,69 @@
|
|
|
1
1
|
//#region src/repository/resilience.ts
|
|
2
2
|
/**
|
|
3
|
+
* Resilience primitives — the single retry/abort contract every kit and
|
|
4
|
+
* every chunked orchestrator (purge, batch imports, outbox relays) uses.
|
|
5
|
+
*
|
|
6
|
+
* One `RetryPolicy` shape across the contract: `QueryOptions.retryPolicy`,
|
|
7
|
+
* `TenantPurgeOptions.retry`, and any kit-internal retry loop all accept
|
|
8
|
+
* the same three knobs. One `withRetry` implementation so backoff math
|
|
9
|
+
* never drifts between call sites.
|
|
10
|
+
*/
|
|
11
|
+
/**
|
|
12
|
+
* Retry policy for transient failures (network blips, write conflicts,
|
|
13
|
+
* busy-locks, connection resets).
|
|
14
|
+
*
|
|
15
|
+
* **Don't retry blindly.** Validation errors, schema errors, permission
|
|
16
|
+
* errors are NOT transient — retrying just delays the same failure.
|
|
17
|
+
* Mongo `WriteConflict`, SQLite `SQLITE_BUSY`, `ECONNRESET` ARE transient
|
|
18
|
+
* — backoff + retry recovers. Pass `shouldRetry` to narrow when you know
|
|
19
|
+
* your driver's error taxonomy:
|
|
20
|
+
*
|
|
21
|
+
* ```ts
|
|
22
|
+
* retryPolicy: {
|
|
23
|
+
* maxAttempts: 3, // default 3 when block present
|
|
24
|
+
* baseDelayMs: 100, // exponential: 100ms, 200ms, 400ms
|
|
25
|
+
* shouldRetry: (err) =>
|
|
26
|
+
* /WriteConflict|SQLITE_BUSY|ECONNRESET/i.test(String(err)),
|
|
27
|
+
* }
|
|
28
|
+
* ```
|
|
29
|
+
*/
|
|
30
|
+
/**
|
|
31
|
+
* Error labels and names that mean "the operation may not have run — running
|
|
32
|
+
* it again is safe". Duck-typed so `repo-core` stays driver-free: MongoDB sets
|
|
33
|
+
* these labels, and other drivers surface the same idea under these names.
|
|
34
|
+
*/
|
|
35
|
+
const TRANSIENT_LABELS = ["RetryableWriteError", "TransientTransactionError"];
|
|
36
|
+
const TRANSIENT_PATTERN = /WriteConflict|LockTimeout|NotWritablePrimary|NotPrimary|PrimarySteppedDown|InterruptedDueToReplStateChange|ShutdownInProgress|HostUnreachable|HostNotFound|NetworkTimeout|SocketException|ECONNRESET|ECONNREFUSED|ETIMEDOUT|EPIPE|SQLITE_BUSY|SQLITE_LOCKED|MongoNetworkError/i;
|
|
37
|
+
/**
|
|
38
|
+
* The default `shouldRetry`: retry a failure that may not have happened, never
|
|
39
|
+
* one that definitely did.
|
|
40
|
+
*
|
|
41
|
+
* This used to be `() => true`, with a comment conceding it was unsafe and
|
|
42
|
+
* asking callers to pass their own. That is a documented hazard rather than a
|
|
43
|
+
* fence — a policy set for network blips also re-ran the write on a duplicate
|
|
44
|
+
* key, a validation failure and a permission denial, three times, with backoff.
|
|
45
|
+
* For a non-idempotent write that is not a slow failure, it is a double write.
|
|
46
|
+
*
|
|
47
|
+
* Retried: replica-set failover, write conflicts, lock timeouts, socket errors
|
|
48
|
+
* — states where the outcome is genuinely unknown.
|
|
49
|
+
*
|
|
50
|
+
* NOT retried: anything the server answered deterministically. A 4xx-shaped
|
|
51
|
+
* `statusCode` is treated as final, since repo-core's own `HttpError` carries
|
|
52
|
+
* one and every kit maps validation/permission failures onto it.
|
|
53
|
+
*
|
|
54
|
+
* Pass `shouldRetry: () => true` to restore the old behaviour explicitly — the
|
|
55
|
+
* point is that it is now a choice someone made, not a default they inherited.
|
|
56
|
+
*/
|
|
57
|
+
function isTransientError(err) {
|
|
58
|
+
if (err === null || typeof err !== "object") return false;
|
|
59
|
+
const e = err;
|
|
60
|
+
if (typeof e.hasErrorLabel === "function") for (const label of TRANSIENT_LABELS) try {
|
|
61
|
+
if (e.hasErrorLabel(label)) return true;
|
|
62
|
+
} catch {}
|
|
63
|
+
if (typeof e.statusCode === "number" && e.statusCode >= 400 && e.statusCode < 500) return false;
|
|
64
|
+
return TRANSIENT_PATTERN.test(`${String(e.name ?? "")} ${String(e.codeName ?? "")} ${String(e.code ?? "")} ${String(e.message ?? "")}`);
|
|
65
|
+
}
|
|
66
|
+
/**
|
|
3
67
|
* Abortable sleep. A plain `setTimeout` promise holds the process through an
|
|
4
68
|
* abort — a cancelled request would silently wait out its full backoff before
|
|
5
69
|
* noticing. Listener is removed on the timer path so repeated retries don't
|
|
@@ -31,7 +95,7 @@ async function withRetry(fn, policy, signal) {
|
|
|
31
95
|
if (!policy) return fn();
|
|
32
96
|
const maxAttempts = policy.maxAttempts ?? 3;
|
|
33
97
|
const baseDelayMs = policy.baseDelayMs ?? 100;
|
|
34
|
-
const shouldRetry = policy.shouldRetry ??
|
|
98
|
+
const shouldRetry = policy.shouldRetry ?? isTransientError;
|
|
35
99
|
let lastErr;
|
|
36
100
|
for (let attempt = 0; attempt < maxAttempts; attempt++) {
|
|
37
101
|
signal?.throwIfAborted();
|
|
@@ -59,4 +123,4 @@ function throwIfAborted(signal) {
|
|
|
59
123
|
signal?.throwIfAborted();
|
|
60
124
|
}
|
|
61
125
|
//#endregion
|
|
62
|
-
export { throwIfAborted, withRetry };
|
|
126
|
+
export { isTransientError, throwIfAborted, withRetry };
|
|
@@ -623,7 +623,37 @@ type AggMeasure = {
|
|
|
623
623
|
op: 'count';
|
|
624
624
|
field?: string;
|
|
625
625
|
where?: FilterInput;
|
|
626
|
-
} |
|
|
626
|
+
} |
|
|
627
|
+
/**
|
|
628
|
+
* Exact distinct count. **Bounded by group memory — see the ceiling below
|
|
629
|
+
* before using it on a high-cardinality field.**
|
|
630
|
+
*
|
|
631
|
+
* It is computed by accumulating the distinct values themselves and taking
|
|
632
|
+
* the size, so every distinct value in a group is held in that group's
|
|
633
|
+
* accumulator at once. Cheap for `countDistinct` over a status or a region;
|
|
634
|
+
* expensive over a user id, an email, or a session token, where the
|
|
635
|
+
* accumulator grows with the data rather than with the answer.
|
|
636
|
+
*
|
|
637
|
+
* MongoDB caps a single group accumulator at 100MB. Past that the pipeline
|
|
638
|
+
* fails — loudly, which is the right outcome, but it fails at the size the
|
|
639
|
+
* data reaches rather than at the size anyone planned for.
|
|
640
|
+
*
|
|
641
|
+
* Mitigations, in the order worth trying:
|
|
642
|
+
* 1. narrow the groups (`where`, a tighter `groupBy`) so each accumulator
|
|
643
|
+
* holds less;
|
|
644
|
+
* 2. pass `allowDiskUse` so the stage may spill instead of failing — it
|
|
645
|
+
* makes the query slow rather than impossible, and does NOT make every
|
|
646
|
+
* accumulator shape safe;
|
|
647
|
+
* 3. for genuinely unbounded cardinality, count distinct in its own
|
|
648
|
+
* request grouped by `(group keys, the distinct field)` and sum the
|
|
649
|
+
* rows — one pass more, no accumulator growth.
|
|
650
|
+
*
|
|
651
|
+
* There is deliberately no approximate mode: a distinct count that is
|
|
652
|
+
* quietly within a few percent is exactly the plausible-looking wrong number
|
|
653
|
+
* this contract exists to avoid. If an estimate is acceptable, the caller
|
|
654
|
+
* should ask for one explicitly rather than inherit it.
|
|
655
|
+
*/
|
|
656
|
+
{
|
|
627
657
|
op: 'countDistinct';
|
|
628
658
|
field: string;
|
|
629
659
|
where?: FilterInput;
|
|
@@ -1084,6 +1114,32 @@ interface AggRequest {
|
|
|
1084
1114
|
* each partition).
|
|
1085
1115
|
*/
|
|
1086
1116
|
topN?: AggTopN;
|
|
1117
|
+
/**
|
|
1118
|
+
* How `countDistinct` is computed.
|
|
1119
|
+
*
|
|
1120
|
+
* - **`'accumulator'`** (default) — collect the distinct values, take the
|
|
1121
|
+
* size. One pass, and the cheapest thing there is while a group's distinct
|
|
1122
|
+
* values fit in memory. They are held in that group's accumulator at once,
|
|
1123
|
+
* so the cost grows with the DATA, not with the answer: fine over a status
|
|
1124
|
+
* or a region, and a 100MB accumulator failure over a user id or an email.
|
|
1125
|
+
*
|
|
1126
|
+
* - **`'grouped'`** — pre-group by the distinct field so each distinct value
|
|
1127
|
+
* becomes a row, then count rows. One extra `$group`, and nothing
|
|
1128
|
+
* accumulates: cardinality stops being a memory question. Slower on small
|
|
1129
|
+
* groups, which is why it is not the default.
|
|
1130
|
+
*
|
|
1131
|
+
* Results are IDENTICAL either way, including the parts that are easy to get
|
|
1132
|
+
* wrong — a missing field is not a distinct value, an explicit `null` is, and
|
|
1133
|
+
* `where` excludes rows from the count without excluding them from the other
|
|
1134
|
+
* measures. `tests/integration/countdistinct-strategies.test.ts` asserts the
|
|
1135
|
+
* two strategies agree row for row rather than asserting expected numbers,
|
|
1136
|
+
* because a strategy that is merely self-consistent is the failure mode.
|
|
1137
|
+
*
|
|
1138
|
+
* `'grouped'` REFUSES a measure set it cannot re-aggregate (`avg`,
|
|
1139
|
+
* `percentile`, `stddev`, or a second `countDistinct`) instead of returning a
|
|
1140
|
+
* plausible wrong number — see the error for the alternatives.
|
|
1141
|
+
*/
|
|
1142
|
+
countDistinctStrategy?: 'accumulator' | 'grouped';
|
|
1087
1143
|
/**
|
|
1088
1144
|
* Driver-tunable performance knobs (allowDiskUse, maxTimeMs,
|
|
1089
1145
|
* indexHint). See `AggExecutionHints` for the per-kit support
|
|
@@ -1879,6 +1935,14 @@ interface ChangeEvent<TDoc = unknown> {
|
|
|
1879
1935
|
doc?: TDoc;
|
|
1880
1936
|
/** Commit timestamp as reported by the backend. */
|
|
1881
1937
|
timestamp: Date;
|
|
1938
|
+
/**
|
|
1939
|
+
* Resume token for THIS event (kit-specific opaque shape — Mongo's
|
|
1940
|
+
* change-document `_id`). Persist the last one you finished processing
|
|
1941
|
+
* and hand it back as `WatchOptions.resumeAfter` / `startAfter` to
|
|
1942
|
+
* continue from that point across restarts. Absent on backends whose
|
|
1943
|
+
* feed is not resumable.
|
|
1944
|
+
*/
|
|
1945
|
+
resumeToken?: unknown;
|
|
1882
1946
|
}
|
|
1883
1947
|
/** Options for `StandardRepo.watch()`. */
|
|
1884
1948
|
interface WatchOptions {
|
|
@@ -1890,6 +1954,40 @@ interface WatchOptions {
|
|
|
1890
1954
|
* for at-least-once consumption across restarts.
|
|
1891
1955
|
*/
|
|
1892
1956
|
resumeAfter?: unknown;
|
|
1957
|
+
/**
|
|
1958
|
+
* Like `resumeAfter`, but also resumes past an invalidate event (Mongo
|
|
1959
|
+
* `startAfter` — the token of the invalidate itself). Mutually exclusive
|
|
1960
|
+
* with `resumeAfter`; kits reject a call that sets both.
|
|
1961
|
+
*/
|
|
1962
|
+
startAfter?: unknown;
|
|
1963
|
+
/**
|
|
1964
|
+
* Ceiling on events held between the stream and a slower consumer.
|
|
1965
|
+
* Default 1024. `0` disables the bound; any other non-positive-integer
|
|
1966
|
+
* (negative, fractional, `Infinity`, `NaN`) is REFUSED rather than treated
|
|
1967
|
+
* as unbounded — silently removing the ceiling is the failure this option
|
|
1968
|
+
* exists to prevent.
|
|
1969
|
+
*
|
|
1970
|
+
* A change feed has no backpressure: the server pushes, and a kit bridging
|
|
1971
|
+
* it into an async iterator buffers whatever the consumer has not reached.
|
|
1972
|
+
* An unbounded buffer therefore turns a consumer that is merely SLOWER than
|
|
1973
|
+
* the write rate into unbounded memory growth — and the symptom is an OOM
|
|
1974
|
+
* far from the cause.
|
|
1975
|
+
*
|
|
1976
|
+
* Overflow ends the iterator with an error carrying `resumeToken` — the last
|
|
1977
|
+
* event the consumer ACKNOWLEDGED, meaning the last one it came back from.
|
|
1978
|
+
* Deliberately not the last DELIVERED event: resuming past one the consumer
|
|
1979
|
+
* was still processing drops it. The error also carries `deliveredToken` for
|
|
1980
|
+
* diagnostics; resuming from that reintroduces the loss.
|
|
1981
|
+
*
|
|
1982
|
+
* It never drops events either way. A change feed silently missing a delete
|
|
1983
|
+
* is the kind of wrong that surfaces as a reconciliation mystery months
|
|
1984
|
+
* later, so the asymmetry is chosen on purpose: redelivering costs a
|
|
1985
|
+
* duplicate, skipping costs a lost write.
|
|
1986
|
+
*
|
|
1987
|
+
* A change stream is an INGESTION mechanism, not a durable queue. Consumers
|
|
1988
|
+
* that must not miss events persist the resume token and stay idempotent.
|
|
1989
|
+
*/
|
|
1990
|
+
maxBufferedEvents?: number;
|
|
1893
1991
|
}
|
|
1894
1992
|
/**
|
|
1895
1993
|
* Options for `StandardRepo.cursor()` streaming reads. The open index
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@classytic/repo-core",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.28.0",
|
|
4
4
|
"description": "Driver-agnostic repository primitives: hooks, Filter IR, operations, pagination, cache contract. Foundation for mongokit, sqlitekit, pgkit, and prismakit. Lean by design — no plugins ship here; each kit owns its own.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"sideEffects": false,
|