@cloudbitmaps/roaring 0.25.0 → 0.27.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/PRIVACY.md +15 -9
- package/README.md +2 -2
- package/dist/bin/export-segments.js +175 -17
- package/dist/bin/export-segments.js.map +3 -3
- package/dist/index.d.ts +80 -11
- package/dist/index.js +153 -11
- package/dist/index.js.map +3 -3
- package/package.json +2 -2
package/dist/index.d.ts
CHANGED
|
@@ -163,7 +163,13 @@ export interface CacheOptions {
|
|
|
163
163
|
* lapses, a read of it fails with the fault, as a cold read does. A refresh that fails with anything else, an access denial or a row that will not parse, is not
|
|
164
164
|
* ridden out: the read that meets it throws that error, and the reader is dropped.
|
|
165
165
|
*
|
|
166
|
-
*
|
|
166
|
+
* **A pinned handle is checked on the same interval.** It never moves to another generation, but at most every
|
|
167
|
+
* `genTtlMs` (counted on access) its segment's row is read, one read shared by every pin of the segment, and the
|
|
168
|
+
* pin fails with `NotFoundError` once the segment is destroyed, gone or re-created, or its generation erased or
|
|
169
|
+
* collected, or recorded by its row as another object, even when its reader and chunks are still cached. A transient
|
|
170
|
+
* fault of the registry, or of the footer read an erasure's check makes, keeps it serving.
|
|
171
|
+
*
|
|
172
|
+
* `0` turns this timed refresh off, and the pinned handles' check with it, and so does wiring a bare `IStorageDriver`, which has no registry. That is
|
|
167
173
|
* all it does. The store still moves a segment on to whatever generation is current when its reader cache
|
|
168
174
|
* evicts the segment, when a read has to fetch from a generation a sweep deleted, and when it is invalidated,
|
|
169
175
|
* as its own `load`, `rollback`, `eraseSubject` and `*Into` writes do and {@link CloudRoaring.invalidate} does.
|
|
@@ -889,7 +895,7 @@ export declare class CloudRoaring {
|
|
|
889
895
|
* duration, or fix the source first and load after. A writer that lands *during* the rewrite is caught: the
|
|
890
896
|
* rewrite's publish is refused **by the fence** — the publish lands its compare-and-swap only
|
|
891
897
|
* while the pointer is still on the generation the rewrite streamed — and the entry says `note: 'superseded'`,
|
|
892
|
-
* so re-run. Forward-only alone would NOT refuse it: the rewrite's number goes above everything
|
|
898
|
+
* so re-run. Forward-only alone would NOT refuse it: the rewrite's number goes above everything listed near the row's own numbers,
|
|
893
899
|
* so the rewrite would out-rank the newer generation and then collect it.
|
|
894
900
|
*
|
|
895
901
|
* A racing **erasure** is caught before that, and reported the same way. It collects with `keep: 0`, taking
|
|
@@ -1057,7 +1063,8 @@ export declare class CloudRoaring {
|
|
|
1057
1063
|
* It deletes nothing. The generations above the new pointer stay put — which is what makes this reversible —
|
|
1058
1064
|
* and are then *above* `currentGen`, where collection never looks, and still known to the row, so rolling forward onto
|
|
1059
1065
|
* one needs no flag. They remain until loads pass them (the first load whose
|
|
1060
|
-
* number one of them holds numbers above them all,
|
|
1066
|
+
* number one of them holds numbers above them all, those within 1,024 of what the row names and any contiguous run
|
|
1067
|
+
* touching them, and collection then keeps `keep` of what is below its pointer, the
|
|
1061
1068
|
* generation that load superseded first), {@link CloudRoaring.dropSegment}
|
|
1062
1069
|
* deletes them, or {@link CloudRoaring.eraseSubject} does: all of those present when it rewrites, and, when the
|
|
1063
1070
|
* current generation does not hold the id, only those holding it and any no read of the segment can open, whatever
|
|
@@ -1366,7 +1373,8 @@ export declare class CloudRoaring {
|
|
|
1366
1373
|
* the unwrapped DEK, and decrypts with them, chunks it had never fetched before the shred included, until its
|
|
1367
1374
|
* snapshot re-resolves: within `cache.genTtlMs` when the store has a registry and a `genTtlMs` above 0, and
|
|
1368
1375
|
* otherwise only when its caches let the segment go or a read has to fetch from a generation a sweep collected.
|
|
1369
|
-
* A pin
|
|
1376
|
+
* A pin on a store with a timed refresh fails within `cache.genTtlMs` once its segment is destroyed or its
|
|
1377
|
+
* generation erased; on a store with none it answers until it is told or both caches let it go.
|
|
1370
1378
|
* - **Another process.** Erasing on one box invalidates nothing on the others; each store bounds its own
|
|
1371
1379
|
* staleness by `cache.genTtlMs`, and a store with no registry (a bare `IStorageDriver`), with
|
|
1372
1380
|
* `cache.genTtlMs: 0`, or on a `StorageChunkSource` of your own has no bound at all: it converges only when a read happens to miss its caches. If a compliance deadline depends on every
|
|
@@ -1777,7 +1785,9 @@ export declare class Segment {
|
|
|
1777
1785
|
* `*Into` of one throws before it reads anything. Materialise one side first, with `materializeMany({ operands: { src: seg }, outputs: [{ dest, expr: { or: ['src'] } }], keep })`.
|
|
1778
1786
|
*
|
|
1779
1787
|
* **A pin knows its object, not only its number.** A name purged and loaded again starts again at generation
|
|
1780
|
-
* 0, so a pin of the old segment never reads the new one
|
|
1788
|
+
* 0, so a pin of the old segment never reads the new one. On a store with a timed refresh the pin fails with
|
|
1789
|
+
* `NotFoundError` at its next check, within `cache.genTtlMs`, what it had already read included, and `invalidate`
|
|
1790
|
+
* does not bring it back: take a new pin. On a store with none, what it has already read still answers, as the
|
|
1781
1791
|
* instant it pinned, and anything it would have to fetch fails with `NotFoundError`, as a swept pin's does.
|
|
1782
1792
|
*
|
|
1783
1793
|
* **It is a hold, and `pin({ leaseUntil })` makes it a bounded one.** Without a lease nothing stops a collection
|
|
@@ -1799,7 +1809,11 @@ export declare class Segment {
|
|
|
1799
1809
|
* segment whose row is later dropped or destroyed fails once it must open its object again, rather than go empty
|
|
1800
1810
|
* part-way through a call. **A pin
|
|
1801
1811
|
* keeps the key its reader unwrapped for as long as that reader stays open, and answers from the chunks it
|
|
1802
|
-
* decoded for as long as they stay cached.**
|
|
1812
|
+
* decoded for as long as they stay cached, until its next check.** On a store with a timed refresh a pin checks its
|
|
1813
|
+
* segment's row every `cache.genTtlMs` (from the moment it was taken, whether or not it was read) and fails with
|
|
1814
|
+
* `NotFoundError` once the segment is destroyed, is another incarnation, or its generation is collected or recorded by its row as another object (an object lost or replaced out of band
|
|
1815
|
+
* under a row that still names it is found only when the pin must fetch). This
|
|
1816
|
+
* store invalidates a pin:
|
|
1803
1817
|
*
|
|
1804
1818
|
* - a `load`, a `rollback` or an `*Into` invalidates a pin of the segment it writes;
|
|
1805
1819
|
* - `dropSegment` invalidates a pin of the segment it drops, and `retireExpired` a pin of each segment its ledger
|
|
@@ -1812,16 +1826,18 @@ export declare class Segment {
|
|
|
1812
1826
|
* where the object is still there, and fails with `NotFoundError` where it is gone. It caches nothing more of what it
|
|
1813
1827
|
* began reading before the invalidation, so the pin's next read opens the object again too.
|
|
1814
1828
|
* Anything else leaves the pin as it is: after a `destroySegment` beside this store, or an erasure, a drop or a
|
|
1815
|
-
* retirement through another store, in this process or another, the pin answers from what it holds
|
|
1829
|
+
* retirement through another store, in this process or another, the pin answers from what it holds for at most
|
|
1830
|
+
* `cache.genTtlMs` on a store with a timed refresh, and then fails. On a store with none it answers until this
|
|
1816
1831
|
* store's reader cache evicts the pin's reader and its chunk cache evicts the chunks the pin decoded, or
|
|
1817
1832
|
* {@link CloudRoaring.invalidate} is called on this store. A combine on another store refuses the pin. Where the
|
|
1818
1833
|
* object the pin reads has been deleted, by an erasure, a drop or a sweep, a chunk the pin has not cached fails at
|
|
1819
|
-
* once.
|
|
1834
|
+
* once. With no timed refresh nothing bounds any of that, as nothing bounds anything else a pin holds.
|
|
1820
1835
|
*
|
|
1821
|
-
* A replacement a pin
|
|
1836
|
+
* A replacement a read of the pin found is remembered, so its later chunk reads fail with no request, until the store
|
|
1822
1837
|
* forgets it: an invalidation of the segment does, as does a later `pin()` of the same version that opens the object
|
|
1823
|
-
* again
|
|
1824
|
-
* call {@link CloudRoaring.invalidate} on this store: the pin then reads the object again.
|
|
1838
|
+
* again; a replacement the pin's own check found is forgotten only by an invalidation. The store remembers at most `cache.readerMax` of them. So once a restore puts the pinned object back
|
|
1839
|
+
* into the same row, call {@link CloudRoaring.invalidate} on this store: the pin then reads the object again. A
|
|
1840
|
+
* restore into a purged and re-created row does not revive a pin on a store with a timed refresh: take a new pin.
|
|
1825
1841
|
*
|
|
1826
1842
|
* `pin()` reads the registry row and opens the generation at once, so the pin knows its object before its first
|
|
1827
1843
|
* read. With a registry, pins of one generation taken while its row is unchanged share one reader while the store
|
|
@@ -2128,6 +2144,59 @@ export declare class Segment {
|
|
|
2128
2144
|
* {@link ValidationError} before any request.
|
|
2129
2145
|
*/
|
|
2130
2146
|
andNotInto(dest: Segment, excludes: Segment[], options?: CombineOptions): Promise<MaterializeResult>;
|
|
2147
|
+
/**
|
|
2148
|
+
* How many ids `this ∩ others…` holds (minus `exclude`): the length of what {@link intersect} streams, without
|
|
2149
|
+
* streaming it. Every key all the operands hold is decoded and counted as the stream reads it, and a key only some
|
|
2150
|
+
* operands hold is not read, so the call makes the requests {@link intersect} makes and returns a number instead of
|
|
2151
|
+
* the ids. It is exact: it never answers a key from an index, so it trusts nothing {@link count} trusts.
|
|
2152
|
+
*
|
|
2153
|
+
* The range, `concurrency`, `budget`, `allowAbsentOperands` and `exclude` are {@link intersect}'s; a write option fails
|
|
2154
|
+
* the call with {@link ValidationError} before any request, and so does an empty `others` (to count one segment, use
|
|
2155
|
+
* {@link count}).
|
|
2156
|
+
*
|
|
2157
|
+
* Like a stream, a count that runs while a segment is erased or reloaded can describe two instants, within the bound
|
|
2158
|
+
* the streams state. A union or a difference, and an intersection with an `exclude`, looks at its operands'
|
|
2159
|
+
* generations every `concurrency` keys; an intersection with no `exclude` makes no such look, since it decodes every
|
|
2160
|
+
* key and so shows a move in the chunks it decodes, as its stream does. Either way it plans again above the last key
|
|
2161
|
+
* it took when one moved. A pinned handle counts the generation it pinned, and each pinned operand's object is
|
|
2162
|
+
* looked up once before the first key (one footer read each, which `budget` does not charge, and which the stream
|
|
2163
|
+
* does not make), and again at each look where there is one, so an erasure of what it pinned fails the count with
|
|
2164
|
+
* {@link NotFoundError}; a
|
|
2165
|
+
* fault the look meets other than a transient one, which is retried, is raised as it is. A call that involves a
|
|
2166
|
+
* handle whose lease has ended, or has ended by the next look, throws {@link LeaseExpiredError}.
|
|
2167
|
+
*/
|
|
2168
|
+
intersectCount(others: Segment[], options?: CombineOptions): Promise<number>;
|
|
2169
|
+
/**
|
|
2170
|
+
* How many ids `this ∪ others…` holds (minus `exclude`): the length of what {@link union} streams, without streaming
|
|
2171
|
+
* it. A chunk key only one operand holds, that no `exclude` holds and the range does not cut, is answered from that
|
|
2172
|
+
* operand's index, so a union of segments that mostly do not overlap reads little of them; every other key is
|
|
2173
|
+
* decoded and counted. On handles that are not pinned it makes no more requests than {@link union} does, and reads
|
|
2174
|
+
* fewer bytes when the shared keys are few. An empty `others`, with no `exclude` and no range, is {@link count} of
|
|
2175
|
+
* `this`, answered from the registry row when it can be; when that figure is 0 it adds one registry read, to refuse
|
|
2176
|
+
* a segment that does not exist as the stream does.
|
|
2177
|
+
*
|
|
2178
|
+
* What it trusts: the index, for each key it answers without decoding, as {@link count} does, so a party who can write
|
|
2179
|
+
* an object's index can make that key's figure wrong; a decoded key is exact. A source with no per-chunk counts
|
|
2180
|
+
* decodes every key. It takes the options {@link intersectCount} takes; the erasure bound, the lease checks and the
|
|
2181
|
+
* pinned handles are as for {@link intersectCount}, a pinned operand adding a footer read before the first key and at
|
|
2182
|
+
* each look. An empty `others` on a pinned handle is {@link count} of the pin, with no look.
|
|
2183
|
+
*/
|
|
2184
|
+
unionCount(others: Segment[], options?: CombineOptions): Promise<number>;
|
|
2185
|
+
/**
|
|
2186
|
+
* How many ids `this \ (excludes…)` holds: the length of what {@link andNot} streams, without streaming it. A chunk
|
|
2187
|
+
* key of `this` that no exclude holds and the range does not cut is answered from its index and not read; a key an
|
|
2188
|
+
* exclude shares is decoded and counted. On handles that are not pinned it makes no more requests than {@link andNot}
|
|
2189
|
+
* does, and reads fewer bytes when the excludes overlap little of `this`. An empty `excludes`, with no range, is
|
|
2190
|
+
* {@link count} of `this` (the stream of it is the whole of `this`), answered from the registry row when it can be,
|
|
2191
|
+
* with one registry read added when that figure is 0.
|
|
2192
|
+
*
|
|
2193
|
+
* It takes the options {@link andNot} takes (every one of {@link intersectCount}'s but `exclude`, which a count of a
|
|
2194
|
+
* difference has no use for, since `excludes` is its subtraction) and trusts what {@link unionCount} trusts. The
|
|
2195
|
+
* erasure bound, the lease checks and the pinned handles are as for {@link intersectCount}.
|
|
2196
|
+
*/
|
|
2197
|
+
andNotCount(excludes: Segment[], options?: CombineOptions): Promise<number>;
|
|
2198
|
+
/** The one body of the three count methods, after each has checked its own options. */
|
|
2199
|
+
private countCombine;
|
|
2131
2200
|
/**
|
|
2132
2201
|
* The one body of the three combines, read as ids or as chunks. `operands` is `others` for `intersect` and
|
|
2133
2202
|
* `union`, and `excludes` for `andNot`, which takes no `exclude` option.
|
package/dist/index.js
CHANGED
|
@@ -3973,6 +3973,29 @@ var CALL_KEYS = {
|
|
|
3973
3973
|
"allowAbsentOperands",
|
|
3974
3974
|
"exclude"
|
|
3975
3975
|
]),
|
|
3976
|
+
intersectCount: keysOf()([
|
|
3977
|
+
"after",
|
|
3978
|
+
"through",
|
|
3979
|
+
"concurrency",
|
|
3980
|
+
"budget",
|
|
3981
|
+
"allowAbsentOperands",
|
|
3982
|
+
"exclude"
|
|
3983
|
+
]),
|
|
3984
|
+
unionCount: keysOf()([
|
|
3985
|
+
"after",
|
|
3986
|
+
"through",
|
|
3987
|
+
"concurrency",
|
|
3988
|
+
"budget",
|
|
3989
|
+
"allowAbsentOperands",
|
|
3990
|
+
"exclude"
|
|
3991
|
+
]),
|
|
3992
|
+
andNotCount: keysOf()([
|
|
3993
|
+
"after",
|
|
3994
|
+
"through",
|
|
3995
|
+
"concurrency",
|
|
3996
|
+
"budget",
|
|
3997
|
+
"allowAbsentOperands"
|
|
3998
|
+
]),
|
|
3976
3999
|
into: keysOf()([
|
|
3977
4000
|
"after",
|
|
3978
4001
|
"through",
|
|
@@ -4050,6 +4073,9 @@ var MISPLACED = {
|
|
|
4050
4073
|
range: outside(CALL_KEYS.into, CALL_KEYS.range),
|
|
4051
4074
|
andNot: outside(CALL_KEYS.into, CALL_KEYS.andNot),
|
|
4052
4075
|
combine: outside(CALL_KEYS.into, CALL_KEYS.combine),
|
|
4076
|
+
intersectCount: outside(CALL_KEYS.into, CALL_KEYS.intersectCount),
|
|
4077
|
+
unionCount: outside(CALL_KEYS.into, CALL_KEYS.unionCount),
|
|
4078
|
+
andNotCount: outside(CALL_KEYS.into, CALL_KEYS.andNotCount),
|
|
4053
4079
|
andNotInto: outside(CALL_KEYS.into, CALL_KEYS.andNotInto),
|
|
4054
4080
|
exportSegments: ["codec"]
|
|
4055
4081
|
};
|
|
@@ -4874,7 +4900,7 @@ var CloudRoaring = class _CloudRoaring {
|
|
|
4874
4900
|
* duration, or fix the source first and load after. A writer that lands *during* the rewrite is caught: the
|
|
4875
4901
|
* rewrite's publish is refused **by the fence** — the publish lands its compare-and-swap only
|
|
4876
4902
|
* while the pointer is still on the generation the rewrite streamed — and the entry says `note: 'superseded'`,
|
|
4877
|
-
* so re-run. Forward-only alone would NOT refuse it: the rewrite's number goes above everything
|
|
4903
|
+
* so re-run. Forward-only alone would NOT refuse it: the rewrite's number goes above everything listed near the row's own numbers,
|
|
4878
4904
|
* so the rewrite would out-rank the newer generation and then collect it.
|
|
4879
4905
|
*
|
|
4880
4906
|
* A racing **erasure** is caught before that, and reported the same way. It collects with `keep: 0`, taking
|
|
@@ -5112,7 +5138,8 @@ var CloudRoaring = class _CloudRoaring {
|
|
|
5112
5138
|
* It deletes nothing. The generations above the new pointer stay put — which is what makes this reversible —
|
|
5113
5139
|
* and are then *above* `currentGen`, where collection never looks, and still known to the row, so rolling forward onto
|
|
5114
5140
|
* one needs no flag. They remain until loads pass them (the first load whose
|
|
5115
|
-
* number one of them holds numbers above them all,
|
|
5141
|
+
* number one of them holds numbers above them all, those within 1,024 of what the row names and any contiguous run
|
|
5142
|
+
* touching them, and collection then keeps `keep` of what is below its pointer, the
|
|
5116
5143
|
* generation that load superseded first), {@link CloudRoaring.dropSegment}
|
|
5117
5144
|
* deletes them, or {@link CloudRoaring.eraseSubject} does: all of those present when it rewrites, and, when the
|
|
5118
5145
|
* current generation does not hold the id, only those holding it and any no read of the segment can open, whatever
|
|
@@ -5489,7 +5516,8 @@ var CloudRoaring = class _CloudRoaring {
|
|
|
5489
5516
|
* the unwrapped DEK, and decrypts with them, chunks it had never fetched before the shred included, until its
|
|
5490
5517
|
* snapshot re-resolves: within `cache.genTtlMs` when the store has a registry and a `genTtlMs` above 0, and
|
|
5491
5518
|
* otherwise only when its caches let the segment go or a read has to fetch from a generation a sweep collected.
|
|
5492
|
-
* A pin
|
|
5519
|
+
* A pin on a store with a timed refresh fails within `cache.genTtlMs` once its segment is destroyed or its
|
|
5520
|
+
* generation erased; on a store with none it answers until it is told or both caches let it go.
|
|
5493
5521
|
* - **Another process.** Erasing on one box invalidates nothing on the others; each store bounds its own
|
|
5494
5522
|
* staleness by `cache.genTtlMs`, and a store with no registry (a bare `IStorageDriver`), with
|
|
5495
5523
|
* `cache.genTtlMs: 0`, or on a `StorageChunkSource` of your own has no bound at all: it converges only when a read happens to miss its caches. If a compliance deadline depends on every
|
|
@@ -5571,6 +5599,7 @@ var CloudRoaring = class _CloudRoaring {
|
|
|
5571
5599
|
throw err;
|
|
5572
5600
|
}
|
|
5573
5601
|
const pinnedAt = at === null ? { generation: null, version: null, fingerprint: null } : { generation: at.generation, version: at.version, fingerprint: at.fingerprint };
|
|
5602
|
+
if (at !== null) crbm.startPinChecks(pinnedAt, at.checkedAtMs);
|
|
5574
5603
|
const pins = /* @__PURE__ */ new Map([[segmentKey(ref), pinnedAt]]);
|
|
5575
5604
|
return makeSegment({
|
|
5576
5605
|
engine: this.engineWithPins(pins),
|
|
@@ -5816,7 +5845,7 @@ function operandsLabel(verb) {
|
|
|
5816
5845
|
}
|
|
5817
5846
|
function refuseNoOperands(verb) {
|
|
5818
5847
|
throw new ValidationError6(
|
|
5819
|
-
verb === "intersect" ? "intersect requires at least one segment to intersect with; to read every id of one segment, use iterate()" : "intersectInto requires at least one segment to intersect with; to copy this segment into dest, call materializeMany({ operands: { src: thisSegment }, outputs: [{ dest, expr: { or: ['src'] } }], keep })"
|
|
5848
|
+
verb === "intersect" ? "intersect requires at least one segment to intersect with; to read every id of one segment, use iterate()" : verb === "intersectCount" ? "intersectCount requires at least one segment to intersect with; to count one segment, use count()" : "intersectInto requires at least one segment to intersect with; to copy this segment into dest, call materializeMany({ operands: { src: thisSegment }, outputs: [{ dest, expr: { or: ['src'] } }], keep })"
|
|
5820
5849
|
);
|
|
5821
5850
|
}
|
|
5822
5851
|
var AS_IDS = {
|
|
@@ -5938,7 +5967,9 @@ var Segment = class _Segment {
|
|
|
5938
5967
|
* `*Into` of one throws before it reads anything. Materialise one side first, with `materializeMany({ operands: { src: seg }, outputs: [{ dest, expr: { or: ['src'] } }], keep })`.
|
|
5939
5968
|
*
|
|
5940
5969
|
* **A pin knows its object, not only its number.** A name purged and loaded again starts again at generation
|
|
5941
|
-
* 0, so a pin of the old segment never reads the new one
|
|
5970
|
+
* 0, so a pin of the old segment never reads the new one. On a store with a timed refresh the pin fails with
|
|
5971
|
+
* `NotFoundError` at its next check, within `cache.genTtlMs`, what it had already read included, and `invalidate`
|
|
5972
|
+
* does not bring it back: take a new pin. On a store with none, what it has already read still answers, as the
|
|
5942
5973
|
* instant it pinned, and anything it would have to fetch fails with `NotFoundError`, as a swept pin's does.
|
|
5943
5974
|
*
|
|
5944
5975
|
* **It is a hold, and `pin({ leaseUntil })` makes it a bounded one.** Without a lease nothing stops a collection
|
|
@@ -5960,7 +5991,11 @@ var Segment = class _Segment {
|
|
|
5960
5991
|
* segment whose row is later dropped or destroyed fails once it must open its object again, rather than go empty
|
|
5961
5992
|
* part-way through a call. **A pin
|
|
5962
5993
|
* keeps the key its reader unwrapped for as long as that reader stays open, and answers from the chunks it
|
|
5963
|
-
* decoded for as long as they stay cached.**
|
|
5994
|
+
* decoded for as long as they stay cached, until its next check.** On a store with a timed refresh a pin checks its
|
|
5995
|
+
* segment's row every `cache.genTtlMs` (from the moment it was taken, whether or not it was read) and fails with
|
|
5996
|
+
* `NotFoundError` once the segment is destroyed, is another incarnation, or its generation is collected or recorded by its row as another object (an object lost or replaced out of band
|
|
5997
|
+
* under a row that still names it is found only when the pin must fetch). This
|
|
5998
|
+
* store invalidates a pin:
|
|
5964
5999
|
*
|
|
5965
6000
|
* - a `load`, a `rollback` or an `*Into` invalidates a pin of the segment it writes;
|
|
5966
6001
|
* - `dropSegment` invalidates a pin of the segment it drops, and `retireExpired` a pin of each segment its ledger
|
|
@@ -5973,16 +6008,18 @@ var Segment = class _Segment {
|
|
|
5973
6008
|
* where the object is still there, and fails with `NotFoundError` where it is gone. It caches nothing more of what it
|
|
5974
6009
|
* began reading before the invalidation, so the pin's next read opens the object again too.
|
|
5975
6010
|
* Anything else leaves the pin as it is: after a `destroySegment` beside this store, or an erasure, a drop or a
|
|
5976
|
-
* retirement through another store, in this process or another, the pin answers from what it holds
|
|
6011
|
+
* retirement through another store, in this process or another, the pin answers from what it holds for at most
|
|
6012
|
+
* `cache.genTtlMs` on a store with a timed refresh, and then fails. On a store with none it answers until this
|
|
5977
6013
|
* store's reader cache evicts the pin's reader and its chunk cache evicts the chunks the pin decoded, or
|
|
5978
6014
|
* {@link CloudRoaring.invalidate} is called on this store. A combine on another store refuses the pin. Where the
|
|
5979
6015
|
* object the pin reads has been deleted, by an erasure, a drop or a sweep, a chunk the pin has not cached fails at
|
|
5980
|
-
* once.
|
|
6016
|
+
* once. With no timed refresh nothing bounds any of that, as nothing bounds anything else a pin holds.
|
|
5981
6017
|
*
|
|
5982
|
-
* A replacement a pin
|
|
6018
|
+
* A replacement a read of the pin found is remembered, so its later chunk reads fail with no request, until the store
|
|
5983
6019
|
* forgets it: an invalidation of the segment does, as does a later `pin()` of the same version that opens the object
|
|
5984
|
-
* again
|
|
5985
|
-
* call {@link CloudRoaring.invalidate} on this store: the pin then reads the object again.
|
|
6020
|
+
* again; a replacement the pin's own check found is forgotten only by an invalidation. The store remembers at most `cache.readerMax` of them. So once a restore puts the pinned object back
|
|
6021
|
+
* into the same row, call {@link CloudRoaring.invalidate} on this store: the pin then reads the object again. A
|
|
6022
|
+
* restore into a purged and re-created row does not revive a pin on a store with a timed refresh: take a new pin.
|
|
5986
6023
|
*
|
|
5987
6024
|
* `pin()` reads the registry row and opens the generation at once, so the pin knows its object before its first
|
|
5988
6025
|
* read. With a registry, pins of one generation taken while its row is unchanged share one reader while the store
|
|
@@ -6526,6 +6563,111 @@ var Segment = class _Segment {
|
|
|
6526
6563
|
);
|
|
6527
6564
|
return this.combineInto("andNotInto", dest, excludes, options);
|
|
6528
6565
|
}
|
|
6566
|
+
/**
|
|
6567
|
+
* How many ids `this ∩ others…` holds (minus `exclude`): the length of what {@link intersect} streams, without
|
|
6568
|
+
* streaming it. Every key all the operands hold is decoded and counted as the stream reads it, and a key only some
|
|
6569
|
+
* operands hold is not read, so the call makes the requests {@link intersect} makes and returns a number instead of
|
|
6570
|
+
* the ids. It is exact: it never answers a key from an index, so it trusts nothing {@link count} trusts.
|
|
6571
|
+
*
|
|
6572
|
+
* The range, `concurrency`, `budget`, `allowAbsentOperands` and `exclude` are {@link intersect}'s; a write option fails
|
|
6573
|
+
* the call with {@link ValidationError} before any request, and so does an empty `others` (to count one segment, use
|
|
6574
|
+
* {@link count}).
|
|
6575
|
+
*
|
|
6576
|
+
* Like a stream, a count that runs while a segment is erased or reloaded can describe two instants, within the bound
|
|
6577
|
+
* the streams state. A union or a difference, and an intersection with an `exclude`, looks at its operands'
|
|
6578
|
+
* generations every `concurrency` keys; an intersection with no `exclude` makes no such look, since it decodes every
|
|
6579
|
+
* key and so shows a move in the chunks it decodes, as its stream does. Either way it plans again above the last key
|
|
6580
|
+
* it took when one moved. A pinned handle counts the generation it pinned, and each pinned operand's object is
|
|
6581
|
+
* looked up once before the first key (one footer read each, which `budget` does not charge, and which the stream
|
|
6582
|
+
* does not make), and again at each look where there is one, so an erasure of what it pinned fails the count with
|
|
6583
|
+
* {@link NotFoundError}; a
|
|
6584
|
+
* fault the look meets other than a transient one, which is retried, is raised as it is. A call that involves a
|
|
6585
|
+
* handle whose lease has ended, or has ended by the next look, throws {@link LeaseExpiredError}.
|
|
6586
|
+
*/
|
|
6587
|
+
async intersectCount(others, options) {
|
|
6588
|
+
checkOptions(
|
|
6589
|
+
options,
|
|
6590
|
+
"intersectCount",
|
|
6591
|
+
CALL_KEYS.intersectCount,
|
|
6592
|
+
["allowAbsentOperands"],
|
|
6593
|
+
MISPLACED.intersectCount
|
|
6594
|
+
);
|
|
6595
|
+
return this.countCombine("intersect", others, options ?? void 0);
|
|
6596
|
+
}
|
|
6597
|
+
/**
|
|
6598
|
+
* How many ids `this ∪ others…` holds (minus `exclude`): the length of what {@link union} streams, without streaming
|
|
6599
|
+
* it. A chunk key only one operand holds, that no `exclude` holds and the range does not cut, is answered from that
|
|
6600
|
+
* operand's index, so a union of segments that mostly do not overlap reads little of them; every other key is
|
|
6601
|
+
* decoded and counted. On handles that are not pinned it makes no more requests than {@link union} does, and reads
|
|
6602
|
+
* fewer bytes when the shared keys are few. An empty `others`, with no `exclude` and no range, is {@link count} of
|
|
6603
|
+
* `this`, answered from the registry row when it can be; when that figure is 0 it adds one registry read, to refuse
|
|
6604
|
+
* a segment that does not exist as the stream does.
|
|
6605
|
+
*
|
|
6606
|
+
* What it trusts: the index, for each key it answers without decoding, as {@link count} does, so a party who can write
|
|
6607
|
+
* an object's index can make that key's figure wrong; a decoded key is exact. A source with no per-chunk counts
|
|
6608
|
+
* decodes every key. It takes the options {@link intersectCount} takes; the erasure bound, the lease checks and the
|
|
6609
|
+
* pinned handles are as for {@link intersectCount}, a pinned operand adding a footer read before the first key and at
|
|
6610
|
+
* each look. An empty `others` on a pinned handle is {@link count} of the pin, with no look.
|
|
6611
|
+
*/
|
|
6612
|
+
async unionCount(others, options) {
|
|
6613
|
+
checkOptions(
|
|
6614
|
+
options,
|
|
6615
|
+
"unionCount",
|
|
6616
|
+
CALL_KEYS.unionCount,
|
|
6617
|
+
["allowAbsentOperands"],
|
|
6618
|
+
MISPLACED.unionCount
|
|
6619
|
+
);
|
|
6620
|
+
return this.countCombine("union", others, options ?? void 0);
|
|
6621
|
+
}
|
|
6622
|
+
/**
|
|
6623
|
+
* How many ids `this \ (excludes…)` holds: the length of what {@link andNot} streams, without streaming it. A chunk
|
|
6624
|
+
* key of `this` that no exclude holds and the range does not cut is answered from its index and not read; a key an
|
|
6625
|
+
* exclude shares is decoded and counted. On handles that are not pinned it makes no more requests than {@link andNot}
|
|
6626
|
+
* does, and reads fewer bytes when the excludes overlap little of `this`. An empty `excludes`, with no range, is
|
|
6627
|
+
* {@link count} of `this` (the stream of it is the whole of `this`), answered from the registry row when it can be,
|
|
6628
|
+
* with one registry read added when that figure is 0.
|
|
6629
|
+
*
|
|
6630
|
+
* It takes the options {@link andNot} takes (every one of {@link intersectCount}'s but `exclude`, which a count of a
|
|
6631
|
+
* difference has no use for, since `excludes` is its subtraction) and trusts what {@link unionCount} trusts. The
|
|
6632
|
+
* erasure bound, the lease checks and the pinned handles are as for {@link intersectCount}.
|
|
6633
|
+
*/
|
|
6634
|
+
async andNotCount(excludes, options) {
|
|
6635
|
+
checkOptions(
|
|
6636
|
+
options,
|
|
6637
|
+
"andNotCount",
|
|
6638
|
+
CALL_KEYS.andNotCount,
|
|
6639
|
+
["allowAbsentOperands"],
|
|
6640
|
+
MISPLACED.andNotCount
|
|
6641
|
+
);
|
|
6642
|
+
return this.countCombine("andNot", excludes, options ?? void 0);
|
|
6643
|
+
}
|
|
6644
|
+
/** The one body of the three count methods, after each has checked its own options. */
|
|
6645
|
+
async countCombine(verb, operands, options) {
|
|
6646
|
+
const name = `${verb}Count`;
|
|
6647
|
+
const exclude = verb === "andNot" ? NO_SEGMENTS : this.excludesOf(options);
|
|
6648
|
+
const others = segmentList(
|
|
6649
|
+
operands,
|
|
6650
|
+
verb === "andNot" ? `${name}: \`excludes\`` : `${name}: \`others\``
|
|
6651
|
+
);
|
|
6652
|
+
if (verb === "intersect" && others.length === 0) refuseNoOperands("intersectCount");
|
|
6653
|
+
const handles = [this, ...others, ...exclude];
|
|
6654
|
+
this.assertSegments(name, verb === "andNot" ? "an exclude" : "an operand", others);
|
|
6655
|
+
this.assertSegments(name, "an exclude", exclude);
|
|
6656
|
+
this.assertLeases(handles);
|
|
6657
|
+
const engine = this.combineEngine(handles) ?? this.engine;
|
|
6658
|
+
const read = options == null ? void 0 : readOptions(options);
|
|
6659
|
+
const guard2 = handles.some((h) => h.leaseHold !== void 0) ? () => this.assertLeases(handles) : void 0;
|
|
6660
|
+
const opts = guard2 === void 0 ? read : { ...read, guard: guard2 };
|
|
6661
|
+
const refs = others.map((o) => o.ref);
|
|
6662
|
+
if (verb === "andNot") return engine.combineCount([this.ref], refs, "all", verb, opts);
|
|
6663
|
+
return engine.combineCount(
|
|
6664
|
+
[this.ref, ...refs],
|
|
6665
|
+
exclude.map((o) => o.ref),
|
|
6666
|
+
verb === "union" ? "any" : "all",
|
|
6667
|
+
verb,
|
|
6668
|
+
opts
|
|
6669
|
+
);
|
|
6670
|
+
}
|
|
6529
6671
|
/**
|
|
6530
6672
|
* The one body of the three combines, read as ids or as chunks. `operands` is `others` for `intersect` and
|
|
6531
6673
|
* `union`, and `excludes` for `andNot`, which takes no `exclude` option.
|