@cloudbitmaps/roaring 0.16.0 → 0.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/PRIVACY.md +16 -0
- package/README.md +4 -4
- package/dist/bin/export-segments.js +531 -59
- package/dist/bin/export-segments.js.map +4 -4
- package/dist/index.d.ts +223 -36
- package/dist/index.js +543 -60
- package/dist/index.js.map +4 -4
- package/dist/lease-guards.d.ts +34 -0
- package/dist/roaring-codec.d.ts +23 -0
- package/package.json +2 -2
package/PRIVACY.md
CHANGED
|
@@ -136,7 +136,9 @@ bucket.
|
|
|
136
136
|
| the store whose verb made the call (`eraseSubject`, `dropSegment`, `retireExpired`) | on return, for every read that starts after it — it invalidates what it cached, and its pins then fail; a read already in progress there can still yield it from a chunk it had requested before |
|
|
137
137
|
| another store, with a registry and a `cache.genTtlMs` above 0 | within `cache.genTtlMs` (default 2 s), when its snapshot re-resolves, **while the registry can be read** (see below) |
|
|
138
138
|
| another store with **no registry** (a bare `IStorageDriver`), with `cache: { genTtlMs: 0 }`, or on a storage source built with **no clock** | **no bound** — only when its caches happen to let the segment go, or something tells it |
|
|
139
|
+
| `seg.pinAt` of a generation the erasure rewrote, in any store | on return it fails with `NotFoundError`, since the object is deleted (and, with a registry, the row has moved on); during the erasure, between the rewrite's publish and its delete of the old generation, it can still open that generation, and a handle opened then is a pin, covered by the row below |
|
|
139
140
|
| a pinned handle (`seg.pin()`) in another store | **no bound** — until that store's reader cache evicts the pin's reader (in a store with a timed pointer refresh, a small generation's reader holds all of its chunks, decoded or not) and its chunk cache evicts the chunks the pin decoded, or something tells it |
|
|
141
|
+
| a leased pinned handle (`seg.pin({ leaseUntil })`) in another store | until its lease ends, at most 14 days after it was taken: its reads then throw `LeaseExpiredError`, whether or not its reader still holds the chunks. Until then, as for any pinned handle in the row above |
|
|
140
142
|
|
|
141
143
|
**An outage of the registry extends that bound.** A refresh that cannot read the row because of a transient fault
|
|
142
144
|
(throttling, a 5xx, a dropped connection) keeps serving the generation the reader already holds, and the key it
|
|
@@ -161,6 +163,20 @@ yours.
|
|
|
161
163
|
A read served from a stale cache is bounded by the same window and answers from memory, so nothing on the
|
|
162
164
|
storage side — a bucket policy, a lifecycle rule, the object's own deletion — can shorten it.
|
|
163
165
|
|
|
166
|
+
**A lease keeps superseded generations, and erasure ignores it.** A pinned handle taken with `pin({ leaseUntil })` keeps
|
|
167
|
+
its generation out of a load's collection until the lease ends: at most 14 days after it was taken, and a 60-second margin
|
|
168
|
+
after that. While it does, the bucket holds a generation that a newer load has superseded, including ids that load
|
|
169
|
+
removed. `eraseSubject`, `eraseIdFromSegment`, `dropSegment`, `destroySegment` and a retention expiry never read a lease:
|
|
170
|
+
they delete the generations they must, a leased one included, so a lease never holds an erased subject's data past the
|
|
171
|
+
bounds above, and erasure, shred, drop and retention always win over it. A rewrite, a shred and a drop clear the row's
|
|
172
|
+
leases; an erasure that deletes a generation without rewriting leaves an entry for it in the row, which spares nothing and
|
|
173
|
+
is pruned by the next publish or lease write after it ends. A lease write does not delay an erasure: a row that differs
|
|
174
|
+
from the one the erasure read only in its leases does not refuse it. What a lease holds against is an ordinary load's
|
|
175
|
+
collection, and the generation is collected by the next listing pass after the lease ends: within 16 later loads of the
|
|
176
|
+
segment, and a segment that is never loaded again keeps it until an erasure, a drop or a retention expiry takes it. A
|
|
177
|
+
leased handle stops answering when its lease ends, and its reads then throw `LeaseExpiredError`. The lease entry in the
|
|
178
|
+
registry row is a random holder id, a generation number and an instant, with no id and no count.
|
|
179
|
+
|
|
164
180
|
**Your exit path** (and a building block for a **data-portability / Art. 20** response): `store.exportSegments(sink,
|
|
165
181
|
{ format })` (and the `export-segments` CLI) dumps every registered segment's current generation to a portable
|
|
166
182
|
file — `roaring` (loadable by any roaring library) or `ndjson` (newline ids) — readable without CloudBitmaps.
|
package/README.md
CHANGED
|
@@ -69,10 +69,10 @@ name.
|
|
|
69
69
|
|
|
70
70
|
## What it costs
|
|
71
71
|
|
|
72
|
-
A cold `intersect` of two 500,000-id segments sharing 100 of their 1,999 chunks made 6 GETs and took
|
|
73
|
-
million** at list prices: measured on S3 in-region, in the run of 2026-10-
|
|
74
|
-
bytes than it needs, which cross-region can cost more than the requests it saves. A segment's first `store.load()`
|
|
75
|
-
|
|
72
|
+
A cold `intersect` of two 500,000-id segments sharing 100 of their 1,999 chunks made 6 GETs and took 92.63 ms at the median, **$2.40 per
|
|
73
|
+
million** at list prices: measured on S3 in-region, in the run of 2026-10-05, whose client had 128 sockets. On a spread layout it reads more
|
|
74
|
+
bytes than it needs, which cross-region can cost more than the requests it saves. A segment's first `store.load()` made
|
|
75
|
+
2 PUT + 3 GET, **$11.20 per million** at the default prices, pointer included: the requests are measured on S3 in-region, in the same run. A cold `count()` is one
|
|
76
76
|
pointer read: it reads no payload and no object.
|
|
77
77
|
|
|
78
78
|
The trade is stated plainly. A membership check that misses the cache costs a ranged GET against object storage,
|