slatedb 0.5.0-aarch64-linux → 0.7.0-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +150 -44
- data/lib/slatedb/3.3/slatedb.so +0 -0
- data/lib/slatedb/3.4/slatedb.so +0 -0
- data/lib/slatedb/4.0/slatedb.so +0 -0
- data/lib/slatedb/database.rb +51 -27
- data/lib/slatedb/reader.rb +38 -27
- data/lib/slatedb/snapshot.rb +17 -9
- data/lib/slatedb/transaction.rb +17 -9
- data/lib/slatedb/version.rb +1 -1
- metadata +4 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: af0687022f9a14510d013da646aefbd6fedfa2b770235460a1e9d416b05d444a
|
|
4
|
+
data.tar.gz: ea04403866e4d528bb85e07a5a08392a6e2e73b073eea7c7685d2324fae72cf2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 5dd9197393629666a17468ae03966a34cad89313a8aeac54bcec427f40baf91ae1f7964fdcd1a06d0a4d2847170a013b6da0c6aafa49a8cb0256466cde133427
|
|
7
|
+
data.tar.gz: 2c01206156a4ff9c9a037df96b08c936ca055d1888dd52adc4817b34d0a99a81e174d7ab9078ac01fc2a6293885758ec8ba63924d85ac38f05fa0477a8760aa5
|
data/README.md
CHANGED
|
@@ -30,6 +30,38 @@ gem install slatedb
|
|
|
30
30
|
|
|
31
31
|
Precompiled native gems are published for common Linux and Darwin platforms. Building from source still requires a Rust toolchain.
|
|
32
32
|
|
|
33
|
+
## Upgrading to SlateDB 0.18.1
|
|
34
|
+
|
|
35
|
+
This update moves the Rust engine from 0.17.0 to 0.18.1. Existing database files
|
|
36
|
+
remain readable. New Ruby options keep the existing defaults unless enabled.
|
|
37
|
+
|
|
38
|
+
- Two-phase checkpoints: `begin_checkpoint` returns a handle with `id` and `wait`.
|
|
39
|
+
Both checkpoint methods accept `scope: :all` to capture pending writes, or
|
|
40
|
+
`scope: :durable` (the existing default) for writes already durable.
|
|
41
|
+
- Point reads accept `max_concurrent_sources:` on databases, readers, snapshots,
|
|
42
|
+
and transactions; database metadata reads accept it too.
|
|
43
|
+
- `Database.open` accepts `experimental_incremental_compaction:` and
|
|
44
|
+
`sst_block_alignment:`. Both are disabled by default; see the warnings below.
|
|
45
|
+
- Snapshot and transaction range/prefix scans now accept `order:`, like database
|
|
46
|
+
and reader scans. Reader `merge_operator:` is now applied rather than ignored.
|
|
47
|
+
- **Breaking:** Ruby Proc/lambda merge operators are no longer accepted. They
|
|
48
|
+
silently used concatenation on other threads, which could change stored values.
|
|
49
|
+
Keep `:concat` / `:string_concat` for concatenation. For custom updates, use a
|
|
50
|
+
serializable transaction that reads the value and writes the computed result;
|
|
51
|
+
handle `SlateDb::TransactionError` conflicts in the application. Databases with
|
|
52
|
+
unresolved custom merge operands need an application-specific migration before
|
|
53
|
+
upgrading. Do not reopen them with `:concat` unless concatenation was their
|
|
54
|
+
intended merge rule.
|
|
55
|
+
- Reader disk-cache keywords remain supported, but upstream deprecated the
|
|
56
|
+
underlying `CachedObjectStore` in 0.18.1. The new mirror and Rust trait-based
|
|
57
|
+
extension APIs are not exposed by this gem yet.
|
|
58
|
+
|
|
59
|
+
Upstream also fixes clone-compaction retention/data-loss cases, writer ownership
|
|
60
|
+
validation after WAL replay, write stalls, upload retry/concurrency behavior, and
|
|
61
|
+
iterator fetch cancellation. It improves memtable seeks and point-read/scan work.
|
|
62
|
+
See the full [0.18.0 release notes](https://github.com/slatedb/slatedb/releases/tag/v0.18.0)
|
|
63
|
+
and [0.18.1 release notes](https://github.com/slatedb/slatedb/releases/tag/v0.18.1).
|
|
64
|
+
|
|
33
65
|
## Usage
|
|
34
66
|
|
|
35
67
|
### Basic Operations
|
|
@@ -64,6 +96,73 @@ SlateDb::Database.open("/tmp/mydb") do |db|
|
|
|
64
96
|
end # automatically closed
|
|
65
97
|
```
|
|
66
98
|
|
|
99
|
+
### Point-Read Concurrency
|
|
100
|
+
|
|
101
|
+
```ruby
|
|
102
|
+
db.get("key", max_concurrent_sources: 4, cache_blocks: false)
|
|
103
|
+
db.get_entry("key", max_concurrent_sources: 4) # value plus sequence/timestamps
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
`max_concurrent_sources` defaults to 1 (serial fetching); zero also means 1.
|
|
107
|
+
SlateDB checks the newest source alone, then fetches up to this many sources
|
|
108
|
+
concurrently after a miss. Higher values can reduce latency but increase object
|
|
109
|
+
store requests and bandwidth. The same keyword works with `get` on readers,
|
|
110
|
+
snapshots, and transactions. It does not change isolation or durability.
|
|
111
|
+
|
|
112
|
+
### Compaction Controls
|
|
113
|
+
|
|
114
|
+
SlateDB 0.17 adds controls for how long compaction inputs remain protected from
|
|
115
|
+
garbage collection and when to consolidate sorted runs:
|
|
116
|
+
|
|
117
|
+
```ruby
|
|
118
|
+
SlateDb::Database.open("mydb",
|
|
119
|
+
url: "s3://mybucket/path",
|
|
120
|
+
compactor_checkpoint_lifetime: 30 * 60_000,
|
|
121
|
+
sorted_run_consolidation_threshold: 8) do |db|
|
|
122
|
+
db.put("key", "value")
|
|
123
|
+
end
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
`compactor_checkpoint_lifetime` is in milliseconds and defaults to 15 minutes.
|
|
127
|
+
Reads, scans, snapshots, and transactions using old compaction inputs must finish
|
|
128
|
+
within that time; after it expires, garbage collection can remove those inputs.
|
|
129
|
+
|
|
130
|
+
`sorted_run_consolidation_threshold` starts fallback consolidation when the
|
|
131
|
+
projected sorted-run count exceeds the threshold. It is not a hard limit.
|
|
132
|
+
Zero disables it (the default); use at least two to enable it. SlateDB warns and
|
|
133
|
+
disables the trigger for one. Low thresholds can increase storage reads and writes.
|
|
134
|
+
|
|
135
|
+
SlateDB 0.18 adds an opt-in experimental mode:
|
|
136
|
+
|
|
137
|
+
```ruby
|
|
138
|
+
SlateDb::Database.open("mydb", url: "s3://mybucket/path",
|
|
139
|
+
experimental_incremental_compaction: true) do |db|
|
|
140
|
+
db.put("key", "value")
|
|
141
|
+
end
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
This publishes output from sorted-run compactions at most once a minute before
|
|
145
|
+
the full compaction finishes. It can reduce read amplification and allow earlier
|
|
146
|
+
storage reclamation after the compactor checkpoint expires. Compactions with L0
|
|
147
|
+
inputs still publish only when complete. It is disabled by default, experimental,
|
|
148
|
+
and may change or disappear upstream.
|
|
149
|
+
|
|
150
|
+
### SST Block Alignment
|
|
151
|
+
|
|
152
|
+
`Database.open(..., sst_block_alignment: true)` pads SST data blocks to 4 KiB
|
|
153
|
+
boundaries, including oversized entries. It is disabled by default, uses more
|
|
154
|
+
storage, and cannot be combined with compression.
|
|
155
|
+
|
|
156
|
+
**Upgrade every reader before enabling alignment on a writer.** Older engines
|
|
157
|
+
that do not understand padded SSTs cannot read them. Once padded files exist,
|
|
158
|
+
rolling those readers back is not safe; disabling the option does not rewrite
|
|
159
|
+
existing files.
|
|
160
|
+
|
|
161
|
+
### Key Size
|
|
162
|
+
|
|
163
|
+
Keys can now exceed the previous 65,535-byte limit, up to SlateDB's new
|
|
164
|
+
4,294,967,295-byte limit. Very large keys still require enough memory to process.
|
|
165
|
+
|
|
67
166
|
### Persistent Storage
|
|
68
167
|
|
|
69
168
|
For persistent storage, provide an object store URL:
|
|
@@ -313,53 +412,19 @@ db.batch do |b|
|
|
|
313
412
|
end
|
|
314
413
|
```
|
|
315
414
|
|
|
316
|
-
####
|
|
317
|
-
|
|
318
|
-
You can provide a Ruby Proc/lambda as a custom merge operator:
|
|
319
|
-
|
|
320
|
-
```ruby
|
|
321
|
-
# Counter merge operator (adds numbers)
|
|
322
|
-
counter_merge = ->(key, existing, new_value) {
|
|
323
|
-
existing_num = existing ? existing.to_i : 0
|
|
324
|
-
(existing_num + new_value.to_i).to_s
|
|
325
|
-
}
|
|
326
|
-
|
|
327
|
-
SlateDb::Database.open("/tmp/mydb", merge_operator: counter_merge) do |db|
|
|
328
|
-
db.merge("visits", "1")
|
|
329
|
-
db.merge("visits", "1")
|
|
330
|
-
db.merge("visits", "1")
|
|
331
|
-
|
|
332
|
-
db.get("visits") # => "3"
|
|
333
|
-
end
|
|
415
|
+
#### Available Merge Operators
|
|
334
416
|
|
|
335
|
-
|
|
336
|
-
max_merge = ->(key, existing, new_value) {
|
|
337
|
-
existing_num = existing ? existing.to_i : 0
|
|
338
|
-
new_num = new_value.to_i
|
|
339
|
-
[existing_num, new_num].max.to_s
|
|
340
|
-
}
|
|
417
|
+
- `:string_concat` (or `:concat`) - Concatenates byte values (built-in)
|
|
341
418
|
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
db.merge("high_score", "250")
|
|
345
|
-
db.merge("high_score", "150")
|
|
419
|
+
Ruby Proc/lambda operators are rejected. Use a serializable read/modify/write
|
|
420
|
+
transaction for custom logic, with application-level conflict handling:
|
|
346
421
|
|
|
347
|
-
|
|
422
|
+
```ruby
|
|
423
|
+
db.transaction(isolation: :serializable) do |txn|
|
|
424
|
+
txn.put("visits", (txn.get("visits").to_i + 1).to_s)
|
|
348
425
|
end
|
|
349
426
|
```
|
|
350
427
|
|
|
351
|
-
The proc receives three arguments:
|
|
352
|
-
- `key` - The key being merged
|
|
353
|
-
- `existing` - The existing value (nil if no value exists)
|
|
354
|
-
- `new_value` - The new merge operand
|
|
355
|
-
|
|
356
|
-
**Note:** Custom Proc merge operators work best with direct `db.merge()` calls. When used with transactions or batches, some merge operations may be processed on background threads and fall back to string concatenation.
|
|
357
|
-
|
|
358
|
-
#### Available Merge Operators
|
|
359
|
-
|
|
360
|
-
- `:string_concat` (or `:concat`) - Concatenates byte values (built-in)
|
|
361
|
-
- Any `Proc` or `lambda` - Custom merge logic
|
|
362
|
-
|
|
363
428
|
### Write Batches
|
|
364
429
|
|
|
365
430
|
Perform multiple writes atomically:
|
|
@@ -467,6 +532,29 @@ SlateDb::Database.open("/tmp/mydb", url: "file:///tmp/mydb") do |db|
|
|
|
467
532
|
end
|
|
468
533
|
```
|
|
469
534
|
|
|
535
|
+
`create_checkpoint` waits until the checkpoint is stored. Its default
|
|
536
|
+
`scope: :durable` preserves the existing behavior. To include writes made with
|
|
537
|
+
`await_durable: false`, use `scope: :all`; no separate `flush` is needed.
|
|
538
|
+
|
|
539
|
+
To separate capturing the write boundary from waiting for storage:
|
|
540
|
+
|
|
541
|
+
```ruby
|
|
542
|
+
db.put("key", "before", await_durable: false)
|
|
543
|
+
handle = db.begin_checkpoint(scope: :all, name: "backup", lifetime: 3_600_000)
|
|
544
|
+
id = handle.id # UUID is available immediately
|
|
545
|
+
db.put("key", "after") # not part of this checkpoint
|
|
546
|
+
checkpoint = handle.wait # => { id: id, manifest_id: ... }
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
With `scope: :all`, writes completed before `begin_checkpoint` starts are included;
|
|
550
|
+
writes started after it returns are excluded. Concurrent writes during the call
|
|
551
|
+
may be included. Starting the checkpoint can still wait for internal flush work;
|
|
552
|
+
it is not a fully nonblocking Ruby call. `wait` releases the Ruby GVL while
|
|
553
|
+
waiting, consumes the handle, and raises `SlateDb::ClosedError` if called again.
|
|
554
|
+
`id` remains readable. Dropping the handle does not cancel or delete the checkpoint.
|
|
555
|
+
If waiting fails, use the UUID with `Admin#list_checkpoints` to check its outcome
|
|
556
|
+
before starting another checkpoint.
|
|
557
|
+
|
|
470
558
|
### Snapshots
|
|
471
559
|
|
|
472
560
|
Point-in-time consistent reads:
|
|
@@ -489,6 +577,11 @@ value = snap.get("key")
|
|
|
489
577
|
snap.close
|
|
490
578
|
```
|
|
491
579
|
|
|
580
|
+
Range and prefix scans on snapshots and transactions accept the same `order:`
|
|
581
|
+
keyword as databases and readers, including `:asc`/`:ascending` and
|
|
582
|
+
`:desc`/`:descending`. Bounds remain lower-inclusive and upper-exclusive in
|
|
583
|
+
either direction.
|
|
584
|
+
|
|
492
585
|
### Reader (Read-Only Access)
|
|
493
586
|
|
|
494
587
|
Open a database in read-only mode, useful for replicas:
|
|
@@ -501,6 +594,14 @@ SlateDb::Reader.open("/tmp/mydb", url: "s3://bucket/path") do |reader|
|
|
|
501
594
|
reader.scan("prefix:").each do |k, v|
|
|
502
595
|
puts "#{k}: #{v}"
|
|
503
596
|
end
|
|
597
|
+
|
|
598
|
+
# Range and prefix scans accept :asc/:ascending or :desc/:descending.
|
|
599
|
+
reader.scan("user:100", "user:200", order: :desc).each do |k, v|
|
|
600
|
+
puts "#{k}: #{v}"
|
|
601
|
+
end
|
|
602
|
+
reader.scan_prefix("user:", from: "100", to: "200", order: :desc).each do |k, v|
|
|
603
|
+
puts "#{k}: #{v}"
|
|
604
|
+
end
|
|
504
605
|
end
|
|
505
606
|
|
|
506
607
|
# Open at a specific checkpoint (pins the reader to that checkpoint's state)
|
|
@@ -532,6 +633,14 @@ SlateDb::Reader.open("/tmp/mydb",
|
|
|
532
633
|
end
|
|
533
634
|
```
|
|
534
635
|
|
|
636
|
+
When the writer uses `merge_operator: :concat`, pass the same operator to
|
|
637
|
+
`Reader.open` so WAL replay and reads apply the correct merge rule. Unknown
|
|
638
|
+
operator names now raise `SlateDb::InvalidArgumentError` instead of being ignored.
|
|
639
|
+
|
|
640
|
+
The `cache_root` and `max_open_file_handles` keywords still work, but SlateDB
|
|
641
|
+
0.18.1 deprecates this legacy disk cache in favor of its separate mirror crate or
|
|
642
|
+
tiered block cache. Those replacement APIs are not yet exposed here.
|
|
643
|
+
|
|
535
644
|
### Admin Operations
|
|
536
645
|
|
|
537
646
|
Administrative operations for database management:
|
|
@@ -558,9 +667,6 @@ admin.delete_checkpoint("uuid")
|
|
|
558
667
|
# Garbage Collection
|
|
559
668
|
admin.run_gc # Run with default settings
|
|
560
669
|
admin.run_gc(min_age: 3600_000) # Set min age for all directories (1 hour)
|
|
561
|
-
admin.run_gc(manifest_min_age: 86400_000) # Custom age for manifest (1 day)
|
|
562
|
-
admin.run_gc(wal_min_age: 60_000) # Custom age for WAL (1 minute)
|
|
563
|
-
admin.run_gc(compacted_min_age: 60_000) # Custom age for compacted (1 minute)
|
|
564
670
|
|
|
565
671
|
# Delete a database (SlateDB >= 0.16.0)
|
|
566
672
|
would_delete = admin.delete_db # Dry run: returns the object paths
|
data/lib/slatedb/3.3/slatedb.so
CHANGED
|
Binary file
|
data/lib/slatedb/3.4/slatedb.so
CHANGED
|
Binary file
|
data/lib/slatedb/4.0/slatedb.so
CHANGED
|
Binary file
|
data/lib/slatedb/database.rb
CHANGED
|
@@ -9,9 +9,18 @@ module SlateDb
|
|
|
9
9
|
#
|
|
10
10
|
# @param path [String] The path identifier for the database
|
|
11
11
|
# @param url [String, nil] Optional object store URL (e.g., "s3://bucket/path")
|
|
12
|
-
# @param merge_operator [Symbol, String,
|
|
13
|
-
#
|
|
14
|
-
#
|
|
12
|
+
# @param merge_operator [Symbol, String, nil] Optional built-in merge operator
|
|
13
|
+
# (:string_concat or :concat). Ruby callbacks are not supported.
|
|
14
|
+
# @param compactor_checkpoint_lifetime [Integer, nil] How long compaction inputs
|
|
15
|
+
# are protected from garbage collection, in milliseconds (default: 15 minutes).
|
|
16
|
+
# Reads using those inputs must finish within this time.
|
|
17
|
+
# @param sorted_run_consolidation_threshold [Integer, nil] Sorted-run count above
|
|
18
|
+
# which fallback consolidation starts. Zero disables it (default); use at least
|
|
19
|
+
# two to enable it. One is disabled by SlateDB with a warning. This is not a hard limit.
|
|
20
|
+
# @param experimental_incremental_compaction [Boolean, nil] Publish sorted-run
|
|
21
|
+
# compaction output before completion (experimental, disabled by default).
|
|
22
|
+
# @param sst_block_alignment [Boolean, nil] Align SST data blocks to 4 KiB
|
|
23
|
+
# boundaries (disabled by default; incompatible with compression).
|
|
15
24
|
# @yield [db] If a block is given, yields the database and ensures it's closed
|
|
16
25
|
# @return [Database] The opened database (or block result if block given)
|
|
17
26
|
#
|
|
@@ -28,28 +37,22 @@ module SlateDb
|
|
|
28
37
|
# @example Open with S3 backend
|
|
29
38
|
# db = SlateDb::Database.open("/tmp/mydb", url: "s3://mybucket/path")
|
|
30
39
|
#
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
case merge_operator
|
|
45
|
-
when Symbol, String
|
|
46
|
-
opts[:merge_operator] = merge_operator.to_s
|
|
47
|
-
when Proc
|
|
48
|
-
# Store the proc to prevent GC and pass to Rust
|
|
49
|
-
@_merge_operator_proc = merge_operator
|
|
50
|
-
opts[:merge_operator_proc] = merge_operator
|
|
40
|
+
def open(path, url: nil, merge_operator: nil,
|
|
41
|
+
compactor_checkpoint_lifetime: nil, sorted_run_consolidation_threshold: nil,
|
|
42
|
+
experimental_incremental_compaction: nil, sst_block_alignment: nil)
|
|
43
|
+
opts = {
|
|
44
|
+
compactor_checkpoint_lifetime: compactor_checkpoint_lifetime,
|
|
45
|
+
sorted_run_consolidation_threshold: sorted_run_consolidation_threshold,
|
|
46
|
+
experimental_incremental_compaction: experimental_incremental_compaction,
|
|
47
|
+
sst_block_alignment: sst_block_alignment
|
|
48
|
+
}.compact
|
|
49
|
+
|
|
50
|
+
unless merge_operator.nil? || merge_operator.is_a?(Symbol) || merge_operator.is_a?(String)
|
|
51
|
+
raise ArgumentError, "merge_operator must be :string_concat or :concat; Ruby callbacks are not supported"
|
|
51
52
|
end
|
|
52
53
|
|
|
54
|
+
opts[:merge_operator] = merge_operator.to_s if merge_operator
|
|
55
|
+
|
|
53
56
|
db = _open(path, url, opts)
|
|
54
57
|
|
|
55
58
|
if block_given?
|
|
@@ -74,6 +77,8 @@ module SlateDb
|
|
|
74
77
|
# @param durability_filter [String, nil] Filter by durability level ("remote" or "memory")
|
|
75
78
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
76
79
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
80
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources to fetch after
|
|
81
|
+
# the first source misses (default: 1; zero also means serial fetching).
|
|
77
82
|
# @return [String, nil] The value, or nil if not found
|
|
78
83
|
#
|
|
79
84
|
# @example Basic get
|
|
@@ -82,11 +87,12 @@ module SlateDb
|
|
|
82
87
|
# @example Get with options
|
|
83
88
|
# value = db.get("mykey", durability_filter: "memory", dirty: true)
|
|
84
89
|
#
|
|
85
|
-
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
90
|
+
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
86
91
|
opts = {}
|
|
87
92
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
88
93
|
opts[:dirty] = dirty unless dirty.nil?
|
|
89
94
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
95
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
90
96
|
|
|
91
97
|
if opts.empty?
|
|
92
98
|
_get(key)
|
|
@@ -102,17 +108,20 @@ module SlateDb
|
|
|
102
108
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
103
109
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
104
110
|
# @return [Hash, nil] A hash with :key, :value, :seq, :create_ts, and :expire_ts, or nil if not found
|
|
111
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
|
|
112
|
+
# miss. See {#get}.
|
|
105
113
|
#
|
|
106
114
|
# @example Inspect metadata
|
|
107
115
|
# entry = db.get_key_value("mykey")
|
|
108
116
|
# entry[:value] # => "myvalue"
|
|
109
117
|
# entry[:seq] # => sequence number
|
|
110
118
|
#
|
|
111
|
-
def get_key_value(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
119
|
+
def get_key_value(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
112
120
|
opts = {}
|
|
113
121
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
114
122
|
opts[:dirty] = dirty unless dirty.nil?
|
|
115
123
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
124
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
116
125
|
|
|
117
126
|
if opts.empty?
|
|
118
127
|
_get_key_value(key)
|
|
@@ -484,6 +493,8 @@ module SlateDb
|
|
|
484
493
|
#
|
|
485
494
|
# @param lifetime [Integer, nil] Checkpoint lifetime in milliseconds
|
|
486
495
|
# @param name [String, nil] Optional name for the checkpoint
|
|
496
|
+
# @param scope [Symbol, String] :durable (default) captures durable writes;
|
|
497
|
+
# :all also flushes pending writes before storing the checkpoint.
|
|
487
498
|
# @return [Hash] Hash with :id (UUID string) and :manifest_id (integer)
|
|
488
499
|
#
|
|
489
500
|
# @example Create a named checkpoint
|
|
@@ -493,13 +504,26 @@ module SlateDb
|
|
|
493
504
|
# @example Create a checkpoint with lifetime
|
|
494
505
|
# checkpoint = db.create_checkpoint(lifetime: 3600_000) # 1 hour
|
|
495
506
|
#
|
|
496
|
-
def create_checkpoint(lifetime: nil, name: nil)
|
|
497
|
-
opts = {}
|
|
507
|
+
def create_checkpoint(lifetime: nil, name: nil, scope: :durable)
|
|
508
|
+
opts = { scope: scope.to_s }
|
|
498
509
|
opts[:lifetime] = lifetime if lifetime
|
|
499
510
|
opts[:name] = name if name
|
|
500
511
|
_create_checkpoint(opts)
|
|
501
512
|
end
|
|
502
513
|
|
|
514
|
+
# Start a checkpoint without waiting for it to be stored.
|
|
515
|
+
#
|
|
516
|
+
# With scope: :all, captures writes completed before this call and excludes
|
|
517
|
+
# writes started after it returns. Writes during the call may be included.
|
|
518
|
+
# Options match {#create_checkpoint}. Dropping the handle does not cancel it.
|
|
519
|
+
# @return [CheckpointHandle] #id is available immediately; #wait returns the
|
|
520
|
+
# :id/:manifest_id hash once stored and consumes the handle. A second wait
|
|
521
|
+
# raises ClosedError. #id remains available after waiting.
|
|
522
|
+
def begin_checkpoint(lifetime: nil, name: nil, scope: :durable)
|
|
523
|
+
opts = { scope: scope.to_s, lifetime: lifetime, name: name }.compact
|
|
524
|
+
_begin_checkpoint(opts)
|
|
525
|
+
end
|
|
526
|
+
|
|
503
527
|
# Get database metrics registry.
|
|
504
528
|
#
|
|
505
529
|
# @return [Metrics] Metrics registry
|
data/lib/slatedb/reader.rb
CHANGED
|
@@ -21,6 +21,7 @@ module SlateDb
|
|
|
21
21
|
# @param cache_root [String, nil] Root folder for the reader's local on-disk
|
|
22
22
|
# object-store cache. Setting this enables the cached object store; when it is
|
|
23
23
|
# not set the cache (and `max_open_file_handles`) has no effect.
|
|
24
|
+
# This legacy cache is deprecated upstream in SlateDB 0.18.1.
|
|
24
25
|
# @param max_open_file_handles [Integer, nil] Maximum number of file handles to keep
|
|
25
26
|
# open in the reader's file-handle cache. When the limit is reached, the least
|
|
26
27
|
# recently used handle is closed (default: 1000). Only takes effect when
|
|
@@ -54,29 +55,31 @@ module SlateDb
|
|
|
54
55
|
manifest_poll_interval: nil, checkpoint_lifetime: nil,
|
|
55
56
|
max_memtable_bytes: nil, cache_root: nil, max_open_file_handles: nil,
|
|
56
57
|
merge_operator: nil)
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
opts
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
58
|
+
unless merge_operator.nil? || merge_operator.is_a?(Symbol) || merge_operator.is_a?(String)
|
|
59
|
+
raise ArgumentError, "merge_operator must be :string_concat or :concat; Ruby callbacks are not supported"
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
opts = {
|
|
63
|
+
follow_latest: follow_latest,
|
|
64
|
+
manifest_poll_interval: manifest_poll_interval,
|
|
65
|
+
checkpoint_lifetime: checkpoint_lifetime,
|
|
66
|
+
max_memtable_bytes: max_memtable_bytes,
|
|
67
|
+
cache_root: cache_root,
|
|
68
|
+
max_open_file_handles: max_open_file_handles,
|
|
69
|
+
merge_operator: merge_operator&.to_s
|
|
70
|
+
}.compact
|
|
65
71
|
|
|
66
72
|
reader = _open(path, url, checkpoint_id, opts)
|
|
73
|
+
return reader unless block_given?
|
|
67
74
|
|
|
68
|
-
|
|
75
|
+
begin
|
|
76
|
+
yield reader
|
|
77
|
+
ensure
|
|
69
78
|
begin
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
reader.close
|
|
74
|
-
rescue StandardError
|
|
75
|
-
nil
|
|
76
|
-
end
|
|
79
|
+
reader.close
|
|
80
|
+
rescue StandardError
|
|
81
|
+
nil
|
|
77
82
|
end
|
|
78
|
-
else
|
|
79
|
-
reader
|
|
80
83
|
end
|
|
81
84
|
end
|
|
82
85
|
end
|
|
@@ -87,13 +90,16 @@ module SlateDb
|
|
|
87
90
|
# @param durability_filter [String, nil] Filter by durability level ("remote" or "memory")
|
|
88
91
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
89
92
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
93
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
|
|
94
|
+
# miss (default: 1; zero also means serial fetching).
|
|
90
95
|
# @return [String, nil] The value, or nil if not found
|
|
91
96
|
#
|
|
92
|
-
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
97
|
+
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
93
98
|
opts = {}
|
|
94
99
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
95
100
|
opts[:dirty] = dirty unless dirty.nil?
|
|
96
101
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
102
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
97
103
|
|
|
98
104
|
if opts.empty?
|
|
99
105
|
_get(key)
|
|
@@ -106,16 +112,19 @@ module SlateDb
|
|
|
106
112
|
#
|
|
107
113
|
# @param start_key [String] The start key (inclusive)
|
|
108
114
|
# @param end_key [String, nil] The end key (exclusive)
|
|
115
|
+
# @param order [Symbol, String, nil] Iteration order (:asc/:ascending or :desc/:descending)
|
|
109
116
|
# @return [Iterator] An iterator over key-value pairs
|
|
110
117
|
#
|
|
111
118
|
def scan(start_key, end_key = nil, durability_filter: nil, dirty: nil,
|
|
112
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, &)
|
|
113
|
-
opts = {
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil, &)
|
|
120
|
+
opts = {
|
|
121
|
+
durability_filter: durability_filter&.to_s,
|
|
122
|
+
dirty: dirty,
|
|
123
|
+
read_ahead_bytes: read_ahead_bytes,
|
|
124
|
+
cache_blocks: cache_blocks,
|
|
125
|
+
max_fetch_tasks: max_fetch_tasks,
|
|
126
|
+
order: order&.to_s
|
|
127
|
+
}.compact
|
|
119
128
|
|
|
120
129
|
iter = if opts.empty?
|
|
121
130
|
_scan(start_key, end_key)
|
|
@@ -138,6 +147,7 @@ module SlateDb
|
|
|
138
147
|
# @param read_ahead_bytes [Integer, nil] Number of bytes to read ahead
|
|
139
148
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
140
149
|
# @param max_fetch_tasks [Integer, nil] Maximum number of fetch tasks
|
|
150
|
+
# @param order [Symbol, String, nil] Iteration order (:asc/:ascending or :desc/:descending)
|
|
141
151
|
# @param from [String, nil] Inclusive lower bound suffix, appended to the
|
|
142
152
|
# prefix, to start scanning from (e.g. prefix "user:" with from "100"
|
|
143
153
|
# starts at "user:100"). Defaults to the start of the prefix.
|
|
@@ -146,7 +156,7 @@ module SlateDb
|
|
|
146
156
|
# @return [Iterator] An iterator over key-value pairs
|
|
147
157
|
#
|
|
148
158
|
def scan_prefix(prefix, durability_filter: nil, dirty: nil,
|
|
149
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil,
|
|
159
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil,
|
|
150
160
|
from: nil, to: nil, &)
|
|
151
161
|
opts = {
|
|
152
162
|
durability_filter: durability_filter&.to_s,
|
|
@@ -154,6 +164,7 @@ module SlateDb
|
|
|
154
164
|
read_ahead_bytes: read_ahead_bytes,
|
|
155
165
|
cache_blocks: cache_blocks,
|
|
156
166
|
max_fetch_tasks: max_fetch_tasks,
|
|
167
|
+
order: order&.to_s,
|
|
157
168
|
subrange_from: from,
|
|
158
169
|
subrange_to: to
|
|
159
170
|
}.compact
|
data/lib/slatedb/snapshot.rb
CHANGED
|
@@ -8,13 +8,16 @@ module SlateDb
|
|
|
8
8
|
# @param durability_filter [String, nil] Filter by durability level
|
|
9
9
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
10
10
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
11
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
|
|
12
|
+
# miss (default: 1; zero also means serial fetching).
|
|
11
13
|
# @return [String, nil] The value, or nil if not found
|
|
12
14
|
#
|
|
13
|
-
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
15
|
+
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
14
16
|
opts = {}
|
|
15
17
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
16
18
|
opts[:dirty] = dirty unless dirty.nil?
|
|
17
19
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
20
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
18
21
|
|
|
19
22
|
if opts.empty?
|
|
20
23
|
_get(key)
|
|
@@ -27,16 +30,19 @@ module SlateDb
|
|
|
27
30
|
#
|
|
28
31
|
# @param start_key [String] The start key (inclusive)
|
|
29
32
|
# @param end_key [String, nil] The end key (exclusive)
|
|
33
|
+
# @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
|
|
30
34
|
# @return [Iterator] An iterator over key-value pairs
|
|
31
35
|
#
|
|
32
36
|
def scan(start_key, end_key = nil, durability_filter: nil, dirty: nil,
|
|
33
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, &)
|
|
34
|
-
opts = {
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
37
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil, &)
|
|
38
|
+
opts = {
|
|
39
|
+
durability_filter: durability_filter&.to_s,
|
|
40
|
+
dirty: dirty,
|
|
41
|
+
read_ahead_bytes: read_ahead_bytes,
|
|
42
|
+
cache_blocks: cache_blocks,
|
|
43
|
+
max_fetch_tasks: max_fetch_tasks,
|
|
44
|
+
order: order&.to_s
|
|
45
|
+
}.compact
|
|
40
46
|
|
|
41
47
|
iter = if opts.empty?
|
|
42
48
|
_scan(start_key, end_key)
|
|
@@ -59,6 +65,7 @@ module SlateDb
|
|
|
59
65
|
# @param read_ahead_bytes [Integer, nil] Number of bytes to read ahead
|
|
60
66
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
61
67
|
# @param max_fetch_tasks [Integer, nil] Maximum number of fetch tasks
|
|
68
|
+
# @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
|
|
62
69
|
# @param from [String, nil] Inclusive lower bound suffix, appended to the
|
|
63
70
|
# prefix, to start scanning from. Defaults to the start of the prefix.
|
|
64
71
|
# @param to [String, nil] Exclusive upper bound suffix, appended to the
|
|
@@ -66,7 +73,7 @@ module SlateDb
|
|
|
66
73
|
# @return [Iterator] An iterator over key-value pairs
|
|
67
74
|
#
|
|
68
75
|
def scan_prefix(prefix, durability_filter: nil, dirty: nil,
|
|
69
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil,
|
|
76
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil,
|
|
70
77
|
from: nil, to: nil, &)
|
|
71
78
|
opts = {
|
|
72
79
|
durability_filter: durability_filter&.to_s,
|
|
@@ -74,6 +81,7 @@ module SlateDb
|
|
|
74
81
|
read_ahead_bytes: read_ahead_bytes,
|
|
75
82
|
cache_blocks: cache_blocks,
|
|
76
83
|
max_fetch_tasks: max_fetch_tasks,
|
|
84
|
+
order: order&.to_s,
|
|
77
85
|
subrange_from: from,
|
|
78
86
|
subrange_to: to
|
|
79
87
|
}.compact
|
data/lib/slatedb/transaction.rb
CHANGED
|
@@ -8,13 +8,16 @@ module SlateDb
|
|
|
8
8
|
# @param durability_filter [String, nil] Filter by durability level
|
|
9
9
|
# @param dirty [Boolean, nil] Whether to include uncommitted data
|
|
10
10
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
11
|
+
# @param max_concurrent_sources [Integer, nil] Concurrent sources after the first
|
|
12
|
+
# miss (default: 1; zero also means serial fetching).
|
|
11
13
|
# @return [String, nil] The value, or nil if not found
|
|
12
14
|
#
|
|
13
|
-
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil)
|
|
15
|
+
def get(key, durability_filter: nil, dirty: nil, cache_blocks: nil, max_concurrent_sources: nil)
|
|
14
16
|
opts = {}
|
|
15
17
|
opts[:durability_filter] = durability_filter.to_s if durability_filter
|
|
16
18
|
opts[:dirty] = dirty unless dirty.nil?
|
|
17
19
|
opts[:cache_blocks] = cache_blocks unless cache_blocks.nil?
|
|
20
|
+
opts[:max_concurrent_sources] = max_concurrent_sources unless max_concurrent_sources.nil?
|
|
18
21
|
|
|
19
22
|
if opts.empty?
|
|
20
23
|
_get(key)
|
|
@@ -66,16 +69,19 @@ module SlateDb
|
|
|
66
69
|
#
|
|
67
70
|
# @param start_key [String] The start key (inclusive)
|
|
68
71
|
# @param end_key [String, nil] The end key (exclusive)
|
|
72
|
+
# @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
|
|
69
73
|
# @return [Iterator] An iterator over key-value pairs
|
|
70
74
|
#
|
|
71
75
|
def scan(start_key, end_key = nil, durability_filter: nil, dirty: nil,
|
|
72
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, &)
|
|
73
|
-
opts = {
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
76
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil, &)
|
|
77
|
+
opts = {
|
|
78
|
+
durability_filter: durability_filter&.to_s,
|
|
79
|
+
dirty: dirty,
|
|
80
|
+
read_ahead_bytes: read_ahead_bytes,
|
|
81
|
+
cache_blocks: cache_blocks,
|
|
82
|
+
max_fetch_tasks: max_fetch_tasks,
|
|
83
|
+
order: order&.to_s
|
|
84
|
+
}.compact
|
|
79
85
|
|
|
80
86
|
iter = if opts.empty?
|
|
81
87
|
_scan(start_key, end_key)
|
|
@@ -98,6 +104,7 @@ module SlateDb
|
|
|
98
104
|
# @param read_ahead_bytes [Integer, nil] Number of bytes to read ahead
|
|
99
105
|
# @param cache_blocks [Boolean, nil] Whether to cache blocks
|
|
100
106
|
# @param max_fetch_tasks [Integer, nil] Maximum number of fetch tasks
|
|
107
|
+
# @param order [Symbol, String, nil] :asc/:ascending or :desc/:descending
|
|
101
108
|
# @param from [String, nil] Inclusive lower bound suffix, appended to the
|
|
102
109
|
# prefix, to start scanning from. Defaults to the start of the prefix.
|
|
103
110
|
# @param to [String, nil] Exclusive upper bound suffix, appended to the
|
|
@@ -105,7 +112,7 @@ module SlateDb
|
|
|
105
112
|
# @return [Iterator] An iterator over key-value pairs
|
|
106
113
|
#
|
|
107
114
|
def scan_prefix(prefix, durability_filter: nil, dirty: nil,
|
|
108
|
-
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil,
|
|
115
|
+
read_ahead_bytes: nil, cache_blocks: nil, max_fetch_tasks: nil, order: nil,
|
|
109
116
|
from: nil, to: nil, &)
|
|
110
117
|
opts = {
|
|
111
118
|
durability_filter: durability_filter&.to_s,
|
|
@@ -113,6 +120,7 @@ module SlateDb
|
|
|
113
120
|
read_ahead_bytes: read_ahead_bytes,
|
|
114
121
|
cache_blocks: cache_blocks,
|
|
115
122
|
max_fetch_tasks: max_fetch_tasks,
|
|
123
|
+
order: order&.to_s,
|
|
116
124
|
subrange_from: from,
|
|
117
125
|
subrange_to: to
|
|
118
126
|
}.compact
|
data/lib/slatedb/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: slatedb
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.7.0
|
|
5
5
|
platform: aarch64-linux
|
|
6
6
|
authors:
|
|
7
7
|
- SlateDB Contributors
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
11
|
+
date: 2026-11-11 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: rake
|
|
@@ -58,14 +58,14 @@ dependencies:
|
|
|
58
58
|
requirements:
|
|
59
59
|
- - "~>"
|
|
60
60
|
- !ruby/object:Gem::Version
|
|
61
|
-
version: '1.
|
|
61
|
+
version: '1.92'
|
|
62
62
|
type: :development
|
|
63
63
|
prerelease: false
|
|
64
64
|
version_requirements: !ruby/object:Gem::Requirement
|
|
65
65
|
requirements:
|
|
66
66
|
- - "~>"
|
|
67
67
|
- !ruby/object:Gem::Version
|
|
68
|
-
version: '1.
|
|
68
|
+
version: '1.92'
|
|
69
69
|
description: A cloud-native embedded key-value store built on object storage
|
|
70
70
|
email:
|
|
71
71
|
- slatedb@example.com
|